Brig. Gen. Pat White (left) had a personalized avatar made for him when he was the deputy commander of the U.S. Army Combined Arms Center-Training at Fort Leavenworth, Kan. The Army is developing training avatars that not only look like an individual soldier, but also reflect some of his real-world capabilities. (Army)
It would be great to have a team of guys, all of them 6 feet 2 inches, 220 pounds, expert marksmen, capable of walking on water and swimming through mud. But while such cookie-cutter avatars are standard in virtual scenarios, they’re a far cry from what you find in the real world.
“The problem with that is it injects falsehood into the simulation, because not everybody possesses the same skills,” said Anthony Rolfe, a senior analyst for the Army’s TCM Gaming, which is working to make avatars match their real-world counterparts.
Rolfe and his colleagues at TCM Gaming, the gaming capability managers for Training and Doctrine Command, work out of the National Simulation Center at Fort Leavenworth, Kan. Their project, Human Dimension Modeling, proceeds from the hypothesis that more accurate online doppelgangers will improve soldiers’ training, in both the virtual and physical realms.
So the analysts created a modification for Bohemia Interactive’s Virtual Battlespace 2, the Army’s primary gaming engine for training. Their software can take information from the Army’s Digital Training Management System, a database used to track individuals’ progress, and port the characteristics into the game.
This mod allows soldiers’ in-game avatars to better resemble them in appearance: height, weight, gender and race. More importantly, it incorporates physical fitness scores, weapons qualifications scores and special skills, such as certificates for airborne qualification or sniper skills. This means instead of superheroes with perfect marksmanship, soldiers play the games with avatars who reflect their own real-life fitness and skills.
“You’re only as fast as your slowest squad member. You don’t go 200 yards at breakneck speed if three of your squad members are lagging behind,” said Mike Peppers, director of the mission training complex at Joint Base Lewis-McChord, Wash. “Any simulation’s value is proportional to its realism. The more realism we can create, the more personalization — it’s a bit of a no-brainer. It will make the simulation crawl and walk stages more effective and efficient.”
The algorithms developed by TCM Gaming can create a fatigue model based on the soldier’s PT scores, body mass index, the amount of weight he is carrying, and the land he is maneuvering on. Trainers know that it is easy for soldiers to fire in ideal conditions, but after running several hundred meters with a full pack, qualification ability degrades. The hybrid avatars reflect this; a fatigued soldier’s avatar will take a knee to rest and recover, or find it harder to sight its target.
“The key thing was to give lifelike traits of a soldier to that avatar, but also be able to do mathematical calculations based on terrain,” said Mike Enloe, chief engineer for the National Simulation Center. “That’s going to affect how you exert energy.”
A Bold Quest
Whether it’s a game of “Madden NFL” or a fantasy quest in “World of WarCraft,” most players want the superhero character: the fastest, strongest, “full-up” version. But when it comes to soldiers using video games for serious training, something surprising happens.
During the Bold Quest exercise, a multiweek international training exercise in October that tested both live and virtual components including TCM Gaming’s avatars, Rolfe found that “100 percent of the soldiers acknowledged that they would rather train with their own avatar than the standard, in-game avatar that’s actually better than they are.”
The Bold Quest scenarios at Fort Benning, Ga., included weapons qualifications, force-on-force exercises, and area reconnaissance and attack missions. The trainees went through the missions with the full-up and personalized avatars, some with the fatigue model and some without.
During these exercises, TCM Gaming also unlocked an unexpected way that virtual training could affect live practice: Students might work harder if they knew they would be using — and growing with — their own avatar.
“On the exit surveys, 90 percent of the soldiers said that they would be motivated to do better in live training performance if they knew that connectivity was there,” Rolfe said.
The element of positive peer pressure likely lies behind the desire to improve. No one wants to be the chunky avatar slowing the group down, and everyone wants to be regarded as the superhero avatar.
When soldiers were asked to identify members of their squad based on their avatar features, Rolfe said, “one soldier made the comment, ‘My avatar looks fat.’ The other soldier said, ‘Because you’re fat in real life.’” Similar comments emerged about shooting ability when soldiers missed targets in the simulation, an accurate reflection of their marksmanship score.
“It wasn’t caustic in nature,” Rolfe said. “It was just gentle soldier ribbing.”
Tying gaming to real life has motivated people to improve their physical fitness in online platforms such as Fitocracy. There, in addition to “leveling up” his or her character, individuals can complete and unlock quests and gain badges. Games and life are no longer relegated to separate circles — instead, they can influence one another more fluidly. Soldiers have games for training, but might find themselves training for games.
Matching avatars to real-life ability changes simulations by a surprising amount. According to Rolfe, using the full-up avatars rather than the personalized, “flawed” avatars increased average movement times by 46 percent.
With the fatigue model, they found soldiers needed double the ammunition to successfully take out their targets.
“It created leadership development skills, to the extent that the leaders had to compensate for the inadequacies of their soldiers,” Enloe said. “Leaders had to become better leaders and make sure that they got their mission completed.”
The decision-making changes, Rolfe said, when virtual decisions are based on the same characteristics that would exist in the real world. Movement time varies if someone has to rest.
“It’s the same as you would see in the mountains of Afghanistan,” he said.
Soldiers also started picking out their best marksmen and setting them up to cover likely avenues of enemy approach. Fire distribution plans and attack positions shifted to take advantage of the different skill levels, something that soldiers were never forced to consider when planning missions with full-up avatars. Things change when only your expert marksmen can shoot with that level of skill.
“If you’re not sniper-qualified, then you shouldn’t have the skills of a sniper in the simulation,” Rolfe said.
Knowing individual and collective skills — and limitations — would make it possible to rehearse certain scenarios more realistically. Leaders could also run through the best combination of people to have in a squad, or re-create past scenarios with different squad combinations to see how the event could have played out.
Changing skill levels at the individual level can also roll up into larger-scale simulation. For example, a crew in a simulation can fire at a certain level of proficiency based on tank gunnery scores. Better representing the skills of the individual will thus change how well the unit as a whole can fight.
“So we go from individuals, to crews, to teams, to companies, to battalions. It’s all feeding from the entity level,” Rolfe said.
More accurate avatars could also play off the strengths of VBS2, which Peppers said lends itself to collective task training at the small unit level, but increasingly up through company level and beyond.
“The focus is less about the individual avatar than it is about the unit as a whole having all of its members, through avatars, play out their combat tasks in a fairly immersive kind of way.”
The avatars shoot and move like their humans, and have the same basic shape and look, but TCM Gaming is not currently choosing to match faces as closely as it could. Photorealism is possible but too time-consuming to be practical.
Computers can take a side and front photo and map major points, such as eyes, nose and mouth, to a skeletal frame. But soldiers seem to recognize different squad members based on the customized data from the Digital Training Management System. Adding truer facial features would be an added level of detail, but it’s not certain how much it would improve training.
“It may be a consideration down the road,” Rolfe said. “There are some trade-offs. For instance, if you’re in full battle gear in the game or in real life, after about 20 meters I can no longer determine who you are based on facial features anyhow.”
While TCM Gaming quickly put together avatars that reflect general soldier characteristics, researchers at the Institute for Creative Technologies (ICT) at the University of Southern California have been working on the photorealistic elements. Their Light Stage, a giant sphere of lights used for targeted illumination and special polarization, has been used to create photorealistic face reproductions that can bridge the uncanny valley.
Their setup, a result of more than a decade of research and development, can collect a data set that can be reworked in different scenarios to match the lighting and conditions — helpful for a realistic-looking avatar in a game such as VBS2. It also develops transitions between facial expressions.
ICT Executive Director Randy Hill says that all of the research indicates that people identify more with virtual humans who are realistic.
“If you’re going to be embodied by an avatar in a virtual environment, if it looks like you, you’re going to have more of an identity with it,” Hill said. “It’s going to feel like it’s really you.”
Lt. Gen. Robert Brown, current head of I Corps and former commander of the Maneuver Center of Excellence at Fort Benning, asked TCM Gaming to determine whether more realistic avatars might improve training. Earlier this year, Brown visited ICT and had his own face captured to personalize his avatar. His hope is that face capture will fit in with a push toward virtual replicas as an aid to motivation and training.
“My common sense tells me the more realistic, the better,” Brown said of photorealistic faces. “But there needs to be more research on that.”
Abhijeet Ghosh, a senior researcher at ICT’s graphics lab who believes military simulations will achieve this photorealistic look down the road, said that the Light Stage could be used to create an avatar with a similar look: “a digital asset that can mimic the real actor.”
Hill said his team wants to make a portable version of Light Stage that is both inexpensive and quick. The device could then be set up at training depots to rapidly capture faces and expressions for avatars. First, the researchers would have to find a way to automate more of the procedure and reduce the processing time.
ICT currently captures a variety of expressions, such as stressed, angry, happy and sad. But if avatars are used for more than force-on-force exercises in VBS2 in the future, trainees might find themselves making more faces at the camera.
“I think it’s going to be important as we train more and more soldiers on cross-cultural interactions to have friendly-looking expressions,” Hill said.
Brown hopes that the avatars will find a home beyond the force-on-force interactions and move into the cultural training realm. By giving trainees surveys and measuring how well they interact with people or notice the surroundings, avatars could even replicate leadership style and more accurately represent how efficient someone is at getting information or advanced situational awareness.
ICT is also conducting a research project looking at why and how people respond to virtual humans, particularly ones that are photoreal. While ICT tackles avatar potential in the long-term, Rolfe said TCM Gaming’s goal was to get out a quick proof of principle at low cost. Within three months of the project’s start in February, they had their first modification ready.
While the personalized avatars are currently linked to VBS2, Enloe envisions a future where the data is platform-agnostic; your avatar could go to any simulation you want. Such an avatar could grow, reflect changing statistics, and be independent of the game engine or type of simulation running.
“My vision of the future is: You are issued an avatar, and that capability is the same across any simulation that you use,” Enloe said. “Your same characteristics come with you logging in.”
Brown imagines a service member getting an avatar as early as the recruitment stage. Rolfe suggested avatar data might become a part of the common access card.
The data itself that feeds into personalized avatars can also continue to grow. The Digital Training Management System can be modified, so additional fields could be added (or restricted) as necessary. However, Brown recognized the challenge of getting various systems to talk. DTMS was designed to capture statistics, not to update an avatar, he said.
With the growing number of mobile applications that can be used to track fitness or even used for military training, keeping all of this data up to date might also involve linking personal, portable devices to a data management system. Already, it is easy to track biorhythms and workouts, Enloe said.
“The technology is there. It’s just getting all the infrastructure together,” he said. “But that would be in the realm of the possible in the future.”
As with many of the Army’s data challenges, one concern is making sure that personal or sensitive information is contained and protected during storage and transfer.
“You’re not going to be doing anything on there that’s in the secret realm, but you don’t want everybody and their brother to have access to someone’s weapons scores that doesn’t need to,” Brown said.
In the near future, TCM Gaming will report on the Bold Quest findings to military leaders, then possibly shift the avatars to the Integrated Training Environment, which involves live training, constructive gaming and virtual simulations. There, it can undergo further testing as the ITE rolls out to various Army posts over the next four years.
Brown estimated that the realistic avatars would not be ready for a couple of years because of the stovepiped nature of the sim world and hesitation from older members of the military who don’t see gaming as a way to train.
Peppers said that mission training complexes, such as the one he oversees at Joint Base Lewis-McChord, stand ready to implement the new avatars into training if and when they arrive.
“I think we’ll start to see some early vestiges in the next couple of years, maybe sooner than that. But today, we do what we can with what we have in front of us right now.”