If there is one mantra among the intelligence community’s full-motion, video enthusiasts it is this: While the aircraft and cameras that gather FMV and lower-rate motion imagery are extremely sophisticated, the process of cataloging and analyzing the resulting data is anything but. Analysts stare at images and tap notes into computers: Man enters courtyard. Man stands in courtyard. Man smokes cigarette.
“That’s probably not the most effective use of the high-caliber, talented individuals we put in Distributed Ground System sites,” said U.S. Air Force Brig. Gen. Scott Bethel in an April speech to industry experts shortly before his retirement.
Some Air Force intelligence architects would prefer to use computer software to handle the tedious job of cataloging the contents of full-motion video streams. Software might also enhance video or automatically alert analysts or operators to suspicious activity. That would free up analysts to take on more important tasks, such as identifying patterns and figuring out their significance.
That goal, which government and military intelligence officials have cited for years, could be within technical reach, industry experts said. The intelligence community is beginning to apply the video game industry’s graphics processing units, or GPUs, to the problem of managing the torrent of imagery. Contractors are installing GPUs in computers on the ground or embedded in aircraft, but wider adoption of automation will mean winning over skeptics in the military who have heard these promises before. On top of that, the need for automation is premised largely on predictions of rising demand for FMV and motion imagery even as the U.S. shifts to a supporting role in Afghanistan after 2014.
At least one intelligence agency’s projections on that score are less than solid. When pressed for details about its estimate of skyrocketing requirements for human analysts beyond 2014, the U.S. National Geospatial Intelligence Agency said its estimate was not based on “plans for future deployments.” NGA uses the unclassified chart on page 18 to make its case that automation can tame the demand to employ more and more analysts.
Bethel, for one, is confident of solid demand going forward. Any suggestion that today’s $30 billion to $40 billion fleet of unmanned planes will be parked in a hangar somewhere is “not very realistic,” he said. Even if the drones are not flown by U.S. Central Command, “eight other combatant commands are poised to consume whatever ISR is available, plus some,” he said, underscoring that he was expressing a personal opinion.
If the projections turn out to be right, part of the solution could spring from the realization that rendering complex graphics in response to a gamer’s decisions is a lot like analyzing, enhancing or retrieving a clip of a specific vehicle or person spotted in Afghanistan, for example. In the game market, Advanced Micro Devices Inc. and Nvidia Corp. have been competing to produce ever more sophisticated GPUs to drive ever more realistic games. When GPUs are paired with the more common CPUs, they can accelerate applications dramatically.
“Instead of being able to process a couple of megapixels in real time, we can process billions of pixels per second,” said Scott Thieret, technical director for Mercury Computer Systems Inc., a company based in Chelmsford, Mass., that develops digital image and signal processing subsystems for aircraft and ground vehicles.
For those who have to make sense of FMV, that additional processing power couldn’t come at a better time. FMV drawn from unmanned aircraft consumes the lion’s share of work at Air Force Distributed Common Ground System sites “and new wide-area motion imagery sensors now being deployed have the potential to vastly increase the amount of raw data collected,” according to a March report by the Rand Corp. “The information explosion resulting from these vast amounts of motion imagery threatens to leave Air Force intelligence analysts drowning in data.”
Commercial firms are banking that drones, like the Reaper aircraft equipped with Gorgon Stare wide-area surveillance pods, will continue to flood analysts with imagery. Those firms are scrambling to harness the power of GPUs in hardware and software for military and intelligence agencies. MotionDSP Inc. of Burlingame, Calif., for example, is working to improve the quality of images captured by unmanned aircraft and surveillance cameras. The company’s Ikena software uses GPUs to perform the computation-intensive process of reconstructing images frame by frame to remove noise and compression artifacts.
“We reconstruct the video so analysts don’t have to squint their eyes trying to see it,” said Sean Varah, MotionDSP chief executive.
That type of technology is particularly important to customers gathering full-motion video in places where fog, sandstorms and darkness often obscure the contents of imagery. Even under clear skies, cameras flying on distant unmanned aircraft can produce images that are hard to decipher. Image quality is then reduced further when U.S. agencies compress the videos to send them to U.S. bases for processing, Varah said.
MotionDSP’s software reconstructs each frame in a video with data drawn from other frames in the series. That job requires so much computing power, however, that when company officials first ran the original algorithm in 2006 it took more than a day to reconstruct a single frame of standard-definition video. GPUs have helped to speed up that process significantly. Customers can now use the Ikena software to process the video feed from a Predator unmanned aircraft as the data is received, Varah said.
That capability is far from common.
“Many companies understand the promise of using graphics cards for image reconstruction, but we are shipping products today,” Varah said. Those products have been sold to the U.S. Air Force and Navy, U.S. intelligence agencies, the U.S. Secret Service, the Naval Criminal Investigation Service and the London Metropolitan Police, Varah said.
Mercury Computer Systems engineers also have developed products that employ high-performance GPUs produced for the consumer market. For Mercury, the trick to using these chips in military systems is packaging them in a way that makes them rugged enough to withstand intense heat, cold and vibration so they can be used in aircraft. In the past, only about one-tenth of the video imagery gathered by aircraft cameras could be processed onboard due to stringent size, weight and power constraints, Thieret said. GPUs are helping Mercury push that number higher.
“There’s a particular algorithm we use for image reconstruction that we have implemented on a high performance Intel processor and on GPUs that we have embedded and deployed in theater today,” Thieret said. “Using the same, exact algorithm running on the same, exact data, the GPU is 200 times faster.”
What’s more, GPUs are extremely energy-efficient, providing a huge improvement in processing per watt. That increase in capability without additional size, weight or power has enabled Mercury to place additional processing power right next to onboard sensors for various systems, including the Gorgon Stare pods.
GPUs also are one ingredient in the latest video search tools being developed by Cognika Intelligence & Defense Solutions. Cognika uses a proprietary algorithm to index and classify information embedded in each frame of a video. “It’s like Google for video,” said Shashi Kant, chief scientific officer for the company based in Cambridge, Mass. The software automatically detects and tags objects, events and activities as the full-motion video is collected and stored. Searches can be performed by image, text or video clip, Kant said. The software can, for example, highlight scenes of people digging near roads or trucks making U-turns. It can also notify analysts whenever normal patterns change. If four people cross a specific road on most days, the software can alert analysts whenever more than 20 people cross that road, said Cognika President Christian Connors.
Much of Cognika’s software is easy to recognize for people who are used to using commercial computer applications. Search results are presented in a point-and-click format that resembles Google. Like Amazon, the software also suggests topics that might interest an analyst.
“It will say, ‘Although you are looking for a specific event, you might also want to consider looking at a similar event that was detected in the video,’Ÿ” Connors said.
The goal is to assist analysts who stare at multiple screens for hours at a time. Instead of spending 70 percent of their time staring at screens and 30 percent analyzing images to determine their significance, Cognika is trying to flip the equation, Connors said.
Cognika plans to offer an online version of the software through its website later this year that will allow users to upload videos, index them and conduct their own searches.
While all this technology might sound promising, a U.S. military officer reached in Afghanistan said this is not the first time he has heard companies promise to automate the job of monitoring full-motion video.
“I am always skeptical of company claims, and technology rarely meets our requirements or even the company’s own advertised capabilities,” the officer said.
To date, tests of video detection systems have been disappointing. Anything that causes the pixels to change in the camera’s view — shifting light, moving clouds, swaying trees, animals — produces false alarms.
“What I’m looking for is the company that can integrated and correlate video motion detection with our ground surveillance radars, cancelling out all the radar noise, eliminating nuisance alarms and detecting purposeful movement toward areas of concern or defended areas,” he said. “I’m in favor of automation, provided it performs as billed, and doesn’t come with hidden technical and sustainment costs that are more expensive than the manpower that might be offset to pay for it.”
Industry officials agree that some early efforts to automate video analysis fell far short of their goals. The task is extremely complex. It requires algorithms that can recognize individual elements of images. It’s not enough to simply match images because a car traveling in one direction looks different when it’s moving in a different direction or viewed from a different angle, said Michael Donaghey, Cognika vice president for sales and business development.
Image recognition also requires enormous processing power. That’s where GPUs can help. Semiconductor manufacturers produce new, increasingly capable GPUs every year.
“It’s Moore’s law on steroids,” Thieret said.
Obviously, no company wants to redesign systems annually to take advantage of that new capability. So Mercury developed software based on open standards. That way, it is compatible with each successive generation of GPUs and CPUs.
“As new processors become available, customers get an automatic boost in performance,” said Dinesh Jain, a Mercury senior product manager. “That means the primes who are developing this very sophisticated intellectual property are able to preserve that and actually see benefit as they move to next-generation technologies.”