Digital Results Group's Ageon ISR tool integrates video and intelligence from multiple sources. Subjects of interest can be tagged and tracked. (Digital Results Group)
Investigators analyzed video from several sources, including security cameras and spectators' cellphones, to identify suspects in the Boston Marathon bombings. / FBI via Getty Images
Desperate for clues in the wake of the Boston bombings, law enforcement officials asked the public to deluge them with videos and cellphone photos taken near the Boston Marathon finish line.
“We would like to review any kind of media that you have out there,” Gene Marquez of the Bureau of Alcohol, Tobacco, Firearms and Explosives told a press conference the day after the attacks.
It didn’t matter, the authorities said, whether it was obviously suspicious or not. Anything might contain useful evidence.
Images immediately began pouring in, from race participants, from bystanders, from various security cameras at stores and businesses. All told, the flood epitomized “big data”: a wash of uncorrelated images and information.
The FBI isn’t saying exactly how it combed through hours of surveillance video and all the public’s digital media to find the suspects. Efficient forensic and intelligence work, however, would require more then just special agents staring at images on screens; it would need visualization tools to display all this in geographic and temporal context.
And on that point, industry officials say, the techniques used in Boston were probably far less advanced than they could have been.
Much of the cutting-edge video analytics, 3-D situational awareness, and tactical terrain visualization software available on the market today was developed for counterinsurgency missions in Afghanistan or Iraq, where success depended on making sense of video and intelligence from a variety of unrelated sources.
Even before the April bombings, the makers of such software — spurred in part by the U.S. drawdown and the need to find new markets — had been pitching their tools to domestic law-enforcement agencies. Their argument, delivered with mixed results, was that their software could make authorities a lot smarter in responding to threats, crises and emergencies in the U.S.
The Boston attacks and subsequent manhunt unfolded on the home turf of a small company called Cognika.
Formed in 2006 to sell analytic software developed at the Massachusetts Institute of Technology, Cognika today specializes in automated video searching software. After a user draws a box around an object of interest in a still image or video clip, Cognika’s algorithms race through hours of video looking for possible matches. Then the user can look through the much-reduced pile to confirm which clips actually match the object that was searched for.
“If you’re starting with 400 hours of video, someone has to look at that 400 hours of video,” said Cognika CEO Christian Connors. “With our tool, it would be down to maybe a half hour.”
Connors said in a case like Boston, the user could have clicked his mouse to crop around the hats or backpacks of the subjects. The search engine would then look for possible matches.
Cognika says it offered its software to the FBI free of charge after the bombings, but the bureau did not take the offer. And in the end, the bureau took just days to find its suspect on video.
Cognika says its technology isn’t just for forensics. Security officers can tag a suspicious object, such as car or truck, and follow it in near-real time from one surveillance camera to the next. The software works with high- or standard-definition video, but should not be confused with facial recognition software, he said. It’s meant to look for similar objects.
Tracking suspects isn’t the only way video clips might be used to create a common operating picture.
In cases where citizens have collected a lot of video and images stamped with metadata — time, location and even directional stamps — it might be possible for an operations center to stitch together a three-dimensional “virtual mockup” of an unfolding event to aid in an investigation or disaster response, said A.J. Clark, president of Thermopylae Sciences and Technology of Arlington, Va.
The virtual replay could be viewed with Thermopylae’s iSpatial software, which lets users add reports, intelligence and other data to Google Earth maps.
Depending on how much data is available, officials could create a 3-D rendering of a high-profile event. They could play through it forward, backward, and walk through the point of view of each image as they go along.
Clark has an interesting idea for how this concept could be used at a sporting event or other gathering: Invite attendees to download an app onto their phones to geo-reference and time-stamp their camera images, then stitch the pictures or videos together to create a 3-D rendering as a souvenir.
Would people be willing to participate?
“Half the people might do it and half the people might not,” Clark said.
MANY FEEDS, ONE PICTURE
Even if you can organize intelligence, images and video clips spatially and temporally, you still risk data overload.
“Pretty soon you’ve changed your clutter from verbal into visual clutter, and you still can’t separate the key elements you need — the information — from the data,” said Chuck Miller, owner of Priority 5 in Needham, Mass. “We set out to develop an application where you could isolate the key factors you needed to deal with, and make your whole view completely interactive.”
The result is the Touch Assisted Command and Control System. TACCS is designed with an open computing architecture to accept all sorts of information, from surveillance videos to smartphone video clips to public Twitter feeds, and field reports.
TACCS is part of the security setup at Port Fourchon, La., where oil from the Gulf of Mexico is fed into pipelines leading to U.S. refineries. Miller is now marketing TACCS for other domestic applications, including urban security and disaster response.
One of Priority 5’s marketing demonstrations has a VIP attending an event at the Verizon Center in Washington, D.C.
Suspicious vehicles are suddenly reported at several locations a few blocks from the venue. An alert pops up in red to say there’s a possible bomb threat. A second alert reports that this could be “about creating a distraction, and the actual target may be the VIP,” Miller said.
When human analysts aren’t on hand to spot a scenario like that, an automated alert analysis system runs in the background to connect the dots. This is done by comparing the alerts and entries arriving to TACCS with combinations of factors and events thought to be indicative of bomb threats, for example.
“What you can do is capture the wisdom, the logic of the analyst, and let the computer do the heavy lift,” Miller said.
Miller’s is hardly the only company working on the problem. Digital Results Group, another Boston company, also offers a visualization and analytic solution that helps make sense of different streams of data. It’s called Ageon ISR and it can integrate videos and intelligence from multiple sources so that they’re easily understood and analyzed. Subjects of interest can be tagged and tracked and followed.
The company has designed Web-based software so analysts can work on a browser, but the firm also offers a lightweight portable server to ingest and process video.
One component of an strategy that could enable better video analysis is high-tech visualization of the ground.
Decision makers in operations centers could be able to view dynamic maps overlaid with clickable icons showing video clips, police alerts, intelligence findings and a host of other information. They could keep the maps standard, or zoom into 3-D when the situation demands. Today, surveillance camera systems are often sold with software to display the camera locations on maps, and police often use terrain viewers such as Google Earth. But the ability to feed a variety of data to a map and create a dynamic common operating picture is not widespread.
The most sophisticated base maps are developed from aerial photography or, increasingly, the laser-based sensors called lidar. The U.S. Geological Survey has gathered lidar over about 30 percent of the country, including most of the Eastern Seaboard. (See the March issue of C4ISR Journal.) The texture and details of buildings can be digitally added to the raw lidar data and to the wireframe geometries of buildings created from aerial photos.
Still, advocates caution that the sophistication can be a trap.
“Sometimes a 3-D model does better than a map. Sometimes it doesn’t. Occasionally, people just get enamored with 3-D and it’s cool and high-tech but not really helping them,” said Ellery Chan, a founder of Precision Lightworks of Melbourne, Fla., which uses aerial photography to produce 3-D models of cities or facilities.
In general, 3-D models help authorities understand the ground. They make “a good sort of base reference map for all kinds of things, like surveillance cameras, viewing traffic patterns, figuring out where large amounts of population are, figuring out lines of sight from one place to another,” Chan said.
There may be a certain irony here: Will U.S. authorities domestically be able to adopt mapping technologies largely developed to help the military? And has military ISR developed a better knowledge of the ground in Afghanistan than authorities have of the United States?
All of this begs another issue: part of adopting the latest visualization, surveillance and mapping software means more cameras and more monitoring. And there’s an important ongoing political and philosophical debate about security versus privacy rights in the U.S. Advocates of adopting new technology emphasize they aren’t pushing for the government to adopt Orwellian surveillance strategies.
“These are questions of privacy, and I’m not solving those issues. I’m just stating the art of the possible,” said Priority 5’s Miller.
Ben Iannotta is the editor of Deep Dive Intelligence. (www.deepdiveintel.com)