Sensor and photographic images merged to offer a 3D view
For his graduate work at the State University of New York, Ye Duan explored computer vision, using a camera to record 2D images of stationary objects from different vantage points and using the digital files to construct 3D renderings.
When Duan accepted his position in the MU College of Engineering, one of his electrical engineering peers — Curt Davis, now director of the college’s Center for Geospatial Intelligence — reviewed his work and asked him if he could “do buildings.”
It turns out he can.
“My first grant from the Leonard Wood Institute (LWI) was to do aerial 3D models of buildings. But there were some limitations because of the lighting at different times of the day,” Duan said. “Then I heard about a new sensor called LIDAR (light detection and ranging).”
As Duan explained, the new technology pulses a laser onto a mirror that spins around very quickly while a computer records the images. Different materials will show up as different colors, and things like windows will appear black.
The LIDAR apparatus Duan utilizes in current 3D modeling research projects was purchased with funding from another LWI project in which he investigated fusing real-time video with LIDAR scans for “enhanced situational awareness” for the purpose of security surveillance. The methodology supports a wide variety of applications and Duan’s lab is exploring the possibilities.
The LIDAR device simultaneously does a 100-meter, 180-degree scan with accuracy of millimeters and at the same time records video.
“It inspires curiosity,” said Duan, who has worked on a variety of additional projects that utilize LIDAR.
“We worked on an NSF [National Science Foundation] highway grant with Rolla [Missouri University of Science and Technology]. I was working with geotech engineers to try and predict if rock is stable, using the scanner to scan the highway,” Duan said. “I will combine a geometric analysis with those images to see how the slope is impacted.”
The researcher explained that when doing video surveillance, it is often not convenient to use a large number of cameras. But if video can be fused with the 3D points in the scans, something can be viewed from a number of different angles. Such a marriage would produce what he calls multi-modality, dynamic urban scene modeling.
“We scan up and down 360 degrees, picking up points and registering their location,” said Brittany Morago,* a doctoral candidate working in Duan’s lab, explaining the process. “You can get almost a full surface but you don’t have data for every point. We find the camera location and project it onto the data, and through the use of algorithms, we get a 3D model.
“With videos, the frames are basically still images. With every 10th frame, I’m going to do what I do for a still image, but for the others there’s a little less computational work,” Morago said.
Giang Bui, another of Duan’s graduate students, said they he has tried several approaches, but this summer, the algorithm he programmed produced good results.
The product is an accurate, navigable 3D model.
“The device can scan one room, but you can’t see the rooms next to you, said Bui. “But if you scan the rooms next to it, you can put them together and have a complete picture of the entire building. We are joining the scans together to make a total picture.”
“For security purposes, you can reconstruct scenes to see if anything is different,” said Morago.
Many of the potential applications for the Duan’s research projects have connections to security. As such, he directs MU’s Center for Urban Safety, housed within the college’s Computer Science Department. Affiliated faculty researchers hail from a variety of departments.
“After a tornado, like in Joplin, it’s not safe to send people into buildings, but we can send a robot in to scan for structural damage,” said Duan, giving an example of the types of problems he is solving with LIDAR.
He has been working with faculty from both civil and mechanical engineering to submit a proposal that would make a 3D model of the downtown area of St. Louis, combined with structural analysis. The project’s goal would be to work ahead of a potential catastrophe to inform emergency responders which routes might be best for rescue and evacuation.
“This year is the two hundredth anniversary of the New Madrid earthquake. There was damage all the way from Memphis to St. Louis. There’s no way to prevent an earthquake, but you can mitigate the damage,” Duan said.
In addition, the LIDAR research holds potential for combat zone advantage.
“You could send an unmanned aerial vehicle ahead to scan a battlefield, and even leave it there throughout the conflict,” Duan said, adding that he already has used LIDAR in military training exercises at Leonard Wood.
“In the event of chemical, biological or nuclear weapon use where soldiers are wearing HAZMAT suits and carrying weapons, this technology can really help them,” he said.
In addition to practical and security uses, the process that Duan and his team are developing also has some lighter applications, such as the review of a controversial play in a sporting event.
“If a bunch of people captured the same scene, you could reconstruct it and find the differences,” said Morago.
Duan said the technology also could be used to construct mixed reality scenes, adding that his students really liked that part of the research.
“We could create a scene with an animated tiger running around the MU campus. You could use your IPhone to follow pictures of Truman and hold it up to the columns and view the scene with Academic Hall still in place,” Duan said.
Morago said that she sees great potential for its use in the entertainment industry, in both films and gaming, adding that it could also be useful in diagnostics.
“Doctors studying posture or movement in their patients would be able to record minute differences,” she said.
Duan and his team agree the work is challenging but enjoyable, and that the possibilities are many and exciting. And because of the breadth of research foci in the College of Engineering, they might get to explore the technology’s potential through collaborations.
“This is a very collegial school,” said Duan. “I’m glad I am here.”
Brittany Morago is one of two graduate students working in Duan’s lab who received NSF fellowships to pursue doctorate degrees. Morago chose to continue on as a member of Duan’s research team; Dmitriy Karpman is doing his graduate work at Stanford University.