Eye popping from MTV to the lab

When linkurl:Song Zhang,;http://www.me.iastate.edu/who-we-are/people-and-offices/faculty-directory/zhang-song.html a mechanical engineer at Iowa State University, was approached by U2's people last spring asking if the band could use his novel 3D imaging technology for a concert performance, he immediately sent an email to his lab asking: "Does anyone know who U2 is and whether or not we should spend time on them?" Zhang had a similarly hard time two years ago deciding if collaborating with alternative rock band Radiohead on a music video for their single "House of Cards" was worth his time. "I don't know music at all," he's quick to admit. It was only when colleagues told him that the band was in fact "a very, very big deal" that he agreed. He's probably remained impervious to the allure of pop culture because Zhang has been very busy for the past decade developing a way to acquire high resolution 3D video in real time. Using a projector that shines a light pattern on a subject, along with a camera that records at a different angle, he is able to reconstruct the 3D image -- down to the last nook, wrinkle, and crease -- in real time, using software he developed. This means that, unlike the creatures of __Avatar__ and __Lord of the Rings__, which required months of labor-intensive animation and motion capture data from real subjects, 3D versions of anyone who steps in front of Zhang's system spring to life instantly. "There's no other system that can really match it," says software developer and Zhang's graduate student, linkurl:Nik Karpinsky.;http://nikkarpinsky.com/ While being wooed by music's A-listers and Hollywood giants (he was approached by Weta Digital, the visual effects company behind __Lord of the Rings__ and __Avatar__) is fun for Zhang, seeing what his technology can do in the hands of biomedical researchers is what really gets him excited. In fact, he started working on this system as a PhD student at Stony Brook University back in 2001 with a grant from the NIH to devise a way to capture the movement of patients' torsos as they inhale and exhale in 3D. The end goal was to develop a noninvasive lung cancer diagnostic tool. "The idea is that people with some kind of lung problem have different chest motions," Zhang explains. Since then, it has also been used to quantify facial movements in patients with facial paralysis. Recently, Zhang teamed up with linkurl:Igor Efimov,;http://efimov.wustl.edu/ a cardiac physiologist at Washington University in St. Louis, to do live imaging of beating hearts. They are trying to synch Zhang's system, which records the mechanical contractions of the heart, to Efimov's fluorescent imaging method, which maps the heart's electric potentials. "This gives us a really unique opportunity to study cardiac physiology in health and disease," Efimov says. But to capture the beating hearts of small mammals such as rabbits and mice, Zhang must speed up the image acquisition rate several orders of magnitude from the usual 30 frames per second. So far, they've achieved a speed of 667 frames per second -- enough to capture a rabbit's beating heart with great detail. But a mouse's heart will require at least twice that speed. It is a feat that will take some time and energy to accomplish, but one that might open the gates to even faster acquisition speeds, allowing for more intricate imaging. "We could do potentially 10,000 frames per second," Zhang says. It's no wonder then, that after all the effort spent in making 3D measurements as precise as possible, Zhang's team was puzzled by the highly-stylized form it took in Radiohead's "House of Cards" music video. It was an equally strange affair for the video's director, linkurl:James Frost,;http://www.zoofilm.net/zoo.html who has directed music videos for other bands such as Coldplay and OK Go and had to make do without traditional video cameras. Instead, he waved patterned sheets of glass between lead singer Thom Yorke and Zhang's camera in order to distort the data and make Yorke appear as though he's slowly evaporating. "A lot of engineers found it quite amusing that we were using this in a very crude format as opposed to what it was really designed for," Frost says with a laugh. Zhang was one of them. "We couldn't understand why they wanted to mess with our system," he says. But he was pleasantly surprised by the final product. "We are engineers," he adds. "We have no idea how art works." Despite the success of Radiohead's video (it earned a Grammy nomination), the system is not quite ready for use by movie directors and video game developers because of the mountains of data it generates. "For one minute of video we have eight and a half gigabytes of data," Karpinsky explains. It is simply too overwhelming to produce a feature length film. Karpinsky has recently been able to compresses a minute of video to around 500 megabytes. "We're going to be able to compress it more and more until the actual file sizes are similar to that of a 2D movie." Then, he says, they can set it loose in Hollywood. But cashing in on the 3D film craze is not in the forefront of Zhang's mind. He envisions his technology pushing the frontiers of medicine by allowing, for example, top surgeons anywhere in the world to use it to perform remote surgery. N. Karpinsky and S. Zhang, "High-resolution, real-time 3D imaging with fringe analysis," Real Time Image Processing, doi: 10.1007/s11554-010-0167-4, 2010. R.P. Mehta et al., "Novel 3-D video for quantification of facial movement," Otolaryngol Head Neck Surg, 138:468-72, 2008. **__Related stories:__***linkurl:2009 Top 10 Innovations;http://www.the-scientist.com/2009/12/1/41/1/ [December 2009]*linkurl:An Integrated Approach to Fluorescence Imaging;http://www.the-scientist.com/article/display/13351/ [28th October 2002]*linkurl:Imaging Cells in Four Dimensions;http://www.the-scientist.com/article/display/12885/ [18th February 2002]

Eye popping from MTV to the lab

Interested in reading more?

Become a Member of