Researchers See A Wealth Of Applications For Virtual Reality

X-ray crystallographer Vivian Cody has found a way to sit--virtually--in the midst of enzyme drug binding sites. She especially likes to experience the feel of it all--the pushes and pulls that a drug goes through as it finds the coziest place to rest on a protein. Although this kind of molecular space exploration sounds like fun and games, some scientists in academia, industry, and government see it as a most serious and advanced application of "virtual reality," an emerging computer technology that its most enthusiastic advocates suggest could change the entire complexion of research. While virtual reality has been used by the military and by space scientists for the last decade, pharmacologists, molecular biologists, and theoretical physicists are beginning to venture into its domain. Simply speaking, the technology provides heightened representation of the real, physical attributes of scientific models. It is a tool that takes visualization and interpretation of data to a new dimension--to the point at which a researcher, in a sense, touches, interacts with, or is engulfed by the model that he or she has created. If Alice Through the Looking Glass were written today, the heroine might have used a head-mounted display to go through to the other side, instead of a mirror. Inside head-mounted displays, explains Walter Robinett, director of the head-mounted display project at the University of North Carolina, "are two 3-inch- square liquid crystal display television screens. In front of them are some lenses so that the image portrayed on the television screen doesn't seem two inches in front of the viewer, but 10 feet off. It creates the illusion that you are in a life-size, 3-D world. Each screen has a different view so you see a stereoscopic, wide-angle view of this computer-simulated place." "On the head-mounted display," Robinett goes on, "is a device called a Polhemus that measures your head position and can detect where your eyes are looking in space. This tells the computer database what coordinates the vertices of polygons creating the virtual world should occupy. It generates an image for each of your eyes 30 times per second, so when your head moves, what you are seeing continually changes, just like what happens when you turn your head from side to side. "The net effect is that you open your eyes and look through this head-mounted display, and you see this 3-D scene. You can push your head forward and walk into the scene. The stuff in front of you gets closer. If you turn your head, the stuff that was in front goes off to the side." Such head-mounted displays were conceived in 1965 and created by Ivan Sutherland at the University of Utah in 1968. Researchers are working on developing software and hardware so a user can see images from the real world while operating in the virtual world. --R.E. Unlike the usual interfaces between humans and computers, such as the mouse or the keyboard, virtual reality hardware--head-mounted displays and gloves--allows people to feel as if they are entering the world inside a computer display, flying around and manipulating the objects they encounter (see accompanying story). Indeed, it promises to accelerate scientific understanding by enabling researchers to refine their hypothesized models of the mysteries of nature. "Science moves by its ability to see and model," says Myron Krueger, one of the innovators of the technology. "Although we downplay our intuitive grasp of things and think mathematical equations are more correct, our intuition is more highly developed than our symbolic sense of the world. New perceptions and perspectives create radical sources for discovery and insight in science." Other scientists agree. Crystallographer Cody, head of drug design at the Medical Research Foundation of Buffalo, is one of the small but growing number of academic, basic science researchers--from biologists to physicists--who have begun to use or investigate the use of VR in their work. Cody and other noncomputer, nonspace, and nonmilitary scientists are being introduced to VR through collaborations with private and public facilities specializing in the technology. Cody felt the power of VR technology last year at one of the centers doing research on it in the United States, the department of computer science at the University of North Carolina, Chapel Hill. As editor of the Journal of Molecular Graphics, Cody says, "I often get to be the first kid on the block to play with new toys like [these]." And because the facility is considered a special resource center, funded by the National Institutes of Health, part of the $300,000 a year it gets must go toward collaborative work. Cody went to the center to see if the university's prototypic VR force-feedback, drug-docking program could reproduce results she gets with the current standard computer software and hardware. She left with a lot more. She found new binding orientations for one of the drugs she works with--the enzyme dihydrofolate reductase (DHFR), a major chemotherapy target of anti-cancer agents in cells. Now she intends to determine if what she saw in the virtual world can exist in the real world by crystallizing the drugs and the enzymes in these new configurations. Although the center has energy data available only for DHFR, the technology represents a major advance over what is currently available, Cody and others say. "The fastest way to do all the energy measurements between a drug and a substrate is to use a supercomputer," says Cody. But since the supercomputer calculates every energy possibility between the atoms, it takes a long time to do the analysis and is costly. VR puts a human being into the loop to both visualize and feel the interaction between the molecules. "The mind is still the best pattern recognition system available," says Cody. What the supercomputer wastes time on doing by rote, a person can quickly do by using sight and touch, she says. Although the technology promises to be revolutionary, "there is still a trade-off between resolution and speed of transmission of images," says Michael McGreevy, head of the National Aeronautics and Space Administration's Virtual Environment Research Laboratory at Ames Research Center in South San Francisco, Calif. At the current level of commercially available virtual reality, for example, a biologist could develop a highly stylized model of a cell and poke around inside. But the technology at the centers and on the market has not been refined to the point at which a biologist can enter into an image of a living cell and watch the chromosomes divide. Not yet, that is. The virtual landscape is expected to become more refined in five years. There are some 50 VR startups, a number of major corporations, about five centers in the U.S., and more in Japan doing research on VR, according to William Bricken, a scientist at the Human Interface Technology Laboratory in Seattle, one such center. But these firms and institutions aren't focusing on refining the technology for bench researchers. Most of the recent applications of virtual reality have been in rendering 3-D architectural drawings, landing astronauts on Venusian landscapes, medical imaging, and telecommunications. Academic scientists make up a small percentage of VR's potential user market and therefore don't provide a substantial economic incentive for product development. But according to NIH computer specialist Richard Feldmann and others, designing affordable, mass-market consumer products, such as virtual video games or virtual telecommunication systems, are driving the VR marketplace. In such applications, people would be able to "walk" into a computer game, or two people separated by distance could "occupy" a computer-generated space and interact in it. Spin-offs from these consumer products and advances in supercomputers should eventually decrease the cost of VR software and hardware, say computer experts. And then scientists can benefit. "Scientists are parasites in that sense," says Feldmann, of NIH's Division of Computer Research and Technology. In five to 10 years, say the experts, virtual experimentation on a wider scale could become a reality. The computer graphics and virtual reality center at North Carolina, under principal investigator Fred Brooks, has been evolving since the early 1970s. Years of support from NIH, the Office of Naval Research, the university, and Burroughs Wellcome Co. of Research Triangle Park, N.C., have finally begun to pay off. "The [force-feedback] drug docking project allows a researcher to feel in his or her hands the electrostatic forces that exist between a drug and its substrate--in real time. No small task," says Richard DuBois, director of the computer technology section of biomedical research in NIH's National Center for Research Resources in Bethesda, Md. To dock drugs at the North Carolina center, the researcher stands in front of a 4' x 3' screen, upon which is displayed a model of a drug molecule and its target protein. By using a robotic arm that acts as a joystick, he or she can move the drug up and down, left and right, and backward and forward. An energy histogram displayed on the screen allows the investigator to eyeball whether the fit between the drug and its substrate is energetically stable. At the same time, the robotic arm responds to repulsive forces between the molecules depicted on the screen. Other projects at the center, including the drug-docking program without force feedback, involve one of the more popular interfaces of VR technology, called the head-mounted display. Force-feedback drug docking using head-mounted displays is not possible yet. "The technology is just not here," says Walter Robinett, chief of the head-mounted display project. But he and others are working on it. Head-mounted displays are the window into the virtual world. They create a stereoscopic, wide-angle view of the virtual world that the wearer can seemingly enter. Inside the displays are two small televisions, one for each eye. The view comes from a computer that generates images 15 to 30 times per second. The picture changes depending on the wearer's movements. An input device allows a person to be transported inside the virtual world. These devices enable a person to make commands that the computer interprets. The computer then changes the image that it sends to the television screens in the head-mounted display. The input device is a joystick that has been transformed into a glove that a person wears on the hand or a joyball with buttons. The glove was originally made by computer scientist Tom Zimmerman because he wanted to play "air guitar"--actually producing the sound of strings with a flick of the wrist. He licensed the glove to Jaron Lanier, CEO of VPL Research Inc. in Redwood City, Calif., one of the 50 or so VR startups. Wide acceptance and application of VR technology isn't going to happen overnight. While there are some commercial VR software and hardware products on the market, the machines are costly, and only the centers today have enough computing power and expertise to create a believable, three-dimensional visual reality in which a researcher can interact. Before VR can be used more widely, observers agree, a considerable number of technical and financial hurdles must be overcome. One criticism of virtual reality as it exists today is that the essential equipment is uncomfortable and unreliable. The head-mounted displays, for instance, are said to nauseate people. "The problem with the glove is that it is made of stretchy material, and it is hard to get it to work from day to day with different sized hands," says Robinett, who uses the ball device. "But from the point of view of seeing a live hand in an environment and seeing the scale of things inside a virtual world in proportion to your own body, the glove is wonderful." An alternative to virtual reality, called artificial reality, overcomes some of the equipment unfriendliness. Invented by Myron Krueger in 1969, it is a system that allows a human being to interact in computer events unencumbered by gloves and head-mounted displays. In Krueger's artificial reality, video cameras send images of a person to a computer, which translates his or her movements and responds to them. One of the fastest-growing manufacturers in the U.S. of gloves and other virtual reality hardware and software is Redwood City's VPL Research. The firm was founded in 1985 by its president, Jean-Jacques Grimaud, and CEO Lanier, who has become a media celebrity of sorts, having been profiled in Playboy, Rolling Stone, and Smithsonian. In June 1989, VPL performed a tour de force--the first commercial demonstration of two people in separate locations sharing a virtual reality experience. Today VPL's customers include telecommunications firms, engineering designers, and forestry researchers. And while VPL employees acknowledge that the technology is still in its early stages, customers say that it is worth getting in on the ground floor to gain an edge on what they predict will become a widely used technology in business. "Before we started selling our products, the only people who could work in this field had to have all of the skills and resources to build the system from the ground up," says Anne Lasko, director of product design at VPL. "They needed an optical engineer to develop the eye phones and an electrical engineer to figure out the tracking. The system we sell is ready to go, so someone can make their own virtual reality. "But there are misconceptions about virtual reality. On the one hand, people have greater expectations of the technology than is currently available. On the other hand, many people underestimate the power of the technology that does exist. It is a very creative software. It is up to the user to see its application to their system." VPL sells a virtual reality system for one person that includes a head-mounted display, the glove, the two advanced graphic workstations, and accessories for about $200,000. The two workstations account for $150,000 of the price. The glove alone goes for $8,800. Most academic researchers would be hard-pressed to find money from their university or from a federal agency to use VR in their lab, some observers say. "If a lone researcher, independent of the groups doing this sort of thing, put in a grant to the government even to do collaborative VR, it would be impossible to get it peer-reviewed," says NIH computer specialist Feldmann. "There just isn't enough of a base of scientists with expertise in applications of this technology to new areas. "Peer review is a control mechanism that is somewhat repressive and reactionary. But once you get some brave souls who show the technology isn't just whiz-bang, but can produce new and interesting results, you might get some funding. This has happened before with other technologies that people originally thought were just toys." Right now, in fact, the 18-month-old Human Interface Technology (HIT) Laboratory in Seattle is in the process of convincing the National Science Foundation to help fund its program. With Tom Furness at its helm, getting money should be no problem, say colleagues. Furness used to head the Dayton, Ohio-based Wright Paterson Air Force Base VR flight simulation project, a VR application used to train pilots. "HIT is a state-funded institution whose purpose is to transfer technology from the states' universities to the private sector," says Bob Jacobson, associate director of the center, based at the University of Washington. HIT is also soliciting $50,000-a-year memberships in a consortium program from industry investors. The pooled money from the members of the consortium supports general research in the lab. In return, industry members gain an advantage over competitors, since they have access to the initial stages of the VR technology. Current members of the consortium include Digital Equipment Corp. of Maynard, Mass., Sun Microsystems of Mountain View, Calif., and VPL Research. For those academic researchers who want to ride the VR train but cannot afford it, collaboration with industry may be the ticket. John Joannopoulus, a theoretical physicist at the Massachusetts Institute of Technology in Cambridge, has such an arrangement with IBM's Veridical Laboratory in Yorktown Heights, N.Y. Hoping eventually to use VR as a visualization tool, Joannopoulus and colleagues are in the process of developing a machine that could simulate the structure of solids. This machine would create three-dimensional, electron-density maps of solids upon being provided only with the atomic number of an element. "Now I do the calculations on the supercomputers at MIT and Carnegie Mellon University," he says. "Office of Naval Research money pays for them. The visualizations of the solid structures I do at IBM." In the near future, he wants to be able to enter into the structure of a solid and see what an electron sees with VR. By disturbing the structure of a pure solid with a contaminant, he wants to see the effect from inside the solid. But with all the money in the world, Joshua Zimmerberg still couldn't do what he wants to in virtual reality. Zimmerberg, chief of the Laboratory of Theoretical and Physical Biology at the National Institute of Child Health and Human Development, wanted to take his two-dimensional fluorescent microscope images, translate them into a three-dimensional volume, and look at a cell as a whole, instead of sectionally. "We would need a supercomputer that doesn't exist yet to allow me to see the cell in 3-D through a head-mounted display," says Zimmerberg, who studies the biophysics of membrane fusion. "At the current time, VR is only good for people who work with computer-generated images. But to do the kind of work I and others want to do is a good reason to push supercomputer development." Maintaining U.S. competitiveness in supercomputers and virtual reality prompted Sen. Albert Gore (D-Tenn.) to visit North Carolina this past January. Gore is concerned that the Japanese will exploit the VR market before the U.S. His concern may be warranted. According to HIT's Jacobson, the Japanese have 10 times as many facilities researching VR as the U.S. NIH's DuBois is concerned as well. He says, "If we want to stay at the forefront of this field, we must be committed to keeping our technical base so we aren't scooped by the Japanese."

Researchers See A Wealth Of Applications For Virtual Reality

Interested in reading more?

Become a Member of