Experiment Design Software: Better Data, Less Work

Abacus Concepts 1918 Bonita Ave. Berkeley, Calif. 94704 (510) 540-1949 Fax: (510) 540-0260 Advanced Systems Management 1200 S. Acadian Suite 110 Baton Rouge, La. 70806 (504) 343-7151 Fax: (504) 383-3048 BBN Software Products Corp. 150 Cambridge Park Dr. Cambridge, Mass. 02140 (617) 873-5000 Fax: (617) 873-4020 BMDP Statistical Software Inc. 1440 Sepulveda Blvd. Suite 316 Los Angeles, Calif. 90025 (310) 479-7799 Fax: (310) 312-0161 Dionex Corp. P.O. Box 3603 1228 Titan Way Sunnyvale, Calif. 94088-3603 (408) 737-0700 Fax: (408) 739-4398 DSP Development Corp. One Kendall Square Building 100 Cambridge, Mass. 02139 (617) 577-1133 Fax: (617) 577-8211 Exeter Software 100 N. Country Rd. Setauket, N.Y. 11733 (516) 689-7838 Fax: (516) 689-0103 Europa Scientific Software Corp. 14 Clinton Dr. Hollis, N.H. 03049-6595 (603) 595-7415 Fax: (603) 889-2168 Galactic Industries Corp. 395 Main St. Salem, N.H. 03079 (800) 862-6004 Fax: (603) 898-6228 Graphpad Software 10855 Sorrento Valley Rd. Suite 203 San Diego, Calif. 92121 (619) 457-3909 Fax: (619) 457-8141 GW Instruments Inc. 35 Medford St. Somerville, Mass. 02143 (617) 625-4096 Fax: (617) 625-1322 HEM Data Corp. 17336 Twelve Mile Rd. Suite 200 Southfield, Mich. 48076 (313) 559-5607 Fax: (313) 559-8008 Idea Works 607 Jackson St. Columbia, Mo. 65203 (314) 875-5827 Fax: (314) 875-5812 Infometrix 2200 Sixth Ave. Denny Bldg., Suite 833 Seattle, Wash. 98121 (206) 441-4696 Fax: (206) 441-0841 Jandel Scientific 2591 Kerner Blvd. San Rafael, Calif. 94901 (415) 453-6700 Fax: (415) 453-7782 Keithly Metrabyte 440 Myles Standish Blvd. Taunton, Mass. 02780 (508) 880-3000 Fax: (508) 880-0179 Labtronics Inc. 95 Crimea St. Unit 2-C Guelph, Ontario N1H 2Y5 Canada (519) 767-1061 Fax: (519) 836-4431 Manugistics 2115 E. Jefferson St. Rockville, Md. 20852 (301) 984-5000 Fax: (301) 984-5094 MathSoft Inc. 210 Broadway Cambridge, Mass. 02139-1901 (617) 577-1017 Fax: (617) 577-8829 MicroCal Software Inc. 1 Roundhouse Plaza Northampton, Mass. 01060 (413) 586-2013 Fax: (413) 585-0126 Millipore Corp. 80 Ashby Rd. Bedford, Mass. 01730 (617) 275-9200 Fax: (617) 275-5550 Minitab 3081 Enterprise Dr. State College, Pa. 16801 (814) 238-3280 Fax: (814) 238-4383 Nicolet Instrument 5225 Verona Rd. Madison, Wis. 53711-9075 (608) 271-3333 Fax: (608) 276-6222 Omega Engineering Inc. 1 Omega Dr. Box 4047 Stamford, Conn. 06907 (203) 359-1660 Fax: (203) 359-7640 Research Systems Inc. 2995 Wilderness Place Suite 203 Boulder, Colo. 80301 (303) 786-9900 Fax: (303) 786-9909 SAS Institute Inc. SAS Campus Dr. Cary, N.C. 27513 (919) 677-8000 Fax: (919) 677-8123 Signal Technology 104 W. Anapamuma Dr. Santa Barbara, Calif. 93116-1950 (800) 235-5787 Fax: (805) 968-2620 Stat-Ease Inc. 2021 East Hennepin Ave. Minneapolis, Minn. (612) 378-9449 Fax: (612) 378-2152 Statistical Graphics Corp. Five Independence Way Princeton, N.J. 08540 (609) 924-9374 Fax: (609) 452-7792 Statistical Sciences 1700 West Lake Ave. North Suite 500 Seattle, Wash. 98109 (206) 283-8802 Fax: (206) 283-8691 StatSoft 2325 E. 13th St. Tulsa, Okla. 74104 (918) 583-4149 Fax: (918) 583-4376 Systat 1800 Sherman Ave. Evanston, Ill. 60201 (708) 864-5670 Fax: (708) 492-3567 The Math Works Inc. 24 Prime Parkway Natick, Mass. 01760 (508) 653-1415 Fax: (508) 653-2997 Trinity Software P.O. Box 960 Main Street Campton, N.H. 03223 (603) 726-4641 Fax: (603) 726-3781 Visual Numerics Inc. 9990 Richmond Ave. Suite 400 Houston, Texas 77042 (713) 954-6496 Fax: (713) 781-9260 Waterloo Maple Software 450 Phillips St. Waterloo, Ontario N2L 5J2 Canada (519) 747-2372 Fax: (519) 747-5284 Wolfram Research Inc. 100 Trade Center Dr. Champaign, Ill. 61820-7237 (217) 398-0700 Fax: (217) 398-0747 With the cost of research continuing to climb, life scientists increasingly are looking beyond their own laboratories for ways to balance their budgets. From their colleagues in industrial research, many are hearing now about the cost-saving potential of an approach to their investigations called design of experiments--DOE for short-- and about the software tools available to automate DOE. DOE is a way of using statistical principles to identify what will be the crucial variables in an experiment before the hands-on work begins. By knowing these variables in advance, a scientist can create an experiment that will need fewer runs to complete and will later require less complex statistical analyses. Even though the experiment and analysis are simpler, if the DOE approach has worked correctly, the results yield more useful information than traditional, ad hoc approaches to experimentation. "All experiments are designed experiments," says Kinley Larntz, chairman of the applied statistics department at the University of Minnesota, St. Paul. "Some of them are just designed very badly." The principles of DOE can be applied without computer assistance, but for large and complex experiments, applying DOE without a computer can be too time-consuming to be practical, scientists say. In such cases, it is wise to turn to software that automates the DOE process (see accompanying story). The ideas behind DOE, at first glance, appear to be nothing more than good science. And, to a degree, this is true. "DOE is really just the application of statistical principles for data collection," Larntz says. Another, more formal definition of DOE is provided by Doug Montgomery, a professor of engineering at Arizona State University in Tempe and author of the textbook Design and Analysis of Experiments (New York, John Wiley & Sons, 1991): "A designed experiment introduces systematic changes into the variables of a system with the objective of discovering which variables are responsible for the observed changes in the output." But when people talk about applying DOE techniques to their research, it's usually with the goal of obtaining the most informative data from the minimum number of trials. "The method is aimed at small experiments where resources are expensive," explains Larntz. "The goal of small, expensive experiments with multiple factors is that every data point you collect is useful. DOE lets you set up the experiment so that you get a good amount of information from every point." Development of the principles behind DOE dates back to the 1920s. Software that automates the designed-experiment method goes back to the early days of mainframes. "DOE software systematically looks at all the experimental factors and levels and picks, out of all the possible combinations, a small subset," says Larntz. "By working with this subset, you get predictive abilities from your analysis to help you predict optimal conditions. Those optimal conditions may not be conditions that you ever actually test, and that's the real beauty of the approach. DOE holds a tremendous potential for scientists." In the last 10 years or so, as with other software, DOE programs have migrated from mainframes to personal computers, becoming available now to most scientists. "Frankly, anyone who does DOE manually today needs to have his head examined," says Montgomery. "If you do it manually, you have to understand more about the theory of statistical design and more of the operational aspects of the statistical methods. If you use a computer program, you only have to understand the basic concepts of experimental design. The software leads you through the rest." DOE software uses one of two approaches to lead the researcher through experiment design. One approach is a question-and-answer dialogue in which the program asks certain questions and the user supplies the answers. The other approach involves a menu structure. The user selects an item from a main menu, which leads to another, more specific menu, which leads to another, and so on until the researcher has provided all the required input. Montgomery prefers the menu approach. "I personally don't like the question-and-answer approach very well," he says. "When you ask someone how many experiments he wants to run, the answer is `zero.' A question-and-answer approach often closes off some of the attractive alternatives. I prefer choosing items from a well-structured, narrow menu, then having other menus pop up and guide you as to what to do. You don't want to allow a sort of a global decision, like how many runs you want, to be made too early in the game." DOE software is available from the larger, well-known statistics software companies such as SAS Institute Inc. of Cary, N.C., and BBN Software Products of Cambridge, Mass., as well as smaller companies that have a DOE package as their sole product. DOE software from the larger vendors is usually supplied as a module that works in conjunction with the company's other statistics products. For example, SAS's DOE offering is called SAS/QC, and it can be integrated with the company's broader SAS System for statistics. Similarly, BBN's RS/Discover works with the company's RS/Series of statistical software. Stand-alone DOE software is available from companies such as Stat-Ease Inc. of Minneapolis, which makes the Design-Ease and Design-Expert packages. The advantage of stand-alone, personal-computer-based DOE programs is an affordable price (Design-Ease costs $395; Design-Expert sells for $795) and the fact that it's possible to buy just the DOE capabilities needed without getting a much larger, more comprehensive statistics system. The advantage of going with a vendor such as SAS or BBN is getting a powerful system geared for a large enterprise. These vendors' products run on a variety of hardware platforms and also support networking, client/server computing, and other state-of-the-art computing functions. Minnesota's Larntz believes that there should be more factorial studies, and consequently more DOE, in the biological sciences. However, as he says, this idea "hasn't taken off very well" yet. One of the ideas behind DOE is that it helps the researcher control variability, which, according to Larntz, could make DOE particularly useful in life sciences. "There is lots of variation in the biological sciences, often more variation relative to response than in physical sciences," Larntz says. "Because of that variation, biological research requires larger studies, but the balanced factorial approach of DOE would work well in this arena." Larntz is currently applying DOE principles to a large- scale, community-based clinical trial of various treatments for opportunistic diseases related to HIV infection. It is what he calls a pragmatic design involving real-world patients and primary-care physicians. Typically, a study such as this would investigate one treatment per patient. Larntz's approach will permit multiple treatments on the patients and investigate the effect of different treatments in combination with each other. "That's the idea of a factorial design," he says. "We're trying to set up an experiment where we are looking at many things simultaneously, possibly the interactions between the drugs. These are DOE ideas." However, while DOE software is available to Larntz, he is not using it for this study. "It's such a very large experiment that designed-experiment principles will prevail without need for any specialized software," he says. "The typical engineering study has 16 runs. This study has 5,000 patients. There wouldn't be a big gain from optimization with DOE software." There are several reasons that some industrial research areas picked up on DOE before others and before academia. For one, the nature of some industrial research lends itself perfectly to DOE. "In the chemical industry, for example, it is relatively easy to manipulate factors such as time, temperature, and concentration," says Mark Anderson, vice president and general manager at Stat-Ease. "DOE is ideal for finding optimum formulations, and its payoffs in the chemical industry are tremendous." Second, the highly competitive nature of some industries forced them to turn to an approach--such as DOE--that might give them an edge. This is particularly true of the semiconductor industry. "They got forced into using these tools and techniques probably earlier than anybody here in the United States because it was a matter of survival," says Montgomery. "They had to really figure out how to understand their products and their processes and how to make them better, faster, and cheaper." A third reason is the nature of some academic research, especially that involving the life sciences. "The difference between a life science application and an engineering-based industrial application," explains Minnesota's Larntz, "is that in a typical large clinical trial, for example, you do not have many different treatments. It's commonly a comparison of drug A vs. drug B. In a typical engineering application, you have time, pressure, temperature, catalysts, and concentrations--lots of factors. You can take advantage of DOE to manage those large factorial designs." Another characteristic of much life sciences research is that the studies are quite large. "It's fairly clear how to design these according to statistical principles without a computer," Larntz says. A fourth reason for the relatively minor use of DOE techniques and DOE software in academic research compared with industry is simply that many academic researchers have not received training in this area. In industry, DOE often falls under the realm of quality control, and with the huge push for better quality control in the U.S., many industrial researchers are being exposed to DOE. It is possible to go through a typical Ph.D. program in many basic science disciplines and not be exposed to these principles. Robert Launsby of Launsby Consulting, Colorado Springs, Colo., tells the story of one client who had gotten a Ph.D. in organic chemistry and then gone on to start his own biotechnology company. When he realized the potential DOE held for his company, he called in Launsby's firm--which specializes in suggesting ways for industrial clients to use DOE principles in their experiments--to implement the approach. After seeing its success, he thought back on his dissertation process and said wistfully, "You know, I wish I'd known about these techniques before I started my dissertation. I would have done it in half the time and would have gotten as good or better results." For scientists faced with a large, complex research investigation, computer programs that bring design-of- experiment (DOE) ideas to bear on the task can save considerable time, materials, and aggravation. Consider the situation faced by Robert Launsby of Launsby Consulting, Colorado Springs, Colo., on a job he did for a biomedical-industry client. Launsby's firm specializes in suggesting ways for industrial clients to use DOE principles to streamline their experiments. In this case, the task was to find the best buffering solution for a particular, expensive enzyme. The challenge was to create, from a list of hundreds of possible buffers, the most effective solution for that enzyme. Relying on experience, the company's scientists were able to initially narrow the list of buffers. But since a buffer is placed into solution--of sodium chloride; magnesium; ethylenediaminetetraacetic acid (EDTA); ethylene glycol-bis- N, N '-tetraacetic acid (EGTA); or the like--the experiment would have to investigate enzyme activity not only in the presence of the different buffers, but also with each buffer in a different solution. The experimenters would have to consider buffer A in sodium chloride, magnesium, EDTA, or EGTA, then buffer B in sodium chloride, magnesium, EDTA, or EGTA, and so on. And since solutions can be formulated in different concentrations, the experiment threatened to become even more complex--buffer A in sodium chloride at a low concentration would have to be tested, for example, and then buffer A in sodium chloride again, but at a higher concentration. Potentially, the client's scientific team was confronted with a very unwieldy experiment. "They were making a reagent for a glucose monitoring device and studying enzyme activity," Launsby explains. "They ended up with something like 96 possible buffer-solution combinations to investigate." The company hired Launsby to apply the principles of DOE to this problem. In the past four years, Launsby Consulting has helped set up close to 500 experiments using the DOE approach. In this case, Launsby used a program called RS/Discover from BBN Software Products, Cambridge, Mass., to implement DOE. The software identified, from the 96 possible combinations of buffer, solution, and concentration supplied by the scientists, 12 that would provide sufficient data to determine the best buffering solution for that particular enzyme. "We could have applied DOE principles by hand and gotten to the same 12 combinations, but it's very labor-intensive," Launsby says. "With DOE software, you supply some information--your variables, your levels, your responses-- and the program leads you through the selection of the best design to meet your needs. The result is an experiment with the minimum number of runs, as opposed to having to run all the combinations." --C.D.P. Caren D. Potter is a freelance science writer based in McKinleyville, Calif.

Experiment Design Software: Better Data, Less Work

Interested in reading more?

Become a Member of