Meta-Analysis Gaining Status In Science And Policymaking

Despite lingering debates, statistical method wins acceptance and exerts influence among researchers. Meta-analysis-a statistical method of quantitatively combining and synthesizing results from individual studies-has begun to exert an influence on science, its practice, and its use in policymaking, despite early opposition and skepticism. One recent example that has attracted considerable media attention concerns the use of "fen-phen" (fenfluramine/phentermine), the popularly prescribed obesity control drugs. A Mayo Clinic meta-analysis of 24 users with rare valvular disease alerted United States health officials to the potential risks of fen-phen in July. A health advisory issued by the Food and Drug Administration (FDA) on July 8 turned up 58 more cases of heart valve damage in users of the drug. FDA has requested warning labels on all prescriptions, and Wyeth-Ayerst Laboratories, a Philadelphia-based maker of fenfluramine, has announced sponsorship of a study of 1,200 users. Harris Cooper, a professor at the Center for Research in Social Behavior at the University of Missouri at Columbia and an early practitioner of the statistical method, affirms that meta- analysis is gaining acceptance within the scientific community. "When meta-analysis first appeared, it received the same kind of reception of any new technique in science. It was met with a great deal of skepticism. Over time, however, acceptance has grown, and it is one of many accepted procedures used in science. Now, it is hard to find anyone who objects to its proper use." NUMBERS BROKER: Morton Hunt hopes his book How Science Takes Stock will help popularize meta-analysis. Meta-analysis burst onto the scientific scene more than 30 years ago. The reception was often hostile as proponents of the statistical method put down traditional methods of research synthesis and literature review. The prominent British psychologist H.J. Eysenck derided meta-analysis as "an exercise in mega-silliness" (American Psychologist, 33:517, 1978), and a peer reviewer of Cooper wrote of his meta-analysis on sex differences in conformity, "I simply cannot imagine any great contribution to knowledge from combining statistical results." Morton Hunt, author of the recently published How Science Takes Stock: The Story of Meta- Analysis (New York, Russell Sage Foundation, 1997) believes that trends in publishing point to Cooper's claim. Hunt writes that in 1977, there were no meta-analyses published; by contrast, searches in five major databanks (ERIC, PsycINFO, Scisearch, SOCIAL SCISEARCH, and MEDLINE) revealed 3,444 published in 1994. Some of the most widely discussed applications of meta-analysis have been in medicine. But it also has weighed in heavily in debates in the social sciences and psychology. "In education and psychology, the practice is well over 20 years old, and its utility is established," Cooper explains. "People know when to use it and when not to use it. Meta-analysis has found its place as a weapon in an arsenal of techniques." While its widespread use indicates a level of maturity and acceptance, meta-analysis continues to face various controversies, which further define its standards and procedures. Moreover, leaders in science see the need for increased promotion and education in the methods of science if meta-analysis is to be influential in policymaking. Early in its development, meta-analysis was touted for its ability to effectively synthesize findings from several independently conducted trials into an overall conclusion. In a 1994 article, Cynthia Mulrow, a professor at the University of Texas Health Science Center, defended meta-analysis as not merely a valid method of systematic review but also "an efficient scientific technique" (C. Mulrow, British Medical Journal, 309:597-9, 1994). In the article, Mulrow recounted a landmark case that suggested meta-analytic combinations of small trials could be equivalent to the results of large trials. A cumulative meta-analysis, which continuously updates findings, demonstrated that streptokinase treatment for coronary heart disease was known to be effective since 1971. Mulrow wrote, "This cumulative type of review indicated that intravenous streptokinase could have been shown to be life saving . . . long before its submission to and approval by the [FDA] and its general adoption in practice." WEAPONS STOCKPILE: Harris Cooper believes that meta-analysis has found a place in the arsenal of science. According to Cooper, such a use of meta-analysis is often hotly contested. A recent exchange in this debate is found in JAMA-Journal of the American Medical Association, where critics dispute the claim that "in most cases, the results of large clinical trials will agree with the meta-analysis of the smaller trials on the same topic" (J.C. Cappelleri et al., JAMA, 276:1332-9, 1996; M.A. Klebanoff et al., JAMA, 277:376,1997; B.T. Johnson et al., JAMA, 277:377, 1997). Some critics, he notes, have seen meta-analysis promoted as making unnecessary, or at least minimizing, the need for large clinical trials. "Such an application in pharmaceutical studies has been debated because often sample populations for tests are simply too small to reach meaningful conclusions," explains Norman Bradburn, a professor in the departments of psychology and public policy at the University of Chicago. "Meta-analysis also has become important given the prevalence of views in bioethics which regard strict random assignments as unethical." Moreover, meta-analysis has been viewed by some as more cost-effective than large trials, he says. DOLLARWISE: Norman Bradburn notes that meta-analysis can point to where research funds are best spent. But Bradburn cautions that meta-analyses are not cost-effective insofar as they are cheaper to do. "A sound meta-analysis is expensive, time-consuming, and labor-intensive," he points out. "It is cost-effective from the point of view that it allows us to get the most out of previous studies, and it can point to where future research dollars are best spent." The view that a meta-analysis of small trials will agree with results of a single large trial has been challenged by recent studies. A large trial investigating the effectiveness of intravenous magnesium treatment for myocardial infarction contradicted earlier conclusions arrived at through a 1993 meta-analysis of seven trials, with 1,301 patients. Critics concluded that the results of this trial show that a meta-analysis based on only small trials should be distrusted and that at least "several medium-sized trials of high quality seem necessary to render results trustworthy" (M. Egger, G.D. Smith, Br. Med. J., 310:752-4, 1995). According to Hunt, critics generally believe that the meta-analysis incorrectly found a benefit from intravenous magnesium because the study was biased by the exclusion of unpublished reports that found "no newsworthy effects." Recent studies in psychology and medicine show that research journals are much more likely to publish studies showing statistically significant results, Hunt writes in How Science Takes Stock. Such a publication bias is thought to lead to overestimates of effect sizes in meta-analyses using only published studies. Meta-analysts have been responsive to this critique. Cochrane Collaboration, an international effort to improve health-care quality and delivery through systematic review and meta-analysis, has moved to reduce the risk of publication bias. The collaboration, which found more than 30,000 published control trials given inappropriate keywords in MEDLINE, is working with the National Library of Medicine to build a complete international register of controlled trials. Currently, the collaboration has no official position on the use of small trials in meta-analysis. "Right now, we focus only on reviews of large clinical randomized trials because there is so much catching up to do in the area of systematic review," says John Cornell, a biostatistician at the Cochrane Center in San Antonio, Texas. FDA has adopted a strong position for its use of meta-analysis. Although it is frequently used in issuing warnings and treatment guidelines, large and randomized clinical trials are relied on and necessary for approval, according to FDA spokesman Don McLearn. While publication bias can taint findings, Cooper argues that this problem is not endemic to meta-analysis. "The statistical method places the problem of publication bias in bold relief," he remarks. "It is not the case that meta-analysis creates the problem. The bias exists for anyone engaged in the literature review. Where there is a need to review scientific literature, the sense is that meta-analysis gives greater credence to the review because it is an application of a scientific method to a formerly subjective pursuit." Resources: Kenneth G. Brown Inc. A meta-analysis consulting firm 4917 Erwin Rd. Durham, N.C. 27707 (919) 408-8067 E-mail: kbinc@mindspring.com Russell Sage Foundation 112 E. 64th St. New York, N.Y. 10021 (212) 750-6037 Fax: (212) 371-4761 http://epn.org/sage.html San Antonio Cochrane Center Audie L. Murphy Memorial Veterans Hospital 7400 Merton Blvd. San Antonio, Texas 78284 (210) 617-5190 Fax: (210) 617-5234 E-mail: cochrane@merece.uthscsa.edu. Suggested Reading: How Science Takes Stock: The Story of Meta-Analysis (M. Hunt, New York, Russell Sage Foundation, 1997) for a lucid overview. David S. Cordray's 'An Assessment from the Policy Perspective" in The Future of Meta-Analysis (K.W. Wachter, M.L. Straf, eds. New York, Russell Sage Foundation, 1990) for an insightful and in-depth treatment of technical and general issues in meta-analysis used in the public policy forum. The Handbook of Research Synthesis (H. Cooper, L.V. Hedges, eds., New York, Russell Sage Foundation, 1994) for a comprehensive treatment of the field. From its inception, meta-analysis, promoted as minimizing subjectivity in the review article, inspired interdisciplinary antagonism and rivalry. The late Thomas Chalmers, former president of Mount Sinai Hospital and a clinical trials specialist at Harvard School of Public Health, criticized the traditional review article as relying more on the credentials of the reviewer than on the evidence reported in studies. He boasted of its more recent use in medicine, "Meta-analysis is the wave of the future. The days of the expert supposedly putting the state of the field in the review article are numbered" (C. Mann, Science, 249:476-9, 1990). Steven Bayard, an Environmental Protection Agency biostatistician and the director of an influential 1993 EPA meta-analysis on secondhand smoke, explains that meta-analysis in its early development and use seemed to reduce the significance of the scientist in the review process and give prominence to the statistician. Meta-analysts were not seen as engaging in scientific debate, but rather as making enemies with the experts, according to Bayard. "Meta-analysis was more a tool of scientific adversity than controversy," Bayard says. E PLURIBUS UNUM: GAO statistician Kwai-Cheung Chan says meta-analysis can promote interdisciplinary cooperation. Kwai-Cheung Chan, a statistician at the General Accounting Office (GAO), dismisses charges and fears that meta-analysis transfers the power of review and synthesis in medicine and other fields to the statistician. Chan, the director of a 1996 GAO meta-analysis of cholesterol treatment in preventing coronary heart disease, argues, "My staff included people from a broad spectrum of fields related to the problem studied. I think meta-analysis done correctly encourages interdisciplinary cooperation." In preparing the 1996 report Cholesterol Treatment: A Review of Clinical Trials Evidence, Chan's GAO staff consulted with 21 outside individuals, including 12 prominent medical researchers. Ironically, the widespread availability of software in meta-analysis has brought on fears that some scientists, lacking training in meta-analytic techniques, will widely misuse the review method without the guidance of statisticians. "With the boom in the field, there are now lots of statistical programs which are simple and mindless to do," Cornell explains. "As long as you put numbers in, you get numbers out. There is a serious danger of misuse and abuse because they are so easy to do. Scientists and journal editors can get caught up in this trend and forget that the statistical techniques of the meta- analysis itself must be carefully evaluated." Despite the widespread use of meta-analysis across disciplines, it has been difficult to gauge its influence on public policy and debate, according to Hunt. In his book, Hunt relates success stories in which meta-analysis appears to have been influential in guiding public policy. In 1983, the Program Evaluation and Methodology Division (PEMD) of GAO was commissioned to review studies of the effectiveness of health and nutrition programs for low-income pregnant and postpartum mothers. Eleanor Chelimsky, the retired director of PEMD, approached the project with caution. The study was ordered by Sen. Jesse Helms (R-N.C.). Chelimsky feared that Helms had decided that the program was not effective, Hunt relates in his book. But the meta-analysis found that while most programs had an inconclusive impact on the health and well-being of infants, nutrition programs had moderately raised mean birth weights and brought about a decrease in the overall percentage of low birth weights. Surprisingly, the meta-analysis had caused a reversal in opinion and led Helms to scrap his plans to reduce or eliminate funding for the program, Chelimsky told Hunt. But Daryl Chubin, the acting assistant director of the White House Office of Science and Technology Policy, expresses a more guarded view on the legislative impact of meta-analysis. "While meta-analysis has become an important tool in science, it has not taken hold in the public mind," Chubin says. "Legislators do not see this tool as necessarily objective and capable of eliminating controversy. It may have great virtues from the perspective of science and statistics, but legislators and other public-minded people are often guided by other values." Rodney Nichols, CEO of the New York Academy of Sciences, sees the need for increased promotion and education. "Generally, meta-analysis is a powerful tool in science," he says. "But assuming a limited background in the methods of science, meta-analysis or any conventional bit of statistical analysis requires a great deal of explanation and elaboration to be persuasive in public debate." Cornell echoes this view, explaining, "Right now, the newest task for meta-analysis is to make the information and process widely accessible to practitioners and the public. The physician and the policymaker need to know what meta-analysis means, what it means to the practice, and what it means to the patient."

Meta-Analysis Gaining Status In Science And Policymaking

Interested in reading more?

Become a Member of