Efforts Increase To Boost Validity Of Meta-Analyses

Meta-analysis, which uses systematic methods to search, combine, and evaluate research literature, has been praised for making the review article more scientific and objective. But studies showing that journals are much more likely to publish the results of positive over negative clinical trials have led some to question the validity and objectivity of meta-analyses. Many in the medical community are concerned that this publication bias may skew the results of meta-analyses, which increasingly are used by doctors in devising recommendations for patient treatment. In hopes of reducing this publication bias, some critics are pushing for a registry of unreported trials. But citing problems like the poor experimental design of many unpublished studies, other critics question if the registry can improve the validity of meta-analyses. REGISTRY REJECTION: New England Journal of Medicine's Jerome Kassirer is concerned about the quality of unreviewed research. "The potential for publication bias concerns us because physicians now will search systematic reviews to determine the best treatment for patients," Richard M. Glass, a deputy editor at JAMA-Journal of the American Medical Association, says. "If positive research results get published more, then the risk of adopting ineffective and even harmful medical practices is greater." Concerned about that risk, more than 100 journal editors who attended last September's International Congress of Biomedical Peer Review and Global Communications in Prague have joined in encouraging researchers to register unreported clinical trials. Following the conference, the British Medical Journal (BMJ) began a registry of clinical trials that have not been published or have been reported only as abstracts. Ian Roberts, director of the Child Health Monitoring Unit of the Institute for Child Health at the University of London, reports that more than 6,000 trials have been registered through October 1997. A strong supporter of the registry, Roberts became aware of the threat of publication bias in the course of his own efforts to produce a systematic review evaluating outcomes for brain and spinal cord injuries. In a BMJ editorial, Roberts, along with BMJ editor Richard Smith, argued that the exclusion of unpublished trials from systematic reviews reduces the ability of reviewers to "detect moderate, but clinically important treatments" (R. Smith, I. Roberts, BMJ, 7109[315]:622, 1997). Furthermore, they held that more frequently reported positive results of trials could lead to overestimates of a treatment's effectiveness, leaving patients open to ineffective and even dangerous treatments. Endorsements from high-profile medical journals like JAMA and the Annals of Internal Medicine have given the registration movement a big boost. Glass explains that journals that directly or indirectly influence medical practice cannot afford to ignore the problem of publication bias. "Systematic reviews have become more important to us and the medical community because [they appear] to be an effective way to render massive amounts of research usable by physicians," Glass says. "Without addressing the threat of publication bias, you really cannot claim any gains in making medicine more sensitive to scientific evidence because you are not presenting a fair picture of research." The New England Journal of Medicine (NEJM) has distinguished itself from other major medical journals with its refusal to endorse the call for universal registration. Jerome P. Kassirer, editor-in-chief of NEJM, says that universal registration simply opens up a vast reserve of unreviewed research for use in systematic reviews. "NEJM hasn't jumped on the . . . bandwagon because we believe unpublished trials have questionable value for researchers," Kassirer explains. "Most of these unpublished studies have not been reviewed for quality and soundness of method. A registry may reduce the potential for publication bias, but use of poor-quality, unpublished studies casts just another shadow of doubt on meta-analysis." Despite the existence of publication bias (K. Dickersin, AIDS Education and Prevention, 9[Supplement 1]:15-21, 1997), meta-analyses have been seen by some as a more objective and sound alternative to the traditional narrative review. Such traditional reviews have been criticized as impressionistic and scientifically unsound because they lack standards for selecting, evaluating, and synthesizing research articles, according to Kay Dickersin, an epidemiologist at Johns Hopkins University and director of the Baltimore center of the Cochrane Collaboration, an international organization aiming to improve health-care quality and delivery through the use of systematic reviews. "In the narrative review, authors will rarely explain their method," Dickersin says. "With the systematic review, you make that process transparent and less subjective. You are really bringing the high standards of science to process of review. When thought about this way," she adds, the use of systematic reviews seems "overdue." While most researchers have no objections to qualitative meta-analyses, the reception of those that statistically combine studies to arrive at a single estimate of a treatment's effectiveness has been mixed (S.P. Hoffert, The Scientist, Sept. 15, 1997, page 1). Meta-analyses, which statistically combine multiple studies to arrive at a single estimate of the efficacy of a treatment, have caught the most fire from critics. This statistical method has divided the research community, as some have seen meta-analysis as an alternative to large clinical trials. In a 1994 article, Cynthia Mulrow, a professor at the University of Texas Health Science Center in San Antonio, recounted a landmark meta-analysis that suggested enough evidence had existed since 1971 to show that streptokinase treatment for coronary heart disease was effective (C. Mulrow, BMJ, 309:597-9, 1994). Controversy erupted when a large clinical trial investigating magnesium treatment for myocardial infarction contradicted an earlier meta-analysis of small, mixed-quality trials showing a positive treatment effect in controlling heart attacks (GISSI-3, Lancet, 343:1115-22, 1994). Some critics say the contradictory results show that "several medium-sized trials of high quality seem necessary to render results [of meta-analyses of small trials] trustworthy" (M.A. Klebanoff et al., JAMA, 277:376, 1997). Other critics attribute the opposing results in the meta-analysis to publication bias and the existence of unpublished trials showing a negative treatment (M. Hunt, How Science Takes Stock: The Story of Meta-Analysis, New York, Russell Sage Foundation, 1997). Although Dickersin believes registries of trials could reduce the potential for disagreement between large trials and meta-analyses, she does not think it will eliminate publication bias. "These kinds of systematic reviews could receive the greatest benefit from the clinical trials registry," Dickersin says. "But you have to ask yourself if the existence of a registry would entirely eliminate contradictory large clinical trials and meta-analyses. The answer is clearly 'no' because many other factors can stand in the way of the validity of meta-analysis." In one earlier study, John C. Bailar III, chairman of the department of health studies at the University of Chicago, found rampant carelessness in a random selection of published meta-analyses (J.C. Bailar III, Journal of Clinical Epidemiology, 48[1]:149-57, 1995). He found one meta-analysis that compared and combined results from unrelated studies. The review appeared in the first volume of the Oxford University Press Effective Care in Pregnancy and Childbirth (T.C. Chalmers et al., eds., 1989). In it, the authors examined the effect of diethylstilbestrol (DES) on the outcome of pregnancies. Bailar found the one study included in the meta-analysis did not deal with DES, but rather a synthetic estrogen of an entirely different chemical structure. Bailar examined another meta-analysis that considered the relationship between chlorinated water and cancer. The study reported increased cancer risk from exposure to chlorinated water for all body sites. Bailar says such a conclusion, which may be statistically valid, shows a lack of understanding of cancer on the part of the meta-analysts. "To date, laboratory experiments show there is no chemical agent which can induce cancer in every organ in the body," Bailar explains. "Mistakes like these show that any quality meta-analysis requires expertise in the subject matter reviewed. Statistical wizardry just isn't enough." AFTER LIFE: Jacques LeLorier notes that many studies that would be rejected by a journal get a second life in meta-analyses. Similar concerns over the quality of studies included in systematic reviews prompted Jacques LeLorier, a professor of medicine at the University of Montreal, to compare meta-analyses and randomized clinical trials that examined similar questions. The Canadian research team led by LeLorier decided to make this comparison after discovering that studies included in meta-analyses often suffer from serious design flaws. "The flaws we found in these articles probably prevented them from being published in more reputable medical journals," LeLorier says. "But when included in a meta-analysis, they suddenly could appear in influential journals like JAMA. The problem with meta-analysis is that it gives rise to this gross double standard in publishing. Many of these studies wouldn't see the light of day if not for their appearance in meta-analyses." LeLorier's research team identified 19 meta-analyses and 12 subsequent randomized clinical trials that addressed the same questions (J. LeLorier, NEJM, 337:536-542, 1997). They found that the meta-analyses would have led to the adoption of ineffective treatments in 32 percent of cases and to the rejection of effective treatments in 33 percent of cases. But LeLorier notes that these discrepancies tend to magnify divergence between the trials and meta-analyses. "In most cases, the results between a trial and a meta-analysis were not statistically significant," LeLorier explains. "To be sure, this difference becomes significant when you consider physicians are not interested in statistical differences, but only if a particular study supports or rejects a treatment. " LeLorier's study also found that subsequent clinical trials offering negative results for a treatment were preceded by a positive meta-analysis in just under half of all cases. "The situation can be explained by the existence of publication bias," LeLorier says. "If you rely on published articles, as most meta-analyses do, then positive findings will be magnified. This is one problem where the trial registry could have a positive effect." RESEARCH-DRIVEN: The Journal of Evidence-Based Medicine frequently publishes abstracts of significant systematic reviews. Efforts to minimize the effects of publication bias have intensified with the rise in use of meta-analyses in forming clinical recommendations. Citing a lack of time for physicians to keep abreast of current research on new treatments, David L. Sackett, editor-in-chief of the Journal of Evidence-Based Medicine and a professor at the National Health Service Research and Development Center for Evidence-Based Medicine in Oxford, England, says meta-analysis has shown enormous potential for easing and speeding the transfer of research in medical practice (D.L. Sackett et al., BMJ, 312:71-2, 1996). Editors at JAMA also see an increasing need for meta-analyses given the rise of evidence-based medicine, a movement that encourages physicians to rely more on current research and systematic reviews in making clinical decisions. "There is a tremendous amount of research out there waiting to be seized on by physicians," Glass says. "JAMA has supported the registry because we would like to publish more systematic reviews. Meta-analysis is one way by which research can begin to have a greater influence on medicine." Other journal editors still view meta-analysis and evidence-based medicine with skepticism. Citing high review standards and editorial concerns, editors at NEJM do not foresee publishing more meta-analyses. "To even consider a meta-analysis for publication, we like to see one which provides summaries and evaluations of all articles appearing in the review," Kassirer says. "I think this quality standard has kept us from publishing many meta-analyses. From an editorial standpoint, we find that traditional reviews are more interesting for the reader. They have the flavor of an author rather than the dry, statistical style of meta-analysis." NEJM also has been reluctant to promote evidence-based medicine. "Quite frankly, we're still guarded about evidence-based medicine because the jury is still out," Kassirer says. "There hasn't been enough empirical evidence which would suggest that it produces better outcomes in medical treatment. We're also concerned that the generation of clinical recommendations from [meta-analyses] is a damaging simplification of medical care. Individual patient, ethical, personal, and a host of other considerations a doctor must make are not factored in such reviews." Bailar has similar concerns about the effect of systematic reviews on science. "[Meta-analyses] can be damaging to the process of science," he says. "Many researchers think meta-analysis will allow us to distill the truth from apparently inconclusive research. But in cases where a research question is not well defined or results are contradictory, you may not want to do a meta-analysis. The exploratory and impressionistic nature of the narrative review helps us reshape a question and begin to understand where our research diverges."

Efforts Increase To Boost Validity Of Meta-Analyses

Interested in reading more?

Become a Member of