Peer Review: Do Studies Prove Its Effectiveness?

The Fourth International Congress on Peer Review in Biomedical Publications, organized by JAMA-The Journal of the American Medical Association and the BMJ Publishing Group, meeting in Barcelona Sept. 14-16, featured three days of presentations of original research on peer review. Despite coming on the heels of the terrorist attacks in the United States, 275 of the 410 registered participants attended. Many scheduled speakers whose flights were canceled were able to E-mail their presentations, which were then delivered. In all, the event featured 39 of 44 scheduled presentations and 58 of 65 planned posters. Many of the 37 studies presented cover the same ground as other studies and other congresses: authorship, quality of reports of randomized studies and statistical review, the review process itself, communicating to readers, publication bias, and ethical and confidentiality issues. Many reports concentrated on the better known journals (true for 44 percent of the studies) and focused on process, not outcome measures. The congress was organized such that a 10-minute discussion followed each 10-minute presentation. This arrangement, along with the decision to have no concurrent sessions, facilitated discussions. Much of the discussion centered on the need for disclosure of financial interest or other conflicts of interest, and of reviewers' names. The two invited presentations covered issues always high on the list of peer review discussions: Fiona Godlee, BioMed Central, made a plea for open review (authors and reviewers are identified). In the other, Douglas Altman, British Medical Journal, discussed statistical review and noted that it is only partially successful in ferreting out studies with methodological and statistical problems. Four studies (first authors: Christine Laine, Lydia Vos, Susan van Rooyen, Graham Mowatt) confirmed that many authors have not met authorship criteria as defined in the Uniform Requirements for Manuscripts Submitted to Biomedical Journals.1 According to the guidelines, "authorship credit should be based only on 1) substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published". Six studies (first authors: Richard Horton, Karin Huwiler, Peter Gøtzsche, Phil Alderson, Yen-Hong Kuo, Veronica Yank) identified problems with the design, description, and interpretation of published studies. One study (Douglas Altman) confirmed, not surprisingly, that statistics in published reports are better if there is a statistician involved from the design stage of a study; another (Phil Anderson) found some evidence that peer review works to filter out the worst of the papers; while another (Tom Jefferson) confirmed the lack of scientific evidence supporting the effectiveness of peer review. Three studies (two studies by Carin Olson and one by Tammy Clifford) found mixed results when trying to verify the tendency to publish studies with positive outcomes, a phenomenon called 'publication bias.' Vittorio Demicheli presented evidence that the quality of published and unpublished economic systematic reviews should be improved. A couple of unexpected findings directly related to peer review: feedback to reviewers from editors did not improve peer review (Michael Callaham), and reviewers do not want acknowledgement (Michael Berkwits). There were studies of topics not covered in previous congresses. To date, electronic, indexed journals are small and lack the complexity of traditional print journals (Ann Weller). Pharmaceutical advertisers rarely use graphics, but when present, graphics are of a low quality (Richelle Cooper). Articles with high reprint requests are frequently cited (Sally Hopewell). Review articles in 'throwaway' journals have methodological problems, but they do a better job at communicating their messages by graphics and readability than reviews in some well-known medical journals (Paula Rochon). One-half of the criticisms in published letters were not addressed by the authors of research articles when replying to those letters (Richard Horton). Two topics that often receive attention in studies of peer review were absent. None of the 37 presentations evaluated reviewer agreement, a trend I support, since reviewer agreement studies are full of problems (from defining what constitutes reviewer agreement to the tendency of editors purposely to pick reviewers of different backgrounds to obtain a balanced view). Studies of anonymity and blind review were also absent. These studies raise issues related to the difficulty of blinding authors and reviewers. However, there is no solid evidence to suggest that open review works better than blind review. In theory open review would eliminate reviewers who have a competing interest and disclosure would identify authors who have a financial interest in the outcome of a study. In a recent systematic review2 of 1,500 articles and studies on peer review published prior to 1998, I concluded that "the underlying strength of editorial peer review is the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published, and conversely, to assure that invalid, nonvaluable works are not published." Peer review's "outstanding weakness is that errors of judgment, either unintentional or intentional, are sometimes made. ... [There is a] lack of guidelines and standards. Editors have great flexibility in their implementation of the process in their journals ... [and] there is not one solid, accepted definition of what constitutes peer review," other than the basic definition of evaluation of a work by one's peers. Statistical problems of published studies have remained one of the most puzzling aspects of the review process. The first study that identified statistical problems in published articles in 1951 that found 73 percent of a group of controlled studies had problems with the control group itself.3 This and quite a number of similar studies since then have led many medical journals to add regular statistical review to its editorial process. The Cochrane Collaboration, "an international network of health care professionals, researchers, and consumers who ... update critical reviews of evidence from randomized controlled studies and other controlled interventions"4 as part of each review, examines merits of each study design and analysis. Given the scope of the Cochrane Collaboration, it is not surprising that they were quite visible at the congress and that six (16 percent) studies came from them. The effort to improve the publication process is driven by editors and others interested in the publication of valid, scientific information. This agenda is not imposed by a policing group, rather it is a continuation of a process begun by a small group of editors of general medical journals who met in Vancouver, B.C. in 1978. This group first established guidelines for the format of manuscripts submitted to their journals and then quickly expanded their scope to include issues about the quality of biomedical publications and to develop some recommended guidelines for the publication process.1 Editors and those interested in medical publication continue to look seriously at how they go about their business. They have identified areas for study to either verify or question current practices. Similar to the findings in the systematic review of peer review by this writer and previous congresses before this one, studies providing unquestionable evidence of the value of peer review remain elusive. Most studies were relatively small, often of only one journal or a small group of journals. So similar studies of different journals might have different results and reach different conclusions. These studies presented at the most recent congress will themselves be subjected to the peer review process and those that are accepted will be published in a special issue of JAMA. Papers presented at the congress needed a clearly stated methodology, results, and conclusions section. This process should help to address some of the methodological shortcomings identified in many of the studies discussed in my analysis of the peer review process. This structured format for the presentations is a welcomed requirement, undoubtedly designed to emphasize the need for rigorous scientific standards so often missing from the study of peer review.2 Given the human dilemma when making judgments of colleagues' works, editorial peer review is a process that will never be totally codified, verified, or universally accepted, nor is it likely to receive one final, definitive stamp of approval (or disapproval). But the continued study and discussion of editorial peer review are absolutely desirable and essential to meet the goal of publishing the most accurate and the best medical findings. Ann C. Weller (acw@uic.edu) is deputy director, Library of the Health Sciences, University Library, University of Illinois at Chicago. References 1. Uniform Requirements for Manuscripts Submitted to Biomedical Journals. jama.ama-assn.org/info/auinst_req. 2. A.C. Weller, Editorial peer review: its strengths and weaknesses. American Society for Information Science and Technology monograph series. Medford, N.J.: Information Today, 2001. 3. O.B.J. Ross, "Use of controls in medical research, JAMA, 145[2]:72-7, 1951. 4. About the Cochrane Collaboration: hivinsite.ucsf.edu/cochrane/.

Peer Review: Do Studies Prove Its Effectiveness?

Interested in reading more?

Become a Member of