Advertisement

Reviewing peer review

Peer review is on every life scientist?s mind lately, it seems. One of the main complaints I heard while researching the linkurl:February cover story;http://www.the-scientist.com/2006/2/1/26/1/ is that the process is inherently difficult to investigate scientifically. Each journal has a somewhat unique system for reviewing papers, and each paper will have a unique journey through a journal?s reviewing machinery. But I?ve learned that even though peer review has obvious imperfections, it?s the b

By | February 8, 2006

Peer review is on every life scientist?s mind lately, it seems. One of the main complaints I heard while researching the linkurl:February cover story;http://www.the-scientist.com/2006/2/1/26/1/ is that the process is inherently difficult to investigate scientifically. Each journal has a somewhat unique system for reviewing papers, and each paper will have a unique journey through a journal?s reviewing machinery. But I?ve learned that even though peer review has obvious imperfections, it?s the best system we?ve got, and simply complaining about what?s wrong with it doesn?t help matters much. Investigating peer review is difficult, sure, but does that mean we shouldn?t even try? Kirby Lee, an assistant professor of clinical pharmacy, and his colleagues at the University of California, San Francisco, have spent a year and a half collecting data about the peer review process at The Lancet, Annals of Internal Medicine, and BMJ. The data, also reported in Nature, stem from interviews with editors, peer review comments from more than 1,000 papers, and audio and video recordings of editorial meetings where editors discussed the papers. Lee told me that his team planned on spending six months collecting data, but had to significantly extend the enrollment period "because the acceptance rate was so low." In fact, out of all the papers included in the study, the journals published only 68. Some other interesting findings: -There appeared to be no bias towards statistically significant findings, although Lee cautioned that the same trend may not be true at other journals, or at the same journals during a different time period. -Major changes editors tend to make between acceptance and final publication include toning down authors? conclusions, and ensuring that the manuscripts disclose funding sources and conflicts of interest. -The methodological quality is higher in accepted papers than rejected ones. Lee said that the study was originally going to include JAMA, but the journal dropped out without providing a clear reason why. He and his colleagues decided to conduct the study to investigate complaints of publication bias, which can skew the research record and affect meta-analyses. They focused on biomedical publishing because what?s published can lead to changes in the way medicine is practiced. In the meantime, Lee and his colleagues continue to analyze the hours of tapes collected from editorial meetings, and have already submitted one paper describing their research, currently under review. "It?s funny," he said. "We?ve done this study on peer review, and now we?re at the mercy of peer reviewers."

Comments

October 30, 2006

The Center for Regulatory Effectiveness www.TheCRE.com has revised the Wikipedia entry on peer review to discuss and document peer review failure at JAMA. The revised entry exposes a politically biased Journal of the American Medical Association (JAMA) article [JAMA, May 24, 2006; 295(20): 2407 - 2410] on the Data Quality Act and atrazine that contains numerous factual errors and misrepresentations. A crucial, obvious error in the article was the assertion that atrazine is being phased out by the European Union because ?Atrazine...has been repeatedly demonstrated to be a potent endocrine disruptor....? JAMA?s peer review process accepted this claim even though the Official Journal of the European Union explicitly stated ?In the 70s, a political decision was taken to reduce to ?zero? the presence of pesticides, independent of their toxicity.? [Emphasis added]\n\nThe CRE Wikipedia revisions also include a discussion of the peer requirements imposed on federal regulatory agencies by the Office of Management and Budget. Federal agencies cannot use or rely on scientific information that does not meet the OMB peer review requirements. Many peer reviewed journals do not meet the OMB peer review requirements.\n\nFor more information about the failed peer review at JAMA, please contact William G. Kelly, Jr.. For more information about the US government?s peer review requirements, please contact Scott Slaughter.\n\n
image: Image of the Day: Drowsy Dragon

Image of the Day: Drowsy Dragon

By

A central bearded dragon (Pogona vitticeps) takes a nap. Researchers have discovered the reptiles' sleep patterns resemble those of mammals and birds.

image: Embryo Watch

Embryo Watch

By

A new culture system allows researchers to track the development of human embryos in vitro for nearly two weeks. 

image: Fat Labs

Fat Labs

By

The tendency of Labrador retrievers to overeat is tied to their genes, researchers report.

Advertisement

Popular Now

  1. The Zombie Literature
    Features The Zombie Literature

    Retractions are on the rise. But reams of flawed research papers persist in the scientific literature. Is it time to change the way papers are published?

  2. First Data from Anti-Aging Gene Therapy
  3. Locating Language within the Brain
  4. A Scrambled Mess
    Features A Scrambled Mess

    Why do so many human eggs have the wrong number of chromosomes?

Advertisement
Diagenode
Diagenode
Advertisement
LabX
LabX
RayBioTech