How to Fix Peer Review

Despite its importance as the ultimate gatekeeper of scientific publication and funding, peer review is known to engender bias, incompetence, excessive expense, ineffectiveness, and corruption.

David Kaplan(
Jun 5, 2005

Despite its importance as the ultimate gatekeeper of scientific publication and funding, peer review is known to engender bias, incompetence, excessive expense, ineffectiveness, and corruption. A surfeit of publications has documented the deficiencies of this system.1234 In September, the fifth in a series of international congresses concerned with how peer review can be improved will convene in Chicago. Yet so far, in spite of the teeth gnashing, nothing is being chewed.

Investigation of the peer-review system has failed to provide validation for its use.1 In one study, previously published articles were altered to disguise their origin and resubmitted to the journals that had originally published the manuscripts.5 Most of these altered papers were not recognized and were rejected on supposed "scientific grounds." Other investigators found that agreement among reviewers about whether specific manuscripts should be published was no greater than would be expected by chance alone.6

Peer review subsumes two functions. First, peer reviewers attempt to improve manuscripts by offering constructive criticisms about concrete elements such as the application of a technique, the strength of results, or the cogency of an argument. The second function of peer review is to render a decision about the biological significance of the findings so that the manuscript can be prioritized for publication. I propose reforming peer review so that the two functions are independent.

Review of a manuscript would be solicited from colleagues by the authors. The first task of these reviewers would be to identify revisions that could be made to improve the manuscript. Second, the reviewers would be responsible for writing an evaluation of the revised work. This assessment would be mostly concerned with the significance of the findings, and the reviewers would sign it.

After receiving the final assessments from several different reviewers, the authors could decide to submit to a journal, sending the manuscript and the signed reviews together. The editors, carrying out the second function of peer review, would then decide to publish or not based solely on this material. The reviewers' identities would be revealed in the publication.

I believe there would be several significant effects of this change in peer review. First, the authors would submit only positive assessments. Consequently, reviews would emphasize why a manuscript should be published instead of why it shouldn't be. Second, investigators would be less likely to publish insignificant findings. They would have to ask colleagues to put their names on the manuscript; consequently, the tendency would be to ask for support for more complete and more compelling sets of findings.

Third, reviewers would be forced to account for their comments. They could not perform just a cursory look without the authors realizing the review was not insightful and did not represent an honest effort. Fourth, although it would be possible to have close friends and relatives review a manuscript, the editors would see who was supporting publication. In their deliberations, the editors would consider the breadth of the reviewers and their relationships to the authors and to the conceptualization promulgated in the manuscript.

Fifth, the editors would be free from adjudicating between authors and reviewers. They could concentrate on the specific arguments put forth for publication. Moreover, the process would be considerably streamlined, since there would be no need to send the manuscript out for review.

This revision of peer review would change the incentives for all involved. The authors would tend to publish results that represent more complete findings and be more satisfied with the outcome, because they could exert lots of control over the review process. The reviewers would tend to be more honest in their evaluations, not wanting to praise work they consider flawed, because their names would be attached to it. Reviewers would not give a cursory and willfully negative evaluation, because the authors could simply not forward their comments. It would be in the reviewers' best interests to help improve manuscripts that have flaws but are potentially important.

The editors would emphasize publication of manuscripts that have the broadest support among scientists in the relevant community or that have the greatest potential to influence the community. Their jobs would be easier because the number of manuscripts submitted would be fewer, although of more substance. This tendency would be facilitated by editors' publicizing the stringent acceptance requirements. For example, editors could request manuscripts with support from reviewers from the same institution and from other institutions. They could request reviewers in the same field and reviewers in related fields.

Peer review is broken. It needs to be overhauled, not just tinkered with. The incentives should be changed so that: authors are more satisfied and more likely to produce better work, the reviewing is more transparent and honest, and journals do not have to manage an unwieldy and corrupt system that produces disaffection and misses out on innovation.