Opinion: Learning from Peer Review

The grant-review process plays significant roles in the education of researchers and in shaping scientific progress.

May 24, 2013
David Irwin, Stephen A. Gallo, Scott R. Glisson

WIKIMEDIA, AREYNAbsent from the many analyses and discussions of scientific peer review are two intangible but very important byproducts: 1) feedback to the applicant and 2) exposure of the reviewers to new hypotheses, techniques, and approaches. Both of these phenomena have a virtual mentoring effect that helps move science forward. Such learning can occur as a consequence of both manuscript review and grant application review, but the review of grant applications, by its very nature, is more iterative and impacts the direction in which research moves very early in the investigation.

The primary mission of reviewing grant applications is to identify the strongest research proposals based on intrinsic scientific merit, hypotheses posed, and feasibility of the approach. At the very least, reviewers will identify the strengths and weaknesses of the application, usually coupled with a scientific merit score, which are then communicated back to the applicant. Thus begins a subtle dialogue between reviewer and “reviewee.”

The simplest form of review is done online, with reviewers sharing their critiques with the applicants electronically. Sometimes, online reviews can include a “chat” or conferral period during which the two (or sometimes three) reviewers may discuss their initial critiques and scores, and then refine either or both to reflect the exchange.

More in-depth reviews can involve regularly convened panels of reviewers who meet over a period of a day or two to review batches of applications grouped by scientific area, learning from each other throughout the process and eventually returning the fruits of their labor back to the applicants in the form of a critique of their proposals. In some of the intramural reviews conducted by the American Institute of Biological Sciences (AIBS), reviews can even involve the applicants directly, in addition to the reviewers and program staff, through an interactive, mid-review teleconference, during which applicants can respond to recommendations and comments in the initial critiques.

In the most elaborate review process, typically reserved for large and/or complex proposals, the review includes a site visit to the PI’s laboratory at the host institution. The review team is typically a group of established scientists and scientist-physicians who have considerable experience and can provide valuable insights and suggestions to the applicant group. Though the setting and procedures are formal and the review groups are not, strictly speaking, advisory groups, there is often an intimate dialogue among applicants and reviewers that can shape and improve the applicants’ research program. Indeed, a survey conducted in 2012 of AIBS reviewers indicated that close to 70 percent believed that feedback to PIs of submitted applications contributed strongly to moving the scientific field forward.

Boosting success

There is a wide range in the extent to which reviewers are allowed to give helpful feedback to applicants. Since the entity commissioning and paying for the review is the funding agency, purists could take the view that the information and messages coming out of the review are solely for the potential funder. Indeed, there was a time when summary statements only went to the funding agency unless they were specifically requested in writing by the applicant. And even today, National Institutes of Health (NIH) reviewers are advised “not to help the applicant rewrite an application through their written critiques, which should stay focused on evaluating the application’s scientific and technical merit.”

But many funding agencies recognize the benefits of sharing panel critiques, and now send copies of review summaries to the applicant upon completion of the review, though many still limit the critiques to the evaluation of strengths and weaknesses. Other funding agencies, such as the Department of Defense, explicitly encourage reviewers to give helpful feedback where appropriate. Similarly, the National Science Foundation’s research grants program provides “feedback for the purpose of improving proposed research and research methods.” As John Glenn, Principal Assistant for Research and Technology for the US Army Medical Research and Materiel Command describes it: “Peer review goes beyond simply ensuring that the highest quality science is selected for funding.”

As a result of this feedback, amended applications often receive improved scores. Indeed, in 2003, an investigator submitting a new NIH application that was amended for the first time had a 45 percent chance of being funded. Even in the tight funding era today, when new submissions have a funding rate as low as 10 percent, the success of revised submissions has only slipped to 39 percent. Not surprisingly, for continuation grants the success rate is higher: around 30 percent when first submitted, and 51 percent when amended.  

A two-way street

Of course, applicants are not the only ones to benefit from peer review; the investigators who take time out from their busy schedules to review the applications submitted by others also have much to gain. Most obviously, they get to observe how strong writing skills and clear presentation lead to successful applications, thereby improving their own grantsmanship. But reviewers also get to learn of new discoveries in the applicants’ preliminary data and new techniques or applications of existing methods that might benefit their own work.

Indeed, the AIBS survey of reviewers indicated that more than 70 percent felt that their participation in peer review was particularly useful in exposing them to emerging scientific areas and technologies. Moreover, reviewers may then choose to track particular investigators presenting their work at scientific meetings and in publications, and may even explore collaborations once the work is in the public domain. And in the shorter term, when reviewers get together during face-to-face panel meetings, their interactions often lead to collaborations. In some respects, the discussions among reviewers during a review panel meeting are very similar to the exchanges that occur during a scientific meeting.

As such, participation in the scientific peer-review process is an exceptional opportunity for both new and seasoned faculty. Not surprisingly, NIH study section members have always enjoyed a higher success rate with their own applications than the general scientific community has, with a current funding rate more than 8 percent higher than the general success rate. Given these clear benefits, scientific peer review should not be viewed as just another hurdle or a nagging chore. Rather, it should be seen as an opportunity to share ideas, learn from others, and embrace the collective effort to move science forward.

All three authors are employees of the American Institute of Biological Sciences’ Scientific Peer Advisory and Review Services (SPARS) Department in Reston, Virginia, where David Irwin is a senior scientist, Stephen A. Gallo is the technical operations manager, and Scott R. Glisson is the director.