Opinion: Learning from Peer Review

The grant-review process plays significant roles in the education of researchers and in shaping scientific progress.

By , , and | May 24, 2013

WIKIMEDIA, AREYNAbsent from the many analyses and discussions of scientific peer review are two intangible but very important byproducts: 1) feedback to the applicant and 2) exposure of the reviewers to new hypotheses, techniques, and approaches. Both of these phenomena have a virtual mentoring effect that helps move science forward. Such learning can occur as a consequence of both manuscript review and grant application review, but the review of grant applications, by its very nature, is more iterative and impacts the direction in which research moves very early in the investigation.

The primary mission of reviewing grant applications is to identify the strongest research proposals based on intrinsic scientific merit, hypotheses posed, and feasibility of the approach. At the very least, reviewers will identify the strengths and weaknesses of the application, usually coupled with a scientific merit score, which are then communicated back to the applicant. Thus begins a subtle dialogue between reviewer and “reviewee.”

The simplest form of review is done online, with reviewers sharing their critiques with the applicants electronically. Sometimes, online reviews can include a “chat” or conferral period during which the two (or sometimes three) reviewers may discuss their initial critiques and scores, and then refine either or both to reflect the exchange.

More in-depth reviews can involve regularly convened panels of reviewers who meet over a period of a day or two to review batches of applications grouped by scientific area, learning from each other throughout the process and eventually returning the fruits of their labor back to the applicants in the form of a critique of their proposals. In some of the intramural reviews conducted by the American Institute of Biological Sciences (AIBS), reviews can even involve the applicants directly, in addition to the reviewers and program staff, through an interactive, mid-review teleconference, during which applicants can respond to recommendations and comments in the initial critiques.

In the most elaborate review process, typically reserved for large and/or complex proposals, the review includes a site visit to the PI’s laboratory at the host institution. The review team is typically a group of established scientists and scientist-physicians who have considerable experience and can provide valuable insights and suggestions to the applicant group. Though the setting and procedures are formal and the review groups are not, strictly speaking, advisory groups, there is often an intimate dialogue among applicants and reviewers that can shape and improve the applicants’ research program. Indeed, a survey conducted in 2012 of AIBS reviewers indicated that close to 70 percent believed that feedback to PIs of submitted applications contributed strongly to moving the scientific field forward.

Boosting success

There is a wide range in the extent to which reviewers are allowed to give helpful feedback to applicants. Since the entity commissioning and paying for the review is the funding agency, purists could take the view that the information and messages coming out of the review are solely for the potential funder. Indeed, there was a time when summary statements only went to the funding agency unless they were specifically requested in writing by the applicant. And even today, National Institutes of Health (NIH) reviewers are advised “not to help the applicant rewrite an application through their written critiques, which should stay focused on evaluating the application’s scientific and technical merit.”

But many funding agencies recognize the benefits of sharing panel critiques, and now send copies of review summaries to the applicant upon completion of the review, though many still limit the critiques to the evaluation of strengths and weaknesses. Other funding agencies, such as the Department of Defense, explicitly encourage reviewers to give helpful feedback where appropriate. Similarly, the National Science Foundation’s research grants program provides “feedback for the purpose of improving proposed research and research methods.” As John Glenn, Principal Assistant for Research and Technology for the US Army Medical Research and Materiel Command describes it: “Peer review goes beyond simply ensuring that the highest quality science is selected for funding.”

As a result of this feedback, amended applications often receive improved scores. Indeed, in 2003, an investigator submitting a new NIH application that was amended for the first time had a 45 percent chance of being funded. Even in the tight funding era today, when new submissions have a funding rate as low as 10 percent, the success of revised submissions has only slipped to 39 percent. Not surprisingly, for continuation grants the success rate is higher: around 30 percent when first submitted, and 51 percent when amended.  

A two-way street

Of course, applicants are not the only ones to benefit from peer review; the investigators who take time out from their busy schedules to review the applications submitted by others also have much to gain. Most obviously, they get to observe how strong writing skills and clear presentation lead to successful applications, thereby improving their own grantsmanship. But reviewers also get to learn of new discoveries in the applicants’ preliminary data and new techniques or applications of existing methods that might benefit their own work.

Indeed, the AIBS survey of reviewers indicated that more than 70 percent felt that their participation in peer review was particularly useful in exposing them to emerging scientific areas and technologies. Moreover, reviewers may then choose to track particular investigators presenting their work at scientific meetings and in publications, and may even explore collaborations once the work is in the public domain. And in the shorter term, when reviewers get together during face-to-face panel meetings, their interactions often lead to collaborations. In some respects, the discussions among reviewers during a review panel meeting are very similar to the exchanges that occur during a scientific meeting.

As such, participation in the scientific peer-review process is an exceptional opportunity for both new and seasoned faculty. Not surprisingly, NIH study section members have always enjoyed a higher success rate with their own applications than the general scientific community has, with a current funding rate more than 8 percent higher than the general success rate. Given these clear benefits, scientific peer review should not be viewed as just another hurdle or a nagging chore. Rather, it should be seen as an opportunity to share ideas, learn from others, and embrace the collective effort to move science forward.

All three authors are employees of the American Institute of Biological Sciences’ Scientific Peer Advisory and Review Services (SPARS) Department in Reston, Virginia, where David Irwin is a senior scientist, Stephen A. Gallo is the technical operations manager, and Scott R. Glisson is the director.

Advertisement

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: Dariusz L.

Dariusz L.

Posts: 12

May 25, 2013

Comparing a grant application peer-review and a manuscript peer-review is misleading.

Grant peer-review is done by a group of scientists who are not anonymous to each other. They review the grant and debate over it. The reviewers have time to focus on the grant and have time to debate. Often they are reimbursed for the job what "forces" them to take this job seiously and provide reliable and responsible comments.

Manuscript peer-review is done anonymously by a single scientist sitting alone in his office, often having not enough time to read carefully and thoroughly the manuscript. These peer reviews are done in a hurry... The job is done for free as a service to the scientific community...

 

Avatar of: THenry02

THenry02

Posts: 12

May 25, 2013

That's a lot more casual than I thought it was.

Avatar of: FJScientist

FJScientist

Posts: 24

May 28, 2013

I encourage early stage investigators to get on review panels as early as possible because it helps to rapidly get rid of whatever misconceptions they may have about the review process. Of course, it is possible to get on the NIH review panels only after you have received an R01, which is a bit of a catch-22. So, how to start?

One of the professional organizations I participate in (the Endocrine Society) has at it's annual meeting mock review panels where junior investigators can at least get exposed to the discussion aspect. It's a good idea to attend if you are a junior investigator.

I would though say that participation in consecutive study section meetings provides the best insight into the review process since you get to see how an application rises or fails when it is revised and resubmitted in response to reveiwers' comments. In a nutshell,  seek out advice, and listen to, your more experienced colleagues. Ask them to read your application and give you frank, honest feedback. Don't be defensive when you are given that advice.

Below, I try to provide some general comments to junior investigators about grant reviews. Be aware that others may provide a different set of comments. So find as many colleagues who can help, hear their advice and then synthesize your own conclusions from that advice. Then see what the reviewers say.

1, Be aware that grant reviews are not manuscript reviews. Many junior investigators get confused on that overarching point because they only have had experience to date with getting manuscripts published. For a manuscript, you some have expectation that the manuscript will be accepted if you answer the reviewers' comments. Not true for a grant application. A 25th percentile concept will remain that regardless of what technical improvements you add in the revised application. So, you have to learn that a grant is about the concept for which you must also demonstrate the capability to execute. Demonstrating the capability alone is insufficient for the reasons below.

2, Sitting on a grants review panel teaches you that there are a lot of applications out there that are just as 'worthy' as your own. Most of those, like your own, will not get funded if they do not stand out from the pack. Knowing that makes you recognize even more that your success depends on pursuing a research direction that is not just topical but even urgent enough that others in the field will want you to do it. Last year's news doesn't cut it. Remember, you are not working in isolation and you want others to think 'if they pull this off, this will affect how I do my own work'.

3, Lastly, reviewing grants exposes you to just how few grants there are. Let's say that there are three R01's available in field X in the next year. That means 15 maximum over a five year grant length. Ask yourself--is this particular grant really so groundshaking that it has the expectation of being within the top 15 subject areas in the entire field over the next five years? Don't be discouraged by this news. Be aware of it and devise your research directions according (starting with the ground-work preliminary data you will need to support said application). Recognizing that simple fact is the first key to success. Good luck!!

 

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement
RayBiotech
RayBiotech

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
R&D Systems
R&D Systems
Advertisement
PITTCON
PITTCON
Life Technologies