During a break at an NIH review panel a few years ago, I was scanning the list of grant applications that were not being scored because they were considered uncompetitive (usually about 50% of all applications). One caught my eye because it was a resubmission from a famous scientist that I knew. I wondered why this accomplished scientist would have his grant summarily rejected twice.
First, I read the reviews of the proposal from its initial submission, which revolved around the technical feasibility of an approach he was implementing. All three reviewers mentioned the same central issue. I then read the response of the applicant, which began: "It is so typical of the status quo, that when their sacred cow is gored, they circle the wagons in defense…." Ouch! I immediately guessed why the grant was rejected the second time.
Although such an emotional response from a well-regarded scientist was surprising, it also made me uncomfortable. It reminded me of my own similar response a decade earlier to what I took as slights by a review panel. In my revised proposal, I was circumspect enough to try to cover my opinions, but I did not take their criticisms seriously, much to my detriment (I was rejected again). Recently, I dug up these old reviews and was chagrined to find nothing insulting from reviewers in them. Time has given me the emotional distance that I had sorely needed.
Serving on multiple review panels has also given me a better perspective. Rather than being self-serving ogres who are part of an elaborate conspiracy to thwart the ambitions of their fellow scientists and maintain the status quo, the reviewers I know are usually motivated by a desire to serve the community and to help fix a system that we all see as inherently flawed. Although some of these reviewers do inadvertently contribute to problems with peer review, it is usually by being too nice rather than too critical. Instead of telling an applicant that their proposal is hopeless, they are far more likely to suggest ways to make it better.
During peer review, grant applications are not judged in isolation, but as a group. By definition, half of all applications are below average. Although most scientific reviewers can agree on why a proposal is important and exciting, it is far more difficult to explain why we don't like the others. The communication problems that plague many applicants in trying to describe the importance of their research also afflict many reviewers in trying to explain the converse.
Good reviewers will take the time and effort to analyze bad proposals and try to provide constructive criticism that can be used to make them better. Some reviewers struggle with describing problems with an overall poor proposal - such as the organization, focus or significance. Somehow, it sounds unscientific to criticize a grant for being unoriginal, vague and boring. Reviewers also know that blunt and specific criticisms will be interpreted as words from Satan. Thus, they usually target obvious technical issues or problems with scope.
Any system will seem unfair when a trivial technical point becomes a major criticism. Fortunately, NIH's proposed reforms in peer review should help by providing a much more explicit and uniform process for scoring grant applications.
But until then, does this mean that trying to interpret the critiques of rejected grants is hopeless? Not at all, but it does help to take a scientific approach when reading them. Just as results from lab experiments provide clues to an underlying biological process, reviewer comments are also clues to an underlying reality (they did not like your grant for some reason). For example, if all reviewers mention the same point, then it is a good bet that it is important and real.
It's okay to get indignant about our ideas being rejected. We should feel passionate about our proposals. But it is counterproductive to consider them personal assaults. I now let negative reviews sit for a couple of weeks, then pretend that they were written by my best friends. This helps me see the truly useful comments that will help my proposal the next time around.
Steven Wiley is a Pacific Northwest National Laboratory Fellow and director of PNNL's Biomolecular Systems Initiative.
Interested in reading more?
Become a Member of
