The Peer-Review System: Pique. and Critique

In 1978, physicist Richard A. Muller of Berkeley was awarded two distinguished prizes—the Waterman Award and the Texas Instruments Foundation Founders’ Prize—for his research on cosmic rays and adaptive optics. The event was particularly notable because Muller had been refused support for this work after peer review by the National Science Foundation, the Department of Energy, the National Aeronautics and Space Administration, and the Department of Defense. Many innovative research projects fail to pass muster under the peer-review system, and, as in this case, the work has to be done without specific funding. Is this a failure of the peer-review system? What are the system’s other costs? The questions are especially appropriate because peer review is almost universally required now for decisions on research proposals, journal publications, and promotions for academic people. There are substantial costs to peer review in terms of the time investment. A study by NSF in 1980 showed that in a representative university, between 16% and 33% of faculty time was devoted to preparing research proposals and to reviewing others’ proposals. That translates into six to 13 hours per week for each scientist. Since that study, competition for funding among scientists has become more acute. Adding to the problem is the relatively low frequency of success in obtaining funding. A scientist can expect to prepare from three to 13 proposals for each one that is funded. Almost the same range of success applies for the acceptance of research reports by peer-reviewed publications. Scientific journals report rejection rates ranging from 10% to 90%. These statistics are amplified, in a way, because an estimated 60% to 85% of papers that are rejected by one journal are then submitted to another journal, and the peer-review process is brought into action all over again. All of this takes time. A more serious problem with the peer-review system is the question of accuracy. Perhaps the most aggravating outcome of a review program is having an article or grant proposal rejected for the wrong reasons. One critic of the peer-review system, endocrinologist David F. Horrobin, has declared that about one-third of reviewers’ comments are accurate and fair, another third are accurate but are concerned with trivia, and the remaining third are inaccurate. In my opinion that is not an exaggeration. There are numerous instances in which propos als or manuscripts are rejected for absurdly incorrect reasons. I once submitted a research proposal to the National Science Foundation requesting support of experiments on the quantitative relations between the effectiveness of auxin, a plant hormone, and its apparent receptor site. My proposal was rejected. Subsequently the NSF program director who had rejected the proposal visited my campus, where I was able to speak with him at a cocktail party. He was very candid in a friendly way, and proceeded to tell me that my proposal would have been accepted if it had suggested a comparative study of the quantitative relationships among all five of the known plant hormones with their receptor sites, instead of dealing only with one. This was an astonishing suggestion. At the time no experimental evidence whatever had been developed for the existence of receptor sites for any plant hormone other than auxin. If I had been able to reveal the presence, much less the quantitative relationships, of receptors for four of the five plant hormones, I would no doubt have been in line for an astounding array of prizes. Picture the review panel of conscientious scientists making unfor- givably irrelevant criticisms on the basis of an imagined level of expertise. It is had enough for reviewers’ criticisms to be wrong, but the almost universal pretensions to rightness on the pert of reviewers make irrelevant criticisms even more irksome to their targets. Another cost of the peer review system is a worrisome tendency for bias against innovative and original proposals. If you propose to work on a heretofore unprobed area of science, or one remote from accepted areas of “hot science,” chances are substantial that your proposal, like Muller’s, will draw extra criticism and mixed reviews. When competition for funding is acute, mixed reviews markedly decrease the probability that the proposal will he funded. As with most democratic structures, the peer-review system is intrinsically conservative, even though the overall goals of science may be intrinsically nonconservative or even revolutionary. A related problem with the reviewing system is its tendency to follow fads. Proposals that deal with subjects enjoying high popularity in the community of scientists become favored over proposals that do not. In biology, for example, the hottest contemporary fad is genetic engineering, and a disproportionate amount of federal money is going to support projects involving genetic manipulation. There is no question that this is a very exciting sector of biology just now, but I believe that its prolific funding is imposing an extra cost by putting restraints on other important areas of science. The most disconcerting aspect of the peer-review system, however, may be the tendency of reviewers to offer censure—sometimes to the point of personal attacks—rather than constructive criticism. It is easy for reviewers, especially in anonymous reviews, to make careless and unconstructive criticisms. Often the comments reflect protectionism on the part of the reviewer rather than a rational critique. I once received a rejection of a paper that contradicted the existing popular concept of the nature of the flowering stimulus in plants. One reviewer took the occasion to make an extensive diatribe about what an unworthy effort I had made. The following is a paraphrase of what he said: Leopold has searched about for a problem small enough that he could tackle it, and when he thought he had such a problem, he drew his trusty sword and attacked it with a hail of blows in all directions; and when the dust had finally settled, he found that he had only killed a fly. On reading this review, I protested to the journal editor that such a personal attack had no place in the science community. The editor agreed with me, and the review was thrown out. Perhaps it is an unpleasant reflection on scientists that we have such a penchant for criticism. We are involved in science because we enjoy it, and one would expect that we would respond with joy to clever and innovative results from another laboratory as readily as we would to results from our own. But our tendencies for criticism far out-distance our tendencies for praise and shared excitement. The result is often an “adversary mentality,” characterized by hostility instead of a mutual enthusiasm for progress. Today’s science is built upon peer review, and there seems prospect that any valid substitute will be found. We are stuck with it—and with its burdensome costs. But there is a message to be gleaned from examining the system’s faults. It is imperative for each scientist who participates in the review system to exercise restraint in the use of criticism, to be cautious that criticisms are accurate and constructive, to be meticulous in avoiding bias against innovative ideas, and to avoid the tendencies toward inflation of fads. The costs can be made manageable only if we as peer reviewers avoid demonstrations of pique, and instead show skills at critique. Plant physiologist A. Carl Leopold is W.C. Crocker Scientist at the Boyce Thompson Institute and an adjunct professor at Cornell University.

The Peer-Review System: Pique. and Critique

Interested in reading more?

Become a Member of