Quality Judgments, Cost Concerns Must Be Separated In Peer Review

Peer Review Few discussions arouse as much emotion in the scientific community as the controversy over peer review. As the number of unfunded grants started to rise in the early 1990s, so did complaints about unfair criticisms by reviewers. Many whose applications were unsuccessful claimed that the decisions often were based on reviewers' comments that were inaccurate, irrelevant, or glibly disparaging. While critics of the peer-review system called for reform, its defenders countered that the only problem with the system is a lack of funding to support meritorious research. In reality, the shortage of funding and the fairness of the review process are two separate problems that must be addressed. The need for reform of the existing system has been perceived for some time. At a 1994 meeting called to discuss administrative changes in the National Institutes of Health's peer-review process (D.M. Barnes, Journal of NIH Research, 6[10]:10, 1994), fairness concerns were raised. Issues that were discussed included the qualification of reviewers, the depth and range of competence represented in a study section, and the tendency of reviewers not to deal with the "intellectual substance" of a proposal, but to "focus on technical details." Some have gone as far as to call the entire peer review system flawed and outdated (A.E. Sowers, The Scientist, Oct. 16, 1995, page 12). Among other problems, they cite costs and a lack of fairness (R.H. Tomar, Journal of NIH Research, 6[12]:14-18, 1994). Others have gone even further and likened NIH study sections to the Soviet "Politburos," a panel of top citizens chosen by top citizens that had absolute power (C.W. McCutchen, The Scientist, Oct. 16, 1995, page 13). Opposing them are those who maintain that there is nothing wrong with the process or the reviewers, who are "doing their best," despite a shortage of funds, which they contend is the real problem (B. Herman, The Scientist, Feb. 5, 1996, page 12). The shortage of funding for science is indeed a serious matter. By NIH's own accounts, some programs can fund only 10 percent to 15 percent of approved applications (B. Agnew, Journal of NIH Research, 7[7]:40, 1995). If this funding gap were the only issue, no one would question the quality of reviews. The mere fact that controversy has arisen over the fairness of evaluation is a clear indication that there is a problem with the peer-review process-and it must be taken seriously. Those of us who have served as members and chairmen of review panels for various institutions in different countries should try to put the problem into a broader perspective. Experience with different review procedures teaches us that there are various ways of conducting peer review. The peer-review "system" is not a homogeneous entity. Criticisms of specific practices need not be taken as an assault on the whole. Rather than attack or defend the system, we need to ask: What does peer review intend to accomplish, and how can it work under severe fiscal constraints? We cannot overlook the fact that in a tight funding situation, all peers are competitors. What once was reasonable competition for adequate resources has become fierce competition for scarce resources. It is na"ve to think that this does not affect peer review.Using peer review to distribute insufficient funds will not increase the level of support. Instead, it encourages funding decisions based on minor or marginally relevant criticisms of technical details and unsubstantiated negative remarks. In such cases, distinctions made by even the most objective reviewers may appear irrational. Objective, rational judgments of quality simply cannot be made under the extraneous constraint of fiscal feasibility. Quality Control The key to a fair peer-review system is to have a quality-control mechanism that ensures reviewers have the highest level of competence and the evaluation process is free of conflicts of interest. It also must focus the review on the three key issues: (1) Is the scientific question asked in the proposal clear and important? (2) Is the approach original and feasible? (3) Is the investigator worthy of trust? In addition, quality control must ensure that the evaluation is responsible, meaning that the reasoning leading to every judgment-favorable or unfavorable-should be clearly and explicitly documented. In scientific journals, the editor still serves as a control mechanism and provides an effective court of appeal. This control is being lost in granting agencies, in which reviewers have the only-and thus the final-say. Challenges to the substance of their criticism are routinely written off as "differences in scientific judgment," giving automatic-and potentially undue-credence to the original review. Challenges to their competence are not even discussed. This need not be so. I remember a time when executive secretaries (now scientific review administrators, or SRAs) were sufficiently in command of the subject matter within the realm of their responsibility to spot shortcomings in an application. They would ask applicants to clarify specific points prior to reviews and remind reviewers that they were drifting away from important issues. These executive secretaries ensured that the review was responsible and well-reasoned. They provided each applicant with a critical summary of the reviewers' comments, and of ensuing discussion that led to the vote of the priority score. The recent abandonment of this practice may have relieved SRAs of an unmanageable work load, but it also has left a gaping hole in the process as it abolished a crucial point of quality control. One result is that there is often little relation between the priority score and the unedited written comments of the reviewers. Many of us have seen a review report that praised the importance and timeliness of a proposal, the originality of the approach, and the competence and record of the applicant but assigned a low priority score, which precluded funding. The need for quality control becomes obvious when one reads such reports. Keeping the judgment of quality under supervision and separate from economic issues will restore the integrity of peer review, but it does not ensure the equitable distribution of inadequate resources. The options for solving this problem are limited and require that funding decisions be based on criteria other than quality alone. At first glance, this may appear objectionable. Yet one must remember that making funding decisions on the basis of quality alone works only if one does not have to choose between proposals of equal quality. However, these are precisely the choices that have to be made in today's situation. As one recent study-section member graphically put it (H.C. Palfrey, Journal of NIH Research, 6[11]:14, 1994): "This exercise is in and of itself extremely frustrating, because one is then confined to splitting hairs between equally meritorious proposals." When distinctions must be made between shades of excellence that cannot be rationally distinguished by the human mind, making funding decisions on grounds other than quality becomes inevitable. If one chooses an option in which this is done openly, one can keep the funding system honest. If one insists on making distinctions in quality that cannot be made on clear, rational grounds, one invites corruption, with favoritism or prejudices deciding the judgment. One such option is to enter all proposals rated as outstanding into a lottery. All projects that are responsibly judged as having equally high merit would have an equal chance of being selected for funding. This is preferable to today's situation, in which decisions are made on the basis of petty distinctions that thrive under fiscally constrained conditions in which personal biases, rivalries, and animosities reign. The second option is to have the fiscal decision-the choice between highly recommended proposals-made by an administrator or executive committee, whose erudition and competence transcend specialized expertise. They would have a broad vision of the evolution of science and no vested interests in any area under review. At the same time, these executives can provide the second-tier review of scientific substance: a forum for appeal on substantive grounds, and therefore for quality control. This option is in fact widely used by journals, private foundations, and foreign research councils. We might note parenthetically that an option often debated in political circles-assigning priorities for funding by category and targeting programs-will not solve the overcrowding problem. As soon as a target is defined, it will be oversubscribed. No matter how the problem of distributing inadequate funds is solved, the integrity of peer review must be addressed independently. The scientific community has produced more good science and more good scientists-hence, more outstanding applications-than can be supported in today's budgetary climate. However, it must still call the outstanding outstanding. The scientific community betrays itself and betrays science if it bows to social pressure and decimates its ranks by applying spurious criteria to make distinctions of quality that cannot rationally be made. Oleg Jardetzky is a professor of molecular pharmacology and director of the Stanford Magnetic Resonance Laboratory, Stanford University.

Quality Judgments, Cost Concerns Must Be Separated In Peer Review

Interested in reading more?

Become a Member of