Researchers Disagree On NIH Plan To Improve Its Peer-Review Process

Sidebar: The RGA's Committee's Recommendation CREATIVE BOOST: An alternative proposal from UCSF’s Keith Yamamoto adds innovation into the peer-review equation. As officials at the National Institutes of Health consider a proposal to change the peer-review process by which grant applications are considered, researchers offer divided opinions on whether the plan would help or hinder science. A highly disputed issue -- whether the changes would foster creative, innovative science -- has led to a much-discussed alternative offered by a member of an NIH advisory committee. While calling the current system good, NIH is seeking a better way to judge the more than 30,000 applications it receives annually. The process now in place has come under fire by scientists who are upset over the way the agency distributes NIH's limited research funds. Many feel it discourages the pursuit of cutting-edge research in favor of research with "guaranteed" results. "The applicant community -- the scientists, the working investigators -- recognize that creativity is something that is not given the benefit of review. In fact, there's a penalty for putting creative ideas into applications," contends Keith R. Yamamoto, professor and chairman of the department of cellular and molecular pharmacology at the University of California, San Francisco, who developed the much-discussed alternative to the NIH proposal. "That seems to encourage run-of-the-mill, conventional science." The proposal now on the table (see inclusion at right) is a product of NIH's Rating of Grant Applications (RGA) committee, formed by the agency in the fall of 1995 to recommend improvements. The proposed changes involve review criteria, rating scales, and calculation, standardization, and reporting of scores. Scientists generally agree with most of them. But sparking considerable debate are provisions that would reduce the review criteria from six to three, require reviewers to rate each criterion separately, and prevent them from applying a single, overall rating to an application. The plan, parts or all of which could be implemented by fiscal year 1998, was posted on the Internet, and scientists were invited to respond with their opinions. Late last month, the Peer Review Oversight Group (PROG), composed of NIH officials and nongovernmental scientists who make recommendations on development and implementation of the proposed changes, met in Bethesda, Md. At the meeting, members discussed the RGA proposal and Yamamoto's alternative, among other issues. The following recommendations of the National Institutes of Health's Rating of Grant Applications (RGA) committee are taken from NIH's World Wide Web page (www.nih.gov), where the committee's entire report is available. Comments may be sent to dder@nih.gov. The three proposed criteria listed below should be adopted for unsolicited research project grant applications. Significance: The extent to which the project, if successfully carried out, will make an original and important contribution to biomedical and/or behavioral science. Approach: The extent to which the conceptual framework, design (including, as applicable, the selection of appropriate subject populations or animal models), methods, and analyses are properly developed, well-integrated, and appropriate to the aims of the project. Feasibility: The likelihood that the proposed work can be accomplished by the investigators, given their documented experience and expertise, past progress, preliminary data, requested and available resources, institutional commitment, and (if appropriate) documented access to special reagents or technologies and adequacy of plans for the recruitment and retention of subjects. Reviews should be conducted criterion by criterion, and the reviewers' written critiques should address each criterion separately. Applications should receive a separate numerical rating on each criterion. Reviewers should not make global ratings of scientific merit. The rating scale should be defined so that larger numerical values represent greater degrees of the characteristic being rated and the smaller values represent smaller degrees. The number of scale positions should be commensurate with the number of discriminations that reviewers can reliably make in the characteristic being rated. An eight-step scale (0-7) is recommended on the basis of the psychometric literature; however, maximum of 11 steps (0-10) are acceptable. The rating scale should be anchored only at the ends. The performance of end-anchors should be evaluated and other approaches to anchoring should be investigated as needed. Scores should be standardized on each criterion [by each] reviewer and then averaged across reviewers. The exact parameters for this standardization should be defined by an appropriately constituted group. Scores should be reported on the scale used by reviewers in making the original ratings. Scores should be reported with an implied precision commensurate with the information contained in the scores. Two significant digits are recommended. If a single score is required that represents overall merit, it should be computed from the three criterion scores using an algorithm that is common to all applications. The Committee favors the arithmetic average of the three scores; however, an appropriately constituted group should test and choose the algorithm to be used. Under the current system, applications are reviewed according to originality of the proposed research; experimental approach and methodology; availability of resources; proposed budget and duration; and experience of the principal investigator in the proposed area. Reviewers are not required to address each criterion separately. They would be replaced by the following criteria: Significance: The extent to which the project, if successfully carried out, will make an original and important contribution to biomedical and/or behavioral science. Approach: The extent to which the conceptual framework, design (including, as applicable, the selection of appropriate subject populations or animal models), methods, and analyses are properly developed, well-integrated, and appropriate to the aims of the project. Feasibility: The likelihood that the proposed work can be accomplished by the investigators, given their documented experience and expertise, past progress, preliminary data, requested and available resources, institutional commitment, and (if appropriate) documented access to special reagents or technologies and adequacy of plans for the recruitment and retention of subjects. Many researchers who responded to NIH's recommendations endorsed the alternative plan proposed by PROG member Yamamoto. While agreeing with many of the RGA committee's recommended changes, he advocates a four-criteria review that would assess impact, feasibility, creativity/innovation, and investigator/environment. In addition, he calls for "a single quantitative score" to be assigned to each application. The discussion among PROG members at the Bethesda meeting revealed divisions over whether "creativity" should be included as a separate criterion and whether to score each criterion separately. PROG chairwoman Wendy Baldwin, NIH's deputy director for extramural research, declined through an assistant to be interviewed for this article, but noted at the meeting: "We need to be perfectly sure the scoring system is going to give the most information in the most usable way." Responding to opinions submitted by researchers, PROG already has taken steps to attempt to make Baldwin's hope a reality. For example, it has recommended the continuation of assigning a single, overall score to each application and suggested expanding the three-criteria model to include creativity. But even among PROG members, the discussions have been pointed. "The issue becomes, who is this for? That keeps perpetuating," commented Alan I. Leshner, director of the National Institute on Drug Abuse, at the meeting. "Is it for the reviewers? . . . Is it for the applicants?" In order to test the proposal, NIH set up pilot studies by which some of the institutes' reviewers graded applications according to the new, three-criteria plan and to Yamamoto's modified version. The results, reported at the PROG meeting, indicated that the reviewers disliked scoring each criterion individually, although several believed they were useful guides in rating the overall application. "Innovation/creativity," the criterion added by Yamamoto, was difficult to assess concretely as a separate category, they added; some felt it could be addressed adequately under "impact." That led PROG's Mary Jeanne Kreek, professor and senior physician in the laboratory of the biology of addictive diseases at Rockefeller University, to suggest "impact and/or creativity" as a criterion. Support for both the current system and the new proposal is reflected in interviews with several researchers, many of whom participated in NIH peer review in the past. "Basically I'm in agreement with essentially all of [the proposed changes]," notes Geoffrey R. Howe, professor and head of the division of epidemiology at the Columbia School of Public Health. Howe, who once chaired an NIH study section, continues: "After I left I recommended to NIH several of the changes along the lines of what they've come up with. I'm quite encouraged by them." MISSING CRITERIA: ASBMB’s Jack Dixon believes creativity and an overall score are essential parts of the peer-review system. For his part, Jack E. Dixon, president of the Bethesda, Md.-based American Society for Biochemistry and Molecular Biology (ASBMB) and chairman of the department of biological chemistry at the University of Michigan Medical School in Ann Arbor, praises most of the alterations in the RGA proposal. But he found two issues -- the lack of creativity as a criterion and of an overall score -- to be "particularly troubling," and wrote to Baldwin under ASBMB letterhead to say so. He explains that researchers in the United States have been able to make progress "based on novelty and originality and innovation. I think we need to have those as explicit things to deal with." Regarding the lack of an overall score, he cites his "countless hours" of reviewing, adding: "In the absence of a rating system, of allowing the reviewer to give it a score, you trivialize the process." Others say that the RGA proposal's criteria-significance, approach, and feasibility --adequately take creativity into account. "In the three-criteria model, the language is explicit in the combined criterion of impact and creativity, that both should be addressed," says David M. Center, a professor of medicine at Boston University's School of Medicine and a PROG member. "Some people believe that separating them and making a person write a line will make it more efficient, and I agree with that; [but] that's a technical thing. I don't think that's as important as some people make it out to be. The language in both forms is that the reviewer must explicitly address creativity and impact." Columbia epidemiologist Howe believes that the three criteria advocated by the new proposal should not be judged evenly. "The question of an equal rating to all three criteria is not appropriate," Howe declares. "I think more should be given to the criterion of significance." NO IMPROVEMENT: Scoring criteria separately will fractionalize scientists and weaken reviews, says MIT’s Gerald Fink. However, Gerald R. Fink, director of the Whitehead Institute for Biomedical Research in Cambridge, Mass., and the American Cancer Society Professor of Genetics at the Massachusetts Institute of Technology, argues that scoring each criterion separately will lead to "fractionation" of scientists by weakening the judgment of the reviewers and not allowing them to take certain factors into account. According to Fink, NIH should be concentrating on improving the peer portion of the review process. "This is more some sort of bureaucratic way of scoring grants," he says. "But I think the more important thing is who's scoring. The reason I think it didn't [address bettering the peers] is, that's much more difficult." GENERAL USABILITY: Wistar Institute’s David Kritchevsky contends that NIH favors "relevant" over innovative research. Would the recommendations lead to better funding and, by extension, to better science? David Kritchevsky, institute professor at the Wistar Institute in Philadelphia and a researcher who has given up applying for NIH grants, believes that the institutes' reviewers have slowly moved away from funding truly innovative research in favor of generally usable work. "The trouble is that originally this was for the purpose of funding original research," he states. "More and more they're going for not whether it's original but whether it's relevant, and nobody knows what the hell relevant means." Nevertheless, Center feels the changes would tighten an already good process. "What we have attempted to do is make certain that nothing slips through the cracks, that credible grants all get a good review, and that the review is objective and fair," he declares. Yamamoto contends that the current system is "encrypted" so that applicants include their most feasible proposals and avoid creativity, fearing such work will not get funded. "The study sections find that irresistible, because it looks as if every experiment is bound to work," he contends. "The young people say, 'We can't compete with that.' And so we're actually discouraging some of our best young people from ever entering the field because of this encryption in the system." "It looks to me like a bureaucratic solution to the wrong part of the problem," Fink states of the proposal. "I think we should be examining this. But I feel the emphasis should be on how to better peer review, not how to make better fractionation of the score." ADEQUATE PROVISIONS: The RGA proposal ascribes sufficient importance to creativity, PROG member David Center maintains. Howe believes the implications will be positive for both reviewers and applicants. "For the reviewers it's going to clarify the situation. It's going to focus them very specifically, one at a time, on the areas they're supposed to be looking at. It serves the applicants, particularly if they do indeed give more weighting to the significance criterion. It's going to discourage the sort of humdrum, run-of-the-mill applications and encourage them to be innovative." Kritchevsky foresees more work for reviewers, but says it would be worth it if applicants receive better feedback. "The real crux of this stuff is what message gets back to the applicant so he can do better next time. Hopefully, maybe, this will do it, but I wouldn't bet on it. "The only real solution," he continues, "is a lot more money in the system.

Researchers Disagree On NIH Plan To Improve Its Peer-Review Process

Interested in reading more?

Become a Member of