Researchers Get Ready For NIH Reforms

As the agency overhauls its peer-review system, scientists assess the potential consequences. The peer-review system at the National Institutes of Health is in the midst of a critical series of reforms that will alter the way study sections judge and score grant applications. For the first time, reviewers will be required to consider five explicit criteria in judging grants, and one criterion will have the effect of placing a premium on innovative science. But NIH director Harold Varmus rejected a controversial recommendation from an internal committee calling for each criterion to be scored separately, so reviewers will continue to assign a single global score to applications. Additional changes, planned for the next year or two, involve reorganizations of study sections and possibly more changes in scoring. Despite praise for the changes from most quarters, some researchers are concerned about the consequences of the reforms and the details of how they will be applied. At a May 5 meeting of the National Institutes of Health's Peer Review Oversight Group, NIH director Harold Varmus listed and defined five explicit criteria by which all reviewers are to judge each grant application. The five criteria are: Significance: Does this study address an important problem? If the aims of the application are achieved, how will scientific knowledge be advanced? What will be the effect of these studies on the concepts or methods that drive this field? Approach: Are the conceptual framework, design, methods, and analyses adequately developed, well-integrated, and appropriate to the aims of the project? Does the applicant acknowledge potential problem areas and consider alternative tactics? Innovation: Does the project employ novel concepts, approaches, or method? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies? Investigator: Is the investigator appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the principal investigator and other researchers (if any)? Environment: Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of unique features of the scientific environment or employ useful collaborative arrangements? Is there evidence of institutional support? More information can be found at the home page of NIH's Division of Research Grants at http://www.drg.nih.gov/. -R.F. Many changes are being driven by Ellie Ehrenfeld, the new director of NIH's Division of Research Grants (DRG). A molecular virologist who was dean of biological sciences at the University of California, Irvine, Ehrenfeld arrived at DRG in January with a mandate to reform peer review. Ehrenfeld immediately began asking scientists representing many segments of the biomedical community for their opinions of the grant-review process. "I guess there's a little good news and a little bad news," she notes, "the good news being that there seemed to be a fairly uniform set of concerns that almost everybody shared." And, she continues with a chuckle, "The bad news being that there was a uniform set of concerns that almost everybody shared. What that meant is that there weren't a thousand different problems to be solved, . . . but it also meant, of course, that they were real problems. They weren't just minor glitches that only affected small segments of the community." One problem involves the organization of study sections. "It's been many years since the study sections were established," says Ehrenfeld. Despite efforts to modify their directions as fields changed, there has not been "a real look at how our study sections are organized and whether they're serving the scientific community of today. "Is too high a proportion of the most active, cutting-edge, top-notch research going to too small a subset of our study sections, so that they're essentially competing with themselves?" asks Ehrenfeld. "On the other side of that, do we have study sections that are reviewing grants in areas that are perhaps not so productive and not so active right now, although they may have been at one time? Do we have newly emerging fields that really have no appropriate homes?" As a first step toward dealing with this issue, NIH's Peer Review Oversight Group (PROG) announced at a May meeting progress toward a thorough reorganization of study sections in the neurosciences. The situation in the neurosciences is critical because of the 1992 integration of the National Institute of Mental Health (NIMH), the National Institute on Drug Abuse (NIDA), and the National Institute on Alcohol Abuse and Alcoholism (NIAAA)-and their separate funding mechanisms-into NIH. A neurosciences working group, consisting of scientists both within and outside NIH, has produced a draft of a plan that places 22 neuroscience study sections into five clusters: Cellular and Molecular Neuroscience, Developmental Neuroscience, Integrative and Functional Neuroscience, Cognitive Neuroscience, and Brain Disorders and Clinical Neuroscience. The draft plan, which is currently open for public comment, is available at http://www.drg.nih.gov/review/neuroplan.htm. It will be finalized and presented at a PROG meeting in November, and the new study sections will begin reviewing grants received for February 1998 deadlines. UPDATES REQUIRED: NIH's Ellie Ehrenfeld notes a need to reexamine study sections to ensure they serve today's scientific community. Study section reorganizations may be significant to individual fields, but the establishment of specific review criteria will immediately affect every grant application and review section. Applications submitted for the October 1997 deadline, which are scheduled to be reviewed by study sections meeting in February 1998, will be judged by the new criteria. While there has been much discussion within NIH on how many criteria there should be and how they should be defined, "in some ways, the most important thing to me about the establishment of these criteria is not even the listing of the criteria themselves, but the fact that there be a set of explicitly defined criteria, which will serve as a guideline to reviewers," maintains Ehrenfeld. In recommendations released in November 1996, NIH's Rating of Grant Applications (RGA) committee recommended three specific criteria: significance, approach, and feasibility (T.W. Durso, The Scientist, Dec. 9, 1996, page 1). Varmus increased this to five: significance, approach, innovation, investigator, and environment (see story on page 9). At the same time, he rejected the committee's recommendation that the criteria be scored separately, instead retaining the traditional global scoring system. TO EACH ITS OWN: UC-San Francisco's Keith Yamamoto believes there's value to assigning separate scores to individual criteria. Keith R. Yamamoto, professor and chairman of the department of molecular and cellular pharmacology at the University of California, San Francisco, and chairman of the DRG advisory committee, is a strong proponent of using explicit criteria. Without explicit criteria, he points out, reviewers tend to overemphasize just one criterion-feasibility. "This is most clearly seen in reading the summary statements from the study sections, [which] have become long narratives in which, experiment by experiment, the grant is dissected and comments are made on the detailed experimental minutiae that will affect whether the experiment will or won't work." When feasibility becomes the main criterion, Yamamoto observes, large, well-established laboratories assume an advantage. "They will have the strongest well of background information to tap and to cite, they will have the largest number of postdocs to use multiple approaches at the same time, and they will have the most equipment. . . . So a new investigator with a great idea, but only a great idea, is just not going to score in such an arena." Aside from simply making the criteria explicit, the most significant change in the new criteria was the requirement that reviewers make a judgment on an application's "innovation." Howard K. Schachman, a professor in the graduate school at the University of California, Berkeley, who is also NIH ombudsman in the basic sciences, advocates using the innovation criterion to judge grants. "We don't want people to be doing their creative work under the rug on Sunday morning and writing a bread-and-butter proposal for something that's straightforward and you know is going to work," he remarks. "It's an attempt to make the science bolder and more imaginative." "Right now, the people who are in the know . . . know full well that the last thing they want to do is put a really great idea in, because it'll work against them," agrees Yamamoto. "Grant applications are missing the stuff that's interesting and creative because the savvy investigator knows to leave it out, put in conventional ideas, get the monies, and then do the good stuff." CLINICAL CONCERN: Some clinical investigators worry that work to collect basic data might not be considered creative, notes Virginia's Thomas Braciale. But not everyone thinks the innovation criterion is a good idea. Some opposition comes from clinical researchers who fear that the criterion will work against them. Thomas J. Braciale, director of the Beirne B. Carter Center for Immunology Research at the University of Virginia Medical Center in Charlottesville, notes that those investigators are concerned because the research they do is not "hypothesis-driven"; rather, it is intended to collect basic data on the response of patients to drugs, therapies, or techniques. "Therefore, the idea that you can employ novel concepts, novel approaches, or novel methods might not really be appropriate to this kind of research." INNOVATION ADVOCATE: UC-Berkeley's Howard Schachman is concerned that creativity is discouraged in "bread-and-butter" proposals. In the first place, responds Yamamoto, "creativity is only one of the criteria, and it's going to be the overall judgment of the reviewer that assigns the final score. The second side is that there's an enormous amount of creative work being done on the clinical side, and I think they're selling themselves short to say that that's not the case. The ability for clinical researchers to apply innovative technology and certainly innovative thinking to the approaches they're taking to human disease has never been greater, it's going to increase further in the future, and I anticipate that we'll see as many creative grants on that side." Frank Solomon, a professor of biology at the Massachusetts Institute of Technology, has another worry about how reviewers will interpret the innovation criterion. "Some people think that 'innovation' means to look at the same situation in a parallel organism. And some people think it's to come up with something brand new. I'm nervous that the people who want to do [something] brand new are going to be hurt by things that are cut in stone." Solomon is afraid that the move to explicit criteria may be "codifying the uncodifiable. Different grants have different strengths to them, and making a grant touch certain bases and fulfill certain criteria in a series of established categories would in some way take away from how grants really do get evaluated, which is some overall sense of their importance. Sometimes grants are important and wonderful because the demonstrate great creativity. But not all terrific pieces of science are infused with great creativity. . . . It makes me nervous to think that somebody who does yeoman work in one way-provides some sort of solid base or provides a very strong structure by very carefully and meticulously going through a problem-would be penalized because they got a low score in the creativity category." BE FLEXIBLE: Northern Illinois' Linda Schwarz advises reviewers to greet the new criteria with "a reasonable degree of latitude." Echoing that fear is Linda L. Schwarz, director of the office of sponsored projects at Northern Illinois University in DeKalb. She notes: "I would hope that investigators wouldn't feel it necessary to have to challenge existing methodologies simply to meet [the innovation] criterion." Those concerns may be mollified by PROG's-and Varmus's-insistence that reviewers arrive at a single global score for each grant application, without scoring the criteria separately. This is a departure from the RGA committee's two recommendations for scoring the applications: Either there would be no overall score and an application's individual criteria scores could remain separate, or an overall score could be computed by averaging the criteria scores. "I think both of those have fatal flaws," argues Yamamoto. "The problem with giving no score, using only the separate criteria, is that you take away from the study sections the very thing we want them to exercise, and that is their scientific judgment of their overall view of this piece of science. In the end it's that level of subjectivity of judgment that we want these really bright people to tell us about. By averaging the criteria scores he contends there is an explicit statement that [they] are equivalent and distinct, and neither of those things is true." While he advocates that study sections assign a single global score to each application, Yamamoto maintains that there's value in assigning separate scores to individual criteria. "My original proposal was to have letter scores-A, B, C, and D-for each of the criteria in which the letters were not linear. A would be the top 5 [percent] to 10 percent, B would be in the next 20 percent, C would be the 20 percent below that, and then the bottom 50 percent would be Ds." He contends that using nonlinear letter grades instead of numbers "would keep the NIH from its natural tendency to make everything numerical." Whether or not letter grades ever become part of the process, Yamamoto believes that the new requirement that reviewers explicitly consider five criteria in writing their summaries will help NIH's advisory councils make actual funding decisions. (Under NIH's two-tier system, study sections merely rank grant applications. The actual funding recommendations are made by National Advisory Boards or Councils, which take into account not only the study section's score but also additional priorities, such as relevance of a particular grant application to an institute's goals.) Yamamoto notes, "If 1.72 was the breakpoint between getting money and not getting money, and you had 15 grants sitting at 1.72, it would be nice to know how to make a decision between those. So if I were sitting on a council and had access to a set of statements that told me that this grant got a 1.72 because it's a new investigator with no track record but a spectacular idea, the feasibility scores low because the originality of the idea is so good that we have no idea if it's going to work, and then you look at the next grant and that got a 1.72 because everything was sort of B+. I know what I would do to those two grants." The idea that councils would use one of the criteria as a tiebreaker is precisely what worries Tony E. Hugli, a member of the department of immunology at the Scripps Research Institute in La Jolla, Calif. "[The single global score] keeps the review number in the hands of the study section, not in the hands of another layer that could [decide], 'Well, this was a very highly creative grant. It may not have had a high technical merit, but because it had high creativity we can go ahead and fund it.'" MODUS OPERANDI: DRG' s Donna Dean notes "an inherent conservatism" in the the 50-year-old peer review system For now, at least, NIH has decided against separate criteria scores, and PROG has tabled several other suggestions from the RGA committee. Reviewers currently give applications scores ranging from 1.0 (best) to 5.0 (worst). The committee recommended that the scale be reversed, so that higher scores would be better. Additionally, there are 41 different decimal scores possible in the current system, and the committee, noting psychometric data indicating that people have difficulty making such fine-grained distinctions, suggested going to an 8-step (0 to 7) or 11-step (0 to 10) scale. Donna Dean, a senior review manager at DRG, says that these suggestions are less critical and will be revisited in a year or so, after the effects of the current changes have been evaluated. "We are dealing with a system that is 50 years old, and I think there's an inherent conservatism that has built up over the years in terms of approach and style and risk-taking on the part of our study sections," says Dean. She points out that "you can't change a system like that overnight." While Hugli supports NIH's reevaluation of peer review, he notes, "the peer- review process is not what is broken. When funding levels fall below the 30 percent line, the process fails. . . . It is virtually impossible to distinguish between the top 20 percent and 30 percent grants using a subjective system of scoring, which is peer review. Fine-tuning the rating system will not solve this problem." Meanwhile, it's still not clear how the changes will affect writers and reviewers of grant applications. Dean says that a DRG working group is preparing a set of "crisp guidelines" for reviewers on using the new criteria. But these guidelines won't be available until the October 1997 study-section meetings. Without such guidelines, Schwarz advises reviewers to greet the new criteria with "a reasonable degree of latitude. I advise them to take the changes into consideration. I would like to be a fly on the wall when the study sections meet, though. I really would like to hear how they approach these new criteria." Robert Finn, a freelance science writer based in Long Beach, Calif., can be reached online at finn@nasw.org.

Researchers Get Ready For NIH Reforms

Interested in reading more?

Become a Member of