Science Community Gives Mixed Review To 'Triage'

Sidebar:A New Look For Summary Statements Sidebar: What is Triage? The expanded use of triage--under which certain grant applications deemed to be "noncompetitive" (NC) are not discussed in study sections--by the National Institutes of Health Division of Research Grants (DRG) has met with enthusiasic response from many NIH administrators. The new system, however, has engendered controversy in the extramural community. NIH welcomes expanded use of the study-section procedure -- but some extramural scientists are less enthusiastic Triage, which has been instituted on a trial basis in an increasing number of study sections since February of this year, will be used for all investigator-initiated (R01) and "FIRST Award" (R29) research grant applications beginning with those submitted for October/ November 1, 1994 deadlines. NIH expects about half of all applications to fall into the NC category. The aim of the measure is to allot more time for study-section members to discuss applications that stand a chance of getting funded, as well as to decrease the workload for reviewers and study-section heads, called scientific-review administrators, or SRAs (see story on page 8). In addition, modifications in the review system, designed to reduce paperwork and otherwise streamline the process, have been put in place (see story on page 9). For example, although all applicants will continue to receive summary statements about their proposals, these will now contain primarily the initial reviewers' reports. NC applications will lack a summary paragraph given for scored applications. Wendy Baldwin, NIH's deputy director for extramural research, hopes "triage will help focus the discussion among reviewers. "All applicants will receive comments about their proposals," she points out, "and we expect that, by sending the essentially unedited reviewers' comments, applicants will get faster feedback and a clearer view of the assessment of the application." NIH director Harold Varmus says he is pleased with the way triage is working, and that he has had positive feedback from SRAs. "Study-section meetings have improved, and that's what the objective was," he notes. "Applications in the top half [up to the 50th percentile] are getting a fuller and fairer review." Not all extramural scientists agree that triage is a positive change. Susan Leeman, a professor of pharmacology at Boston University School of Medicine, for one, considers it "inappropriate to ask scientists to spend months writing proposals and then not afford them discussion time at the study-section meeting. "If you want to triage half of all applications," she says, "perhaps one possibility would be to have scientists write brief 'intention' proposals and then invite only those who stand a chance of being funded to submit a full application." Some granting agencies already use this method of sifting proposals. Leeman is also "concerned that triage may provide an opportunity for biases to creep into the review process." Lawrence Grossman, Distinguished Service Professor of Biochemistry at Johns Hopkins School of Public Health in Baltimore and a former chairman of the biochemistry department who has served as a member and chairman of an NIH study section, was disturbed this summer when an application on which he was listed as a consultant was designated NC. "The reviewer apparently dismissed the proposal because of a flaw he perceived in a particular technique," Grossman says. The Johns Hopkins researcher contends that "the reviewer's criticism was based on ignorance of the field. Everyone working in this area of research is aware of the complication cited by the reviewer and knows how to circumvent the problem." Although Grossman currently has a MERIT Award--an eight- to 10-year grant given by NIH to established investigators with good "track records" who are working in promising research areas and get priority scores in the top 20 percent--he is disenchanted with the peer-review system "because funding decisions are based on assessment of anticipated results rather than on proven productivity." Anthony Demsey, associate director for referral and review at DRG, comments: "We hear the complaint occasionally that the reviewers made a basic, elemental error in an assessment, and such a complaint normally forms the basis for a rebuttal letter on the grounds that the reviewers lacked appropriate expertise. But this criticism has nothing to do with triage--it's just inevitable that in the peer-review system there will occasionally be human error. "If, however, the implication is that the error was due to casual review, or sloppiness, or carelessness on the part of a reviewer--looking for an easy way out of having to do a full review--well, we just haven't seen that in the triage process. If anything, the reviewers have been overly cautious." Demsey thinks "triage is a good streamlining procedure," and many SRAs and reviewers agree. SRAs are supportive of the new review process, but reviewers' responses to triage have been mixed. Teresa Levitin, SRA for Human Development and Aging-1, contends that "time saved by not having to discuss the NC applications" makes for "richer discussions of those applications that do get reviewed at the meeting." Patricia D'Amore, a member of an early "test" study section, is also positive about triage. In her opinion, "reviewers [should] spend more time judging the science rather than suggesting experiments or trying to help rewrite applications." Thomas Parsons, chairman of the Experimental Virology Review Group, concurs: "Up to now, proposals were tailored--after three or four resubmissions--to what the reviewers think should be done rather than retaining the PI's [principal investigator's] original aim." Levitin voices a similar feeling: "The study section should evaluate the quality of the PI's application, not the collective judgment of the previous study section's critiques." But a senior professor at Harvard Medical School, speaking on condition of anonymity, is adamant that "the job of reviewers is to get good research funded, and reviewers should help PIs. Some of the best scientists write the worst applications because they are too close to their own work." Some study-section members worry that triage will unduly increase an already disproportionate influence exercised by the primary reviewers. Edward Zapolski, SRA of the Metallobiochemistry Review Group, is concerned that the system "may cause reviewers to worry about getting re-proached by colleague-applicants who know which reviewers are on a study section and also know that it only takes one objection [from a study-section member] to bring an NC application back to full review." Additional controversy arises from the changes in the summary statement that have taken place along with triage. A number of scientists, including some reviewers, think the revised summary statements will deprive researchers--perhaps those most in need of advice--of important feedback about their applications. But NIH officials counter that the new summary statements actually provide grant applicants with the individual reviewers' reports, and summary statements for NC applications lack only a summary paragraph describing the study-section discussion of the application. Janet Rasey, a professor of radiation oncology and director of the Research Funding Service at the University of Washington Medical Center in Seattle, who has served two four-year terms as a study-section member, feels conflicted about the new procedures. She thinks "it's great to have more time to discuss the borderline proposals--but triaging 50 percent of applications is unrealistic; there are too many good proposals." In addition, she worries that "summary statements consisting of reviewers' reports may be confusing to PIs--especially when the reviewers disagree with each other." Moreover, Rasey says, "triage may make more work for conscientious reviewers who may feel compelled to write a more thorough critique than they would previously have done." Furthermore, she points out, "reviewers themselves may 'lose out' under the new system: Study-section discussion often helps clarify complex issues in the minds of reviewers, who can then temper their critiques accordingly." Rasey maintains, however, that triage and other remedies obscure "the real problem," which is "that there's not enough money to fund all the really good grants; tinkering with peer review is largely cosmetic." Many of her peers concur. A few people contend that much of the negative response to triage stems from a misunderstanding of the new system. Some SRAs and reviewers say that triage is nothing more than a formalization of the way in which study sections already operate, because applications not likely to be funded are generally given only brief discussions, anyway. However, Levitin and other SRAs, as well as some study-section chairpersons, say applicants will get adequate feedback from the reviewers' reports despite lack of discussion by the whole study section, especially because critiques for NC applications--initially restricted to one page in the February 1994 triage trials--no longer have a page limit, and a number of reviewers will write longer tutorial reports because they want to help PIs. Jane Koretz, a professor of biology and biophysics at Rensselaer Polytechnic Institute who served as a member of an NIH study section from 1989 to 1993, believes the new form of the summary statement is "good for PIs because they get to see conflicts between reviewers that were sometimes 'papered over' in the integrated summary statements written by SRAs." She also thinks the verbatim presentation of the reviewers' remarks on the summary statements may encourage "a higher and more objective standard of review than some reviewers, unfortunately, currently practice." But Koretz does not like the use of triage. She says that "the NC classification institutionalizes a trend toward awarding 'fundable' and 'non-fundable' priority scores that began when NIH funding got tight." She contends that "study sections should be in the business of assessing scientific merit, rather than trying to figure out whether a proposal is competitive or not." Gerhard Ehrenspeck, SRA of the Cellular Biology and Physiology-2 study section, is in favor of triage and thinks "it is working well." He observes: "I have had no negative feedback from reviewers in my study section nor from any of the PIs whose applications were designated NC by my study section in February or July." Others think that, in any case, study sections should stop spending time on applications that stand little chance of succeeding at current funding levels. Garrett Keefer, SRA of the Experimental Virology Review Group, which has used triage since February, predicts that NC designations are likely to be on target. In his February study-section meeting, 27 of 90 proposals were designated NC. Of the 27, one was recalled for full review but in the end got an NC score, anyway. Parsons further suggests that "an application designated NC two or three times in a row should not be permitted to be submitted again in the same form." Meanwhile, scientists will have to deal with triage. In the long run, observers believe, it may save time for applicants because summary statements for NC applications will be returned quickly, allowing more time to revise or make other plans if an application is turned down. Also, they point out, writing revisions to an application that doesn't stand a chance of getting funded is a waste of precious research time. In any event, NIH officials say they hope PIs will give triage a fair chance and provide DRG with objective feedback about the various aspects of the modified system of peer review. Liane Reif-Lehrer is president of Erimon Associates, a consulting firm based in the Boston area, and the author of Grant Application Writer's Handbook (Boston, Jones and Bartlett Publishers, 1995). She gives workshops on grant-proposal writing and related subjects.

Science Community Gives Mixed Review To 'Triage'

Interested in reading more?

Become a Member of