NIH Study Section Members Acknowledge Major Flaws In The Reviewing System

Three times a year, in conference rooms at the National Institutes of Health's Bethesda, Md., campus, about 1,440 permanent members of 100 study sections (accompanied by an annual total of about 1,800 ad hoc members) meet for two days to review 9,000 grant proposals. Critics of the system charge that it is cumbersome; needlessly hard on both reviewers and the reviewed; and rife with the potential for incompetent decisions, abuse of power, and conflicts of interest. Even its supporters acknowledge that it's a system that was devised for another era, a time in which funds were far more plentiful and funding decisions far easier. OPEN TO SUGGESTION: Fred Bookstein urges a new ranking system for grant proposals. "We are operating in the framework of a system that was created 50 years ago essentially to identify the 10 percent of grants that NIH would not fund," notes Donna Dean, acting chief of NIH's Referral and Review Branch. "And now, in my view, we're at the total opposite of that, where we're asking our committees to identify the 10 percent--give or take a few percentage points--that should be funded. Intuitively it seems to me that the decision-making processes and procedures that identify the 10 percent that's bad cannot be those same processes that identify the 10 percent that's the best." A study section meeting is the culmination of a process that begins eight weeks earlier, when each of its members is mailed boxes containing 75 to 100 grant applications. Each of the applications is 25 pages long, not counting supporting documents, which themselves can run to 50 pages or more per application. Although each reviewer is expected to read every one of those applications, he or she is assigned to write detailed critiques of just seven to 12 of them. Each application will have two such reviewers--designated the primary and the secondary--as well as a "reader," a reviewer who must be prepared to discuss the application in detail, but who isn't required to write a critique. At the meeting, once the primary, the secondary, and the reader have delivered their opinions of a particular application, there is general discussion, and then the members (16 to 20 of them in the average permanent, chartered study section) assign priority scores between 1.0 and 5.0. Lower is better, with scores of 1.0 to 1.5 meaning "outstanding," 1.5 to 2.0 meaning "excellent," 2.0 to 2.5 meaning "very good," 2.5 to 3.5 meaning "good," and 3.5 to 5.0 meaning "acceptable." Once everyone has voted, the scores are averaged and multiplied by 100 to give a final score between 100 and 500. The study section's work is then done, and the second part of NIH's "dual-review" system takes over. At this point, an advisory board assigns funding based not only on the study section's judgment of scientific merit, but also on consideration of an institute's programs and priorities. At 76, Howard K. Schachman is old enough to remember when a score of 300 meant certain funding, and even a score of 400 wouldn't doom one's chances. A structural biologist at the University of California, Berkeley, Schachman is also the NIH ombudsman concentrating on basic research. (Lawrence Shulman, former director of the National Institute of Arthritis and Musculoskeletal and Skin Diseases, is NIH's clinical research emissary.) "I'm essentially acting as the eyes and ears of the director to find out what people outside are complaining about, or are pleased with, or would like to see changed," explains Schachman. One thing he would like to see changed is the system of priority scores. "The language is atrocious," he declares. "If you've got a brand-new member of a study section, the first thing they've got to know is that `excellent' is ’MDULíawful." Indeed, in most NIH institutes, applications with final scores above 150 are unlikely to be funded, officials concede. "My own feeling is that it doesn't make any sense to report final priorities to three decimal places," says Schachman. If you get a 120 you might get funded. If you get a 130 you may not get funded. Well, the difference between 120 and 130 is almost indistinguishable in terms of a person's ability to rate." Statistician Fred Bookstein of the University of Michigan, agrees. The veteran reviewer notes: "The grounding of those scales is actually one of the most difficult parts of a study-section member's decision. In my own mind I can't draw the distinction between a 1.1 and a 1.3." Bookstein thinks study section members ought to rank each proposal relative to all the proposals they've seen. Such an approach, he comments, "is more likely to be transferable across study section members, across categories of grant, across levels of risk, and so on." The trouble is that NIH's written rules explicitly prohibit this, requiring that each proposal be judged "independent of other proposals." Schachman thinks that reviewers should use at most only whole and half numbers (1, 1.5, 2, 2.5, and so forth) so that the distinctions they're asked to make would be less fine-grained. And he suggests that different aspects of the proposal be scored separately. "Maybe we ought to disaggregate the scores into three criteria: originality, feasibility, and significance," he suggests. An NIH committee is currently studying the scoring system with an eye toward making changes. The composition of study sections is another perennial source of criticism. As Schachman puts it, "There's one group of people who will say, `I never heard of those reviewers, they're a bunch of young kids.' The other group of people will say, `Look, it's the old-boy's network; no wonder I didn't get funded, because I'm the new kid on the block and they fund their friends.'" PRODUCTS OF THE SYSTEM: NIH ombudsman Howard Schachman, left and reveiw chief Donna Dean acknowledge complaints such as those of Daniel Wetzel. Still others, such as molecular neurobiologist Daniel Wetzel (who was most recently’MDBUí title? at the University of Vermont), complain that their grants are critiqued by incompetent reviewers. Wetzel believes his current unemployment results from his inability to get funded. He has had five incarnations of his grant application--all on stress hormones in Alzheimer's disease--rejected by NIH study sections. He submitted the application initially for a FIRST (First Independent Research Support and Transition) award, then as part of a program project, and finally as an R01 (Research Project Grant). Of one of his rejections, he contends: "There was not anyone in the study section who was familiar with the area at all." His research group, he maintains, "got the sort of questions and comments and criticisms you get from someone who really hasn't used the technology that much. I came very close to being funded," he says. "They were all approved, of course. Everything is always approved. But you're bumped just far enough down sometimes that you don't get funding." Each study section has a scientific review administrator (SRA), an NIH staffer who is expert in the area under review. It is the SRA's responsibility, according to Dean, to keep up with developments in his or her field, to anticipate the direction in which the field is moving, and to recruit reviewers in line with current and future needs and in line with rules mandating diversity in government committees. NIH interprets these rules, says Dean, to mandate not only inclusion of women and ethnic minorities, but also diverse geographical representation, a mix of M.D.'s and Ph.D.'s, a balance of reviewers who are highly competent technically with those who have a broader perspective, and a mix of senior and junior scientists. Most members of the permanent, chartered study sections go through a formal nominating procedure and are appointed for four-year terms. Frequently, though, SRAs add ad hoc reviewers to a study section when in need of special expertise. Ad hoc reviewers also typically make up the entire membership of special study sections, convened a single time to review unique groupings of applications. Being an NIH reviewer is no way to get rich. Although NIH pays all expenses and adds a $150-a-day consultant fee for meeting attendance, this fee is paid only when reviewers are actually in meetings. Not a cent is paid for the many hours of application reading and critique writing leading up to the meeting. "By the time you're all done with it, if you're really doing your job, you're getting $3 or $4 an hour," complains Arthur Sowers, a membrane biologist at the University of Maryland, Baltimore, who has served as an ad hoc reviewer six times in the last eight years. So why does he serve? "It's curiosity more than anything," he explains. "And it's a little bit of ego. There's a small amount of personal recognition. You get in a little bit of a brownie point with the [academic department] chairman, and the chairman gets that with the dean. It's one of those external activities that gives the university some presence in the community." The contentious task of reviewing grants invites questions of competence, ego, and conflict of interest, factors that can and do compromise the integrity of the reviewing process, reviewers acknowledge. Says Sowers: "The psychology is extremely complex. There are subconscious processes going on. I think the No. 1 thing that you have to remember is that the primary reviewer is literally a god. If that guy is incompetent or if he's not familiar with the work, is not interested in the work, he can get up and say virtually anything in favor or against the proposal." Sowers maintains that an individual review can doom a proposal since, in his experience, there is little group discussion following the presentation of the critiques by the primary and secondary. But Schachman, who has recently observed about a dozen study section meetings, disagrees: "In most of the places I went to there was a lot of very good, high-quality, intellectual discussion, a lot of give and take, a lot of arguments over science because of different points of view." However, he says, "There still is a worry that two people could dominate." Sowers points out that "these are extremely complicated proposals, and the only way to really evaluate them is to get people who are in almost exactly the same field. If you do that, you're automatically setting up ally-enemy relationships." For this reason, Schachman advocates adding an additional reader for each application, thus further diluting any one opinion. For this plan to be practical, though, efforts must be made to lower the reviewers' workload. The relative value of the evaluations emanating from study sections is also a source of consternation for critics of the system. Fred Bookstein acknowledges: "There's a temptation to be a bit of a cowboy when you write your original critiques, to enjoy an occasional adverb. I have been one of those cowboys, and I was actually fairly sharply chided from time to time to pull in my outrage and say something helpful. [Reviewers] are trying very hard to be constructive, more than the conventional wisdom has it." Dean agrees that excessive detail in reviews is a worry. "One of the problems that a lot of us at NIH have been concerned about has to do with the overly technical, nit-picky reviews. I don't think reviewers should be evaluating the appropriate pH of solutions. They should be looking at the significance of the ideas." While disappointed applicants have long cried sour grapes when their proposals have failed to find funding, the unfairness may not be all in their heads. NIH's Dean says, "I think conflict of interest is becoming an increasing problem. The fact that we're funding so few of the people who are asking us for money exacerbates that tension." However, she says, "You have to separate out what is a scientific difference of opinion . . . from where an individual might be attempting to benefit from what they learn in the context of a review." NIH has elaborate procedures designed to ensure that no one with a financial or institutional stake in the outcome of an application participates in the discussion. However, notes Dean, with the increasing commercialization of biomedicine, and the accompanying increase in the number of scientists who have multiple collaborations and consulting arrangements, the potential for actual or perceived conflicts is increasing. Still, while there may be a lot of grumbling, there has been no tidal wave of formal complaints. "I would say that it probably would not be more than two or three serious allegations per review round," notes Dean. "The easiest way to deal with it is up front," she says. "When the person submits his grant application he should have a letter with it that says, `I prefer that you not have people from the following in review of my application because . . .' That doesn't happen very often," but when it does, the request is given serious consideration. Another problem study section reviewers must deal with involves applications that are amended and resubmitted repeatedly in response to reviewers' criticisms. A certain amount of this is clearly desirable, says Schachman. However, "I saw an `A8,' which meant that particular grant had been looked at nine times. There are very few such grants, but there are loads of A4s. My own guess is that a lot of the A4s are not likely to be funded, and the individual should be told that in as nice, as forthright, as polite a fashion as possible." Donna Dean agrees. "Perhaps [some reviews should] actually say, `Don't pursue this idea. It isn't going to go anywhere. Don't come back in with amendments. Stop. Step back. Rethink.' These are adults out there who are applying for grants. Maybe we should be more direct." Robert Finn is a freelance science writer based in Long Beach, Calif. (The Scientist, Vol:9, #16 pg 1 , August 21, 1995) (Copyright © The Scientist, Inc.)

NIH Study Section Members Acknowledge Major Flaws In The Reviewing System

Interested in reading more?

Become a Member of