New NIH Procedures To Shield Clinicians From Grants Bias

As Mark Twain may--or, according to some sources, may not--have said about the weather, everybody has grumbled for years that National Institutes of Health peer-review study sections are stacked against clinical research, but nobody ever does anything about it. Now, that's changing, along with a lot of other standard operating procedures in NIH's peer-review system. NIH's Center for Scientific Review (CSR), which runs the panels that review about 70 percent of NIH grant applications, is about to try several new devices to shield clinical-research proposals from unfair competition with laboratory experiments in peer-review panels dominated by basic scientists. By the end of this year, CSR Director Ellie Ehrenfeld plans to create two new "special emphasis panels" for proposed patient-oriented experiments in oncology and cardiology, and to test a variety of approaches for clinical proposals that still don't seem to fit anywhere. NEW DIRECTION: Under Director Ellie Ehrenfeld, NIH's Center for Scientific Review is trying to change the method and speed of grant approvals. Study-section bias has been a long-standing complaint of clinical researchers seeking NIH grants. Evaluating clinical-research proposals "does take special expertise," says Timothy Ley of Washington University in St. Louis, president of the American Society for Clinical Investigation (ASCI), and many CSR study sections don't have clinical scientists on their rosters. Of course, as Ehrenfeld says, peer-review problems are only "one small piece" of the threat facing U.S. clinical research today. Many experts believe that the chief threat is competition from managed care for patients and revenues, which has weakened the ability of academic health centers to support clinical-research activities and raised pressure on clinicians to spend more time treating patients--thus generating fees--and less time on research. But even if NIH peer review is only part of the trouble, "there is a perception of a [peer-review] problem that is so widespread and so supported by the data that determining whether the data are absolutely precise is no longer the issue," Ehrenfeld told the CSR Advisory Committee in February. So "we decided that it was time to stop talking about the problem and simply begin to create some solutions and try them out." CSR, formerly known as the Division of Research Grants, is creating and testing new approaches to old and new problems in other areas, too, and is picking up the pace of change. Among the steps it is now taking or plans over the next few months are these: AIDS: Eleven AIDS study sections are being reconfigured into eight new panels, combining CSR study sections with review panels run previously by the National Institute of Mental Health (NIMH), the National Institute on Drug Abuse (NIDA), and the National Institute on Alcohol Abuse and Alcoholism (NIAAA). These agencies used to be part of the Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA). The new panels will meet for the first time at next month's round of study-section sessions. Vaccines: CSR will create a new vaccine study section on preventive vaccine development for AIDS and other infectious diseases, an idea proposed in the 1995-96 AIDS research review headed by Arnold Levine of Princeton University in Princeton, N.J. Establishment of the new panel also may attract more researchers to vaccine work, which now has the reputation of getting poor study-section scores and thus little chance of NIH funding. Although spurring vaccine research was not the reason for creating the new study section, "it may be that, if you build it, they will come," Ehrenfeld says. Neuroscience: A new array of 21 neuroscience study sections, combining CSR panels with those formerly run by NIMH, NIDA, and NIAAA, will go into business for the first time at next month's round of study-section meetings. With assistance from six panels of extramural scientists, officials of institutes involved in neuroscience research spent more than a year building the new lineup from the ground up--poring over abstracts from several cycles of neuroscience applications and figuring out how to configure study sections that would review about 50 applications every four months. Behavioral science: CSR has started a similar study aimed at reconfiguring behavioral science study sections, aiming at having a plan in hand by this fall, so that a new array of behavioral study sections will be ready to review applications next spring. Integrating the study sections run by CSR and the three former ADAMHA institutes was ordered by Congress when the three institutes were brought under NIH's wing in 1992. Innovation: NIH officials are beginning to assess how well study-section members are adapting to the new grant-review criteria--including innovation and significance--that were used for the first time in the February-March round of study-section meetings. NIH Director Harold Varmus imposed the new standards last year after deciding that study sections were paying too much attention to a project's feasibility, and not enough to creativity and potential impact. In the new standards' first live trial, "about half of the members of the study section really embraced the new guidelines and tried to swing with it," says Betty Gaffney of Florida State University in Tallahassee, who chairs a biophysics and biophysical chemistry study section. But she adds with a chuckle, "there were others who are old hands and they've got their mold and they didn't crack out of it." So study-section chairs and scientific review administrators, the NIH officials who serve as permanent staff to each study section, will have to keep providing "a little encouragement to keep people focused on the new criteria and the emphasis on innovation, rather than getting into the nitpicking mode about the little details," Gaffney says. "Oversight" groups: CSR is still trying to devise a new advisory-group mechanism that will enable outside scientists to constantly assess study-section operations--in effect, peer reviewing the peer reviewers--without driving study-section chairpersons crazy. One such panel has been monitoring the work of study sections in the Cell Development and Function Initial Review Group (IRG) for more than a year, at the cost of some friction with study-section chairs. But Keith Yamamoto of the University of California at San Francisco, chair of the CSR Advisory Committee, is hoping that such outside panels will provide "a hard look at the current organization and function of peer review and render their opinions on how the process might be improved in that area." The groups' reports also may help CSR in a longer-range project to study the structure of the peer review system overall, Yamamoto says. (One possible trick to reduce tensions may be semantic. Currently, the outsider panels are called IRG Oversight Groups. CSR officials are thinking about calling them something -perhaps just advisory groups--to eliminate the suggestive word oversight.) Looking farther ahead, Ehrenfeld has named a 15-member panel of prestigious outside scientists, including National Academy of Sciences President Bruce Alberts, to study the rationale underlying the way today's science is carved up into study-section topics. The blue-ribbon committee, known as the Panel on Scientific Boundaries for Review, was to meet for the first time late last month. In addition to Alberts, the Boundaries Panel includes: Francisco Ayala, University of California (UC), Irvine; David Botstein, Stanford University School of Medicine, Stanford, Calif.; Ellen Frank, University of Pittsburgh School of Medicine and Western Psychiatric Institute and Clinic, Pittsburgh; Ronald Lee, UC-Berkeley; Eduardo Macagno, Columbia University, New York City; Philippa Marrack, National Jewish Medical and Research Center and University of Colorado Health Sciences Center, Denver; Suzanne Oparil, University of Alabama School of Medicine, Birmingham; Stuart Orkin, Children's Hospital, Harvard Medical School, Boston; Arthur Rubenstein, Mount Sinai Medical Center, New York City; Carolyn Slayman, Yale University School of Medicine, New Haven, Conn.; Ralph Snyderman, Duke University Health System and Medical Center, Durham, N.C.; P. Frederick Sparling, University of North Carolina, Chapel Hill; Larry Squire, Department of Veterans Affairs and UC-San Diego; and Peter von Hippel, University of Oregon, Eugene. "Many of our study sections were developed many years ago," Ehrenfeld says. "They certainly have evolved and changed as the science has changed." But she adds, "As far as I know," no one has tried to think through the principles that guide study-section formation. She hopes the Boundaries Panel will consider such questions as whether study sections should be organized around clinical themes or biological processes (NIH has both sorts), and whether some have become "hard" study sections because too much cutting-edge science is referred to them, while others have become backwaters that provide "entitlements" for their scientific areas. Altogether, "we are undergoing an enormous amount of change," Ehrenfeld says. Some of the change, such as efforts to make study sections friendlier to clinical research, CSR is forcing on itself. As Ehrenfeld told the CSR advisory committee in February, "the [clinical-research peer review] problem has been described ad nauseam." Most recently, a special study committee headed by David Nathan, president of the Dana-Farber Cancer Institute in Boston, concluded that although NIH supports clinical research at "appropriate" levels--27 percent of awards accounting for 38 percent of extramural-project funding went to clinical research in 1996--some study sections still seem to be tilted against clinical proposals. Study sections in which clinical research makes up less than 30 percent of the workload score those grants significantly lower than they score laboratory-oriented research, according to a 1994 report. Study sections that see more clinical proposals score both groups equally well. But many CSR study sections don't meet the 30 percent benchmark. This is the problem that Ehrenfeld is now trying to fix. Major, multicenter clinical trials aren't involved; most of those are reviewed by the institute that will fund them. Rather, the trouble comes in reviewing translational research linking a laboratory with a single patient, or small trials carried out in a single center. In an era when, as NIH Director Varmus has said, molecular medicine is beginning to take hold, this research is becoming more important. By the end of the year, CSR officials will create "special emphasis panels" (SEPs) in the oncology and the cardiology IRGs, gathering together clinical proposals that now are scattered through oncology, cardiology, and some other study sections. But even if the new SEPs seem to work, this will solve only about half of what Ehrenfeld and others call the odd-duck problem: diverse clinical-research grant applications that don't fit anywhere except in study sections where they are heavily outnumbered by laboratory science. For the odd ducks, Ehrenfeld may try several approaches: designating a "clinical ombudsman" in low-density study sections (maybe a current or ad hoc member) to weigh a grant application's clinical impact; getting input from two study sections--a highly technical study section to assess the science, and a clinical panel to judge clinical impact; seeking outside opinions from clinicians who are not regular study-section members; and testing a new SEP for Human Investigations, and letting principal investigators decide whether they want to be reviewed the old way or by the new, experimental panel. "On the surface, it sounds like a darn good idea to me," says Ley. "The best clinical research can make outstanding contributions to science," he says, emphasizing that he is not familiar with Ehrenfeld's proposals and is speaking for himself, not ASCI. But, he adds, "this kind of work requires physician-scientists who have specialized training and expertise to perform it and to evaluate it. Clinical research is very difficult to do because the studies frequently take so long. However, it must be supported, because the stakes for good translational research are so high." Will the peer-review changes that CSR is testing solve the problem? Maybe. "The clinical research SEPs and the odd-duck problem are very complex issues," warns Yamamoto. "While I'm not certain that we know enough yet to act decisively, there is a very strong perception of a crisis out there that needs to be addressed." But even if the answers aren't obvious, Yamamoto likes the approach that Ehrenfeld is now taking. "She has energy and the confidence of the community," he says, "and she's out there gathering data and doing experiments to improve peer review and CSR." Bruce Agnew is a science writer based in Bethesda, Md.

New NIH Procedures To Shield Clinicians From Grants Bias

Interested in reading more?

Become a Member of