Scientists Foresee Few Effects Of NSF Review Changes

GOOD NEWS: NSF's Paul Herer found the positive reaction to his agency's new review criteria somewhat surprising. Scientists who review National Science Foundation grant applications soon will have a new system by which to judge them. Beginning October 1, the current four merit-review criteria will be eliminated, and reviewers will be asked to assess research proposals based on the proposals' "intellectual merit and property" and their "broader impacts," such as potential effect on societal needs. The National Science Board (NSB), NSF's governing body, approved the changes in late March. They are the first such revisions since 1981. While researchers say the revisions probably will have little impact on the type of science NSF funds, they generally cheer the changes, saying they clarify NSF's review criteria. In addition, some hope the changes could lead to greater support of innovative research that does more to advance scientific knowledge than is achieved under current funding mechanisms. "It's important that NSF maintain a substantial part of its dedication to supporting the fundamental advances in basic science and try to find and support the very best science that's being done in [the United States]," declares James Brown, a regents professor of biology at the University of New Mexico and president of the Ecological Society of America (ESA). The current system consists of four merit-review criteria: researcher competence, intrinsic merit of the research, utility or relevance of the research, and its effect on the infrastructure of science and engineering. NSF has also drawn up several suggested questions whose answers are meant to assist reviewers in assessing applications for the two new criteria. Recommended questions under the "intellectual merit and quality" criterion include the following: What is the likelihood that the project will significantly advance the knowledge base within and/or across different fields? Does the proposed activity suggest and explore new lines of inquiry? To what degree does the applicant's documented expertise and record of achievement increase the probability of success? Is the project conceptually well-designed? Is the plan for organizing and managing the project credible and well-conceived? And is there sufficient access to resources? For "broader impacts," NSF suggests the following: How well does the activity advance discovery and understanding while concurrently promoting teaching, training, and learning? Will it create/enhance facilities, instrumentation, information bases, networks, partnerships, and/or other infrastructure? How well does the activity broaden the diversity of the participants? Does the activity enhance scientific and technological literacy? And what is the potential impact on meeting societal needs? NSF officials insist the previous criteria were not that bad but acknowledge that they were difficult for reviewers to understand and apply uniformly. "The system wasn't broken, and this wasn't intended to be a fix," states Paul J. Herer, senior adviser for planning and technology evaluation in NSF's engineering directorate and executive secretary of NSB's Merit Review Task Force, which developed the new criteria. "We felt the old criteria didn't reflect and capture the breadth of the foundation's activities." For example, the previous standards were inadequate to judge the increasing number of educational programs, engineering projects, and so-called infrastructure items, such as big-ticket research tools, that NSF now funds. SEEKING SUBSTANCE: Warren Washington of the National Center for Atmospheric Research hopes the changes will lead to more in-depth reviews. Under the old system, reviewers often had difficulty deciding among meritorious proposals. Warren M. Washington, a senior scientist and head of the Climate Change Research Section at the National Center for Atmospheric Research in Boulder, Colo., explains that reviewers "were grappling with the problem of how they make those tough decisions: What projects get funded, and what don't? Often the reviewers would send in [reviews] checkmarked 'very good' or 'good' or 'excellent' or whatever, but that wouldn't help the program managers or the panels who often make the final decision." Washington, the chairman of the Merit Review Task Force, calls the revisions "a streamlining." He hopes they will "encourage the reviewers to give more substance" to their reviews of research proposals. NSF funds life-science studies through its programs in biology, polar research, and social and behavioral sciences. Its science education program attempts to improve the teaching of science from kindergarten through college, and its program in interdisciplinary research connects life scientists with researchers in other disciplines. Potential applicants are applauding the revisions. "In general, I think they're excellent," says New Mexico's Brown. Referring to the new criteria's emphasis on creativity and impact, he adds: "To go that route is excellent and a step in the right direction." The Merit Review Task Force developed the new criteria last fall and posted them on NSF's World Wide Web site (http://www.nsf.gov) to solicit reaction from the science and engineering communities. The 350 E-mail messages received in response-which were not posted but were summarized on the Web site-generally were very positive, according to Herer. This was something that he and others at NSF did not expect. "We were surprised that there were a lot of people in the [science] community who didn't really feel that this was a major thing. . . . For the most part, I think it's something the community viewed as positive. We didn't get a big, strong negative reaction. Even within the foundation and the Science Board, there was very little negativity. It was mostly positive." SEAL OF APPROVAL: Howard Silver of COSSA says social scientists are happy with the revisions "I think they've done a fine job," remarks Howard Silver, executive director of the Washington, D.C.-based Consortium of Social Science Associations (COSSA). "We'll wait to see how it develops in October." Silver observes that COSSA members are happy that societal impact will play a role in how applications are judged, and he feels that the task force did well to tinker with the recommendations initially posted on the Web site. In particular, he was pleased reviewers were given "a little bit more guidance." YOUNG BLOOD: First-time applicants may find things easier, according to Steward Pickett of the Institute of Ecosystem Studies. Steward Pickett, a scientist at the Institute of Ecosystem Studies of Millbrook, N.Y., and ESA's vice president for science, notes that because the new system does not ask specifically for an assessment of the applicant, there could be "a real healthy effect on funding first-timers or people who are relatively new in a scientific career. Those people have not had much of a track record, especially not often in funded research, so that seems to be a healthy kind of change-to focus on the work rather than the people." Despite the praise, many in the research community feel that the new criteria are not all that different from the previous ones and will not change the kinds of projects that NSF funds. Even Washington, who chaired the task force that rewrote the criteria, speculates: "There might be some small shift, but I don't see a drastic shift." Longtime science observer Daniel S. Greenberg called the changes "trivial" in a story headlined "At NSF, New Grant Criteria Differ Little from the Old" (Science & Government Report, April 15, 1997, page 3). The article went on to state that the previous criteria "have been in effect since 1981 without stirring any evident protest. The rewording exercise . . . apparently was motivated by concerns that it looks bad for a public organization to work under old rules in these fast-changing times." In addition, the E-mailed responses turned up what Herer terms "pockets of discontent." Some scientists, for example, were concerned that the changes indicated that NSF was de-emphasizing theoretical and fundamental research, and that the two new criteria had to be weighted equally when applications were judged. Others believed that the new criteria failed to take into account originality and creativity, an issue that has become central to the debate over proposed changes in the National Institutes of Health's peer-review system (T.W. Durso, The Scientist, Dec. 9, 1996, page 1). According to Herer, the task force evaluated such comments and made changes to its initial proposal in response. NSF officials agree that the changes are not radical ones and that the goal was for the foundation to continue funding good science. Washington says that under the new criteria, for example, "even though [proposed research] might be very important for societal benefit, if the idea and proposal put forth are weak, I don't think it should be funded, even though it's especially important." While some respondents to NSF's Web site feared this could happen, Washington insists excellence remains the aim: "If it's not high-quality research, it should not be funded by NSF. . . . I don't think anyone wanted to sacrifice excellence." COSSA's Silver believes the effects of the revisions will be "hard to tell," and he attributes this to conflicting signals sent by reviewers. "On the one hand, there's the sense that the reviewers need more guidance. On the other hand, there's the sense the reviewers sort of do their own thing anyway. This is sort of an experiment to see how this is going to work." Pickett, for one, thinks that it could work just fine. "NSF is viewed in the [science] community as the model for the review process," he asserts. "A lot of the other agencies and institutions, even nongovernmental organizations that deal with granting, are beginning to use more the NSF-style models for review. The implications can go beyond NSF if people in other agencies or organizations are watching carefully." POTENTIAL: Wisconsin’s Bassam Shakhashiri thinks the new criteria could help NSF better explain to the public what it does Bassam Shakhashiri, a professor of chemistry at the University of Wisconsin, Madison, and former head of NSF's science and engineering education directorate, says the changes could prove helpful to NSF's financial well-being and public image. He stresses that NSF staffers must take the lead "in conveying the significance of the new criteria to proposal writers, reviewers, and grantees. They, in turn, can help NSF explain to others in their local communities and to their congressional delegations the true mission of NSF in research and education, and why taxpayer support is warranted in both areas." Shakhashiri concludes: "This is a real opportunity for NSF to communicate a critical message about its very essential role in our society." Other scientists join Shakhashiri in speculating that the new system could lead to substantial long-range benefits. However, they hope that the revisions will foster more innovative research that pushes the envelope of science than does the current peer-review system. "How effective these [changes] are is going to depend on whether NSF, by changing the guidelines and enforcing that, really influences the culture of peer review and the way the panels and program officers operate," Brown comments. "I think that's the problem that NSF and the whole scientific community is kind of groping toward." Because of increasing competition for both funds and first-rate journal space, according to Brown, researchers-"especially younger people-tend to do essentially small, relatively safe, solid projects. There is-at least indirectly, then-almost a selection against creativity, innovation, thinking big, and trying to make major conceptual and empirical advances. In most of the editorial boards and review panels I've been on, there's been a recognition of that and an attempt to get out of that framework, but it's very difficult." Brown believes that this mind-set spills over into NSF applications, but he also contends that "what NSF is doing is responding to the right problem with what on paper looks like the right set of guidelines." Like Brown, Pickett worries that research has become too conservative, but he is "not sure we'll see a shift in the kind of project funded" under the new criteria. The new standards "still are clearly aimed at excellence," he says, but if the scientific community interprets excellence using a more conservative frame of reference, the likely result will be "safe" science. However, Pickett is optimistic that new scientific philosophies can be developed. "The community needs to recognize some of these needs," he remarks. "They need to try to put these needs into practice. They can do that with these new criteria nicely."

Scientists Foresee Few Effects Of NSF Review Changes

May 25, 1997

Interested in reading more?

Become a Member of