NSF Official Scoffs At Allegations, Asking: 'What Secret Filing System?'

In considering the charges brought forth by Kaib and Glitzenstein in their petition to the National Science Foundation, two questions arise: First, are the charges realistic, or are they based on questionable assumptions and misunderstood information? Second. how does NSF actually deal with the kinds of matters raised? KaIb and Glitzenstein have accused NSF of maintaining a secret filing system for the sole purpose of preventing grant applicant’s from seeing and amending erroneous peer review records.” Under NSF’s current “scheme,” they say, a grant application “can be thrown out if a reviewer or panelist makes an unfounded allegation about the scientist’s reputation or character.” In the 37-page “petition for rulemaking” (a legal procedure that, by the way, has no applicability to NSF) Glitzenstein repeatedly misparaphrases statements from NSF reports, takes them out of context, and presents tortured interpretations of NSF regulations and practices. He paints such a distorted picture of the foundation’s proposal review system that those of us who use it and study it find his description virtually unrecognizable. The most serious matter is that he mischaracterizes the system as one that bases significant grant judgments in part on “personal deroga- tory information”—rumors, for example, “that an applicant is a drug abuser or sex offender Glitzenstein then goes on to charge that the foundation goes to great lengths to hide such personal information from applicants. Sensational stuff, and if true, of course, it would be devastating to the foundation’s long-standing reputation for integrity and openness. But why have outside review committees, which closely examine proposal jackets in every program at least once every three years, never found the secret system? Why has the oversight group in our Inspector General’s Office, which samples several hundred cases at random every year, never found the secret system? Why have our program officers—an independent, outspoken lot, a third of them each year visiting scientists who go back to their campuses—never found the secret system? Consider, particularly, the latter: The system is so remarkably secret that the very program officers who purportedly rely on it don’t even know it exists. Like any complicated process that rests on human judgment, NSF’s proposal review system is not perfect: Among other things, a relatively small staff handles a high volume of work, and because not all worthwhile proposals can be funded, difficult comparative judgments must constantly be made and defended. The foundation’s policy of making these judgments on their scientific ‘and educational merits ‘alone is very clear, the general criteria for doing so are spelled out in instructions to applicants, and practices are monitored through various internal and external oversight means. In 1988 the foundation made decisions to award or decline 24,161 proposals, based on recommendations by some 350 program officers, who took into account advice and comments contained in 114,312 written reviews. These reviews, three fourths of which are received through the mail and about one fourth in panels, were the “yield” from 140,994 review requests to 55,999 individuals. Overall, about a third of the proposals were funded, and a much higher proportion could have been, had the money been available. Reviewers’ comments and ratings are advisory, not controlling; it is the program officer’s responsibility to integrate and reconcile comments and to use his knowledge of the field when making recommendations. The foundation’s policy is to provide all of the reviews (and summaries of panel comments, where panels have been held) to the principal investigator, verbatim but without attribution to individual reviewers. Declined proposers are encouraged to revise their proposals along the lines suggested by the reviews and resubmit them. They are also welcome to contact the program officer for any further explanation, and many do. Applicants who are not satisfied then have three further courses of action. The most commonly used one is to seek additional information or explanation from the division director (who supervises several program officers). Two others are spelled out in our instructions to applicants: The principal investigator may make a formal request to the appropriate assistant director to reconsider whether the proposal received a fair and impartial evaluation; if that is not satisfactory, the institution may ask the foundation’s deputy director for further reconsideration. Such reconsiderations are handled by experienced program managers not associated with the original decision. The number of instances in which a reviewer raises anything remotely resembling the “personal derogatory information” described in the petition is vanishingly small. Nevertheless, and despite the impression one would get by reading the petition, the foundation does have procedures in place for dealing with so-called unusable review information. In December 1986, as one response to a comprehensive assessment of the review system by an external committee, NSF Director Erich Bloch issued intemal instructions clarifying the point that program officers should disregard reviewers’ comments showing any evidence of racial, gender, or institutional bias or containing “intemperate personal attacks.” Any such comments are to be forwarded to the applicant - along with the other reviews, with a note from the program’officer explaining the reason(s) why those remarks were not considered. The petition is also written to leave the reader with the impression that once “personal derogatory information” is in NSF’s records, it remains forever easily accessible to program managers. The founda-tion’s action file is the proposal ‘jacket” containing all information (reviews, notes, recommendations, letters) about that one proposal. For awards, these are retained in the program office until a year after the final technical and administrative reports are received, then sent to warehouse storage. Owing to tight space limitations and occasional office moves, the bulk of declination jackets are shipped to the warehouse in a few months. In theory, they can be summoned back from storage for review; in fact, there is rarely reason to recall them, and the process for doing so is slow. Besides the jacket, the foundation maintains a computerized administrative file for each applicant showing his proposal numbers and their titles, dates received and acted upon, amount and duration requested, and whether awarded or declined. Another computerized administrative file, used by my office to perform statistical checks on the system, shows reviewer ratings by proposal number. No “derogatory personal information” or, for that matter, any reviewer comments or program manager comments, are included in those files. Any system in which two thirds of the applicants are declined each year, some of them repeatedly, is bound to leave many unhappy persons in its wake. But my 1986 survey (The Scientists, May 16, 1988, page 21), to which more than 9,500 applicants responded, showed that concerns about, personal and institutional biases were far down the list of applicant worries. Disappointment with the insubstantiality of some reviews and perplexity in cases where reviewers’ comments did not converge were much larger concerns. Surely, the system does have its flaws—a tendency toward conservatism and pickiness under extreme funding pressure is the most serious one, in my view—but there is no need to make wide-reaching, fundamental changes in response to wildly overblown charges. What changes should reasonably be considered? I intend to offer three suggestions as top management and the National Science Board discuss and consider the topic: (I) building in a feedback loop (20 days, as the petition suggests, is not unreasonable) whereby any “personal derogatory information” that appears can be rebutted—these cases are so few that there would be little effect on overall timeliness of decisions, and applicants would be further reassured; (2) revising our basic information to applicants to make clearer that materials in addition to the verbatim reviews (such as program officers’ memoranda, with reviewer information excised) may be requested within a certain time from the decision, either informally from the program or under the Freedom of Information Act; and (3) changing the language in NSF’s application package that already invites proposers to suggest well-qualified reviewers to also invite them to name persons who should not be asked to review—a practice in some of our programs already. Program officers would honor those requests most of time, lam sure, but choosing reviewers must always be left entirely to their judgment. I am confident that these changes, when seen in light of the existing safeguards, would amply deal with the few cases that arise without turning the review system into the kind of legalistic, antagonistic, more bureaucratized process that the petitioners propose. Neither individual proposers, science as a whole, nor the taxpaying public would be better served by changes that go in’ the petitioners’ direction. Jim McCullogh directs NSFs program evaluation staff The survey mentioned is available without charge from NSFs Forms and Publications Unit, Room 232, 1800 G St., N.W., Washington, D.C. 20550

NSF Official Scoffs At Allegations, Asking: 'What Secret Filing System?'

Interested in reading more?

Become a Member of