Does accountability legislation threaten integrity of U.S. research enterprise?

Mildred Dresselhaus coauthored a report to be released Feb. 17. National Academy of Sciences (NAS) President Bruce M. Alberts was so concerned by the accountability provision buried in last year's "omnibus" budget legislation that on Jan. 26 he sent a letter to Office of Management and Budget Director Jacob Lew in which he writes that he's "convinced that the new legislation will have serious, unintended consequences for the nation's research enterprise." Speaking at the recent American Association for the Advancement of Science (AAAS) meeting as part of a symposium on emerging science policy issues, Rosina Bierbaum, associate director of the Office of Science and Technology Policy, called the accountability provision a "sleeper." The provision states that all data produced by federal funding must be available to the public under FOIA, a 1966 law intended to make the government more accountable to the public. Prior to the addition, only sponsor agencies could request access to research results. The following is a preliminary outline of some of the major points of the COSEPUP report due to be released Feb. 17 at a public briefing in the Foundry Building, 1055 Thomas Jefferson St. N.W., room 2004, Washington, D.C. They're taken from the comments of Mildred Dresselhaus, an MIT professor, former AAAS president, COSEPUP member, and report coauthor, who spoke Jan. 23 at the AAAS meeting in Anaheim, Calif. For more information, go to the Web site at www.nas.edu or call the NAS/NAE/IOM/NRC office of news and public information at (202) 334-2138. According to Industry Representatives: Although quantifying results of basic research is not always possible, industry nevertheless supports basic research; in the long run, it pays off and is beneficial to industry. Metrics must be simple, not easily manipulated, and able to drive "constructive behavior." For example, when using indicators such as the number of publications per researcher, the quality, and not just the quantity of those publications should be taken into account. To assess basic research, industry relies on the judgment of individuals with knowledge of research content and how it relates to the objectives of the organization. According to agency representatives (from 10 government agencies, including the Department of Energy, the Department of Agriculture, the National Institutes of Health, the National Science Foundation, and the Environmental Protection Agency): While progress in program management and facility operation can be assigned quantitative values, outcomes of basic research require more than just quantitative measures. There was an "urgent concern" that basic research could not be properly evaluated under GPRA and that "misguided attempts to do so could cause great damage to the U.S. research enterprise." Performance plans tend to emphasize short-term applied research with practical outcomes. Some suggested that such an emphasis might mean less funding for long-term basic research that's difficult to measure against annual milestones. A low level of awareness about GPRA in the research community has resulted in less-than-adequate participation in the process. When the research arm of an agency is small relative to the sum of its other activities, that agency is less likely to include research reports when preparing GPRA performance plans. This must be addressed in order to ensure an accurate portrayal of the federal science and technology investment. Agencies' GPRA performance plans should (but currently do not) address human resources development, including the education and training of graduate students and postdocs. Bibliometric analysis, economic rate of return, case study, retrospective analysis, benchmarking, and peer review are the methods currently available for assessing the quality of research, with peer review being the most popular. "What's the definition of data?" Bierbaum said of one potential problem with the inserted provision. "Is it computer files? Is it notebooks? Is it X-rays?" Symposium participant Rep. Vernon J. Ehlers (R-Mich.), the first research physicist to serve in Congress and the lead author of a year-long effort to revise federal science policy,1 also expressed concern over potential excessive accountability stemming from the FOIA provision and GPRA, saying that it could lead to the funding of only those studies with positive results or immediate applications. GPRA, or the Results Act, requires that all government agencies, including the science affiliated, set objectives for a five-year period, submit an annual performance plan that translates those objectives into intended targets, and submit an annual performance report that states whether the intended goals have been met. GPRA's framers intended that the legislation offer agencies an opportunity to communicate the results of their research to policymakers and the general public. While metrics have been used to evaluate scientific research for decades, many fear that the Results Act presents serious, novel challenges to scientists, especially those involved in basic research, where obvious research applications and impact may be hard to come by. At its worst, GPRA will encourage research project nearsightedness, require unrealistic measurements of research results, and increase the level of bureaucracy. "You have to have some noble failures or you are not reaching far enough," Bierbaum points out, referring to such threats to the research community. "Some places are able to ignore these requirements," said Gretchen B. Jordan of Sandia National Laboratories at a separate AAAS symposium on GPRA and accountability requirements. "But in the federal government sector, and even in state science programs ... if you're funded by the government, you'll most likely have to deal with [them]." She added that some scientists have expressed the need for a "clear and compelling research vision," a vision that the strategic plans required by the Results Act could help foster. Susan Cozzens Several countries, including Australia, Canada, New Zealand, and the United Kingdom, have gotten a jump on the United States in resolving research assessment issues and have already passed legislation with little fuss. "There's a lot of very useful experience there," says AAAS accountability symposium participant Susan Cozzens, the director of the School of Public Policy at the Georgia Tech Institute of Technology. Cozzens, who recently conducted an international study of research assessment at the University of Wollongong in Australia, suggests that there is a "consistency in the way this kind of framework is being approached that comes from the logic of the research itself and is useful across national boundaries." Concerns in the U.S., though, have prompted reviews of GPRA. A National Science and Technology Council 1996 study called "Assessing Fundamental Science" found that while the Results Act should improve performance and strengthen management, existing methods for measuring research output and outcomes are inadequate. An upcoming report echoes these concerns. Due to be released Feb. 17 at a public briefing in Washington, D.C., the study was conducted by the Committee on Science, Engineering, and Public Policy (COSEPUP), a joint unit of the National Academy of Sciences (NAS), National Academy of Engineering (NAE), and the Institute of Medicine (IOM). Speaking at an AAAS accountability symposium, Mildred Dresselhaus, an MIT professor, former AAAS president, COSEPUP member, and study coauthor, outlined some of the major details of the report. (See box at right.) Based on three workshops conducted by COSEPUP, and on other input submitted to the committee, two conflicting viewpoints arose: one, that measuring research and providing quantitative measures annually should be possible; the other, that scientists can, at best, provide measures that appear to respond to GPRA but are essentially meaningless. "The great peril," said Patricia M. Dehmer at the AAAS accountability symposium, "is that GPRA begs us ... to predict the future so that scientists have a goal to which they can aspire, so that you can measure how close they've achieved the goal on an annual basis." Dehmer, manager of the Department of Energy's Office of Basic Energy Sciences, suggested that all involved "resist the desire to quantify those goals in a way that would inhibit serendipitous discovery." www.house.gov/science/science_policy_study.htm

Does accountability legislation threaten integrity of U.S. research enterprise?

Interested in reading more?

Become a Member of