Sound Science Policy Requires Better Data Management

AUTHOR: DARYL E. CHUBIN AND ELIZABETH M. ROBINSON, p.11 How can Congress ensure that the best science continues to be funded, and that a full portfolio of research is maintained? The answer, in large part, is to collect sufficient and relevant data on the research enterprise in the United States, and to see that it is circulated efficiently among decision makers. The information should include, at a minimum, details on how research moneys are spent, on the scientific work force, on the key elements of the research enterprise itself (how scientists spend their time, and what equipment and facilities they must have), and on the measurable outcomes of federally funded research. A recent study by the Office of Technology Assessment, requested by Congress, concluded that today's politically competitive environment requires that Congress set priorities in allocating research funds. A rapid growth in the number of researchers, tight constraints on federal spending, and an ever-increasing number of problems that would benefit from technological solutions all create intense competition for federal funding that is not expected to ease in the foreseeable future. In the ongoing policy-making process, resistance by those outside of Congress to the idea of setting priorities, which has been evident in the past, is beginning to fade. And as it does, the need is more urgent than ever to implement highly efficient mechanisms and procedures in information gathering and dissemination. Priorities, of course, are set through the federal budget process at many levels. But priority setting falls short when the criteria used in considering various areas of research are not made explicit, and when there is no mechanism for evaluating the total federal research portfolio in terms of progress toward broad national objectives. In short, without the timely and properly evaluative data, it is impossible to set priorities responsibly. Unfortunately, the existing information and its management are inadequate to plan and set priorities for U.S. research. At present, the science policymakers in Congress who seek ways of setting these priorities get too much--and too little--help. While they are responsible for determining how best to spend a large, discretionary portion of the federal budget, they are frustrated by the lack of guidance available on how to understand and evaluate competing demands--and the competition can be fierce. While the policymakers realize that participatory democracy creates constituencies for every good cause and for every scarce dollar, today they must deal not only with various proponents of basic research in general, but also with separate lobbies for such "megaprojects" as the superconducting supercollider (SSC), space station Freedom, and the Human Genome Project. Congressional policymakers are inclined to see scientists as objective arbiters of empirical truth who value the use of information. Therefore, they trust that the scientists who advise them and testify before them have at their disposal a full set of systematically derived data upon which they can base the tough decisions they must make on research funding. Information alone, of course, does not compel its use. But it's a start. Most federal research agencies, with the exception of the National Science Foundation and the National Institutes of Health, devote few resources to internal data collection. Consequently, most analyses must rely on NSF and NIH data and indicators alone. This practice risks generalizing results and trends that might not apply to each agency's specific situation. Furthermore, it is not clear how the agencies' data are used to inform research decision making, since some information challenges current policy assumptions and other data are reported at levels of aggregation that are inappropriate for policymakers. Some information might be very detailed about a certain subspecialty--qualitatively elegant beyond the grasp of a nonscientist, but lacking in the quantitative detail that could inform sound fiscal decision making. In other words, while the scientists and science agencies may be thinking "micro," the policymaker may need to hear "macro." Foremost among the organizations that collect and analyze data on the research system is NSF, with its numerous surveys, reports, and electronic data systems. At its inception, in 1950, NSF was given the mandate to collect and monitor federal data on science and technology. Certainly the most visible compendium of data on the research system is its biennial report, "Science and Engineering Indicators." Since 1973, it has been issued by the National Science Board, the foundation's governing body. Additional sources for such information are other federal research agencies, the National Research Council of the National Academy of Sciences and the National Academy of Engineering, the Congressional Research Service, and various professional societies, especially the American Association for the Advancement of Science. Together, these organizations provide a wealth of information: historical pictures of research and development funding efforts; expenditures by specific R&D performers; data on sources of funding; the number of students who enroll in and graduate from programs in science and engineering; characteristics of K-12 science and mathematics programs as well as students who are participating in those programs; and the size, sectors of employment, and activities of the scientific work force. These data provide policymakers a basis for understanding the federal research funding system's sponsors and performers. But even with organizations that devote significant resources to gathering this information, the data are rarely matched to policy questions. For example, which agencies support which fields of research, and in what proportions? How many academic researchers are funded in a given year by each agency? What proportion of these researchers is seeking federal funds? And what happens to proposals that fail to win support? There are four categories of data that could be useful to federal decision makers, none of which, at present, is adequately gathered, organized, or disseminated: Research Moneys: Information is needed on such expenditures as salaries, equipment, and indirect costs for all institutions that perform federally funded research. Better cost accounting and forecasting for such megaprojects as the SSC are surely needed, as continuous upward revisions of cost estimates for projects disrupt decisions on setting priorities within and among fields. Personnel: The size, composition, and distribution, by sector, of the research work force remain crude approximations. Answers to such simple questions as "How many scientists are there?" depend on which definition and data collection method is used, and for what purpose. These data are important for estimating the health of the research system. Furthermore, although individual investigators may apply through their institutions for federal grants, their work is usually performed by teams of researchers. Process: Good data on how research is carried out are sorely lacking. One trend worth examining is the increasing size of research groups, both within the university structure and through federal support of interdisciplinary centers. This trend, which brings academic researchers closer to their counterparts in industry, has policy implications for the cost of research, its interdisciplinary capabilities, the changing demographics of the work force, and the aspirations of young researchers. Outcomes: Because of the fundamental and elusive nature of research, measuring its results is very difficult. However, there are some proxies. When seen as the creation of new knowledge, research can be judged in part by published papers. One tool that has been vigorously developed by measuring the outcomes of research is bibliometrics--the statistical analysis of scientific publications and their attributes. Citation analysis, carefully tailored to address specific research questions posed at an appropriate level of aggregation, can reveal patterns not otherwise discernible. The table on page 11 summarizes OTA's view of these categories, desirable methods of data collection for each, and the principal users of the information intended to help guide and monitor the federal research funding system. A combination of science indicators and other information on the changing parameters of the federal research system would be invaluable to policymakers. As Rep. George Brown (D.-Calif.), chairman of the House Committee on Science, Space, and Technology, has stated: "The conventional wisdom in science has not always served us well, and ... rigorous analysis will allow us to challenge the old myths and ensure true progress in our efforts to improve the research system." Drawing on NSF's expertise to make it the coordinating agency, Congress could instruct every research agency to develop a baseline of information. It could further direct the Office of Science and Technology Policy and the Office of Management and Budget to devise a plan to increase the reporting and use of agency data in the budget process, especially crosscutting information in priority research areas. The Federal Coordinating Council for Science, Engineering, and Technology has already been employed to carry out such analyses in the areas of global climate change, high-performance computing, and science and mathematics education. This mechanism seems to work, and could be more widely emulated. Because better data on the federal research system could be instrumental in the creation or refinement of research policies in the 1990s, it behooves researchers who serve as reviewers and consultants to federal agencies to scrutinize the empirical bases on which resource decisions are being made about their field. Analysis can then raise questions about the information that decision makers have at their disposal. Under almost any plausible scenario for the level of research funding in the 1990s, policymakers will address issues of planning, management, and progress toward national goals. Congress and the executive branch will continue to wrestle with scientific and other national priorities, as changing funding patterns and various pressures from both outside and within the scientific community test the federal research system. In such an environment, selective yet generous research funding will demand well-informed, coherent policies. Policies must respond to, and in some ways anticipate, the consequences of funding decisions on the research system. Improving both the measurement process and its routine use in setting research priorities can help to reveal opportunities and problems relevant to decision making at all levels of the federal government. Information on the research system, and the policy analysis that builds upon it, can, in short, warn about extrapolating the past to manage its future. Like science itself, information and analysis can specify our ignorance. Daryl E. Chubin and Elizabeth M. Robinson were, respectively, the project director and the principal analyst for the Office of Technology Assessment report "Federally Funded Research: Decisions for a Decade, " upon which this essay was based. FEDERAL RESEARCH SYSTEM CATEGORYDESCRIPTIONMETHOD OF DATA GATHERING PRIMARY USERS OF DATA Research Moneys Funding within and across fields and agencies Cross-agency information on proposal submissions and awards, research costs, and the size and distribution of the research work force supported Research expenditures in academia, federal and industrial labs, centers, and university/industry collaborations Agency allocations of costs within research project budgets, by field Megaproject expenditures: their components, evolution over time, and construction and operations costs Agency data collection (and FCCSET)Congress, Agencies, OMB, OSTP Research Personnel Size and how much is federally fundedSize and composition of research groups Agency SurveyAgencies Research ProcessTime commitments of researchers Patterns of communication among researchers Equipment needs across fields (including the fate of old equipment) Requirements for new hires in research positions Agency survey; onsite studies Agencies Research OutcomesCitation impacts for institutions and sets of institutions International collaborations in research areas Research-technology interface, e.g., university / industry collaboration New productions functions and quantitative project-selection measures Comparison of earmarked and peer reviewed project outcomes Evaluation of research projects / programs Bibliometrics; surveys of industry and academia Congress Agencies OSTP KEY: FCCSET - Federal Coordinating Council on Science, Engineering, and Technology; OMB - Office of Management and Budget; OSTP - Office of Science and Technology Policy

Sound Science Policy Requires Better Data Management

Interested in reading more?

Become a Member of