The Under-Reporting Of Research Impact

For researchers to compete strongly for federal funds, the benefits from their work need to receive full accounting and be articulated clearly. The impending implementation of the Government Performance and Results Act of 1993 (GPRA) [Public Law 103-62] has begun to place even more emphasis on this research accounting requirement. Unfortunately, the present informal "system" for tracking and disseminating research products and downstream impacts has many deficiencies, resulting in a gross under-reporting of the broad range of research products, benefits, and outcomes. Historically, there has been no central mechanism for documenting impacts, and no collective will among the federal agencies and their industrial counterparts to expend the resources necessary for a full accounting of benefits. This problem is compounded by the lack of universal agreement on the definitions and scopes of research impacts, outcomes, and benefits; the types of studies necessary to ascertain and document these benefits; the total data that would be required to perform these studies; and the interpretation of the results of such studies. Long-term benefits of research are currently tabulated from Additionally, one research project intrinsically has multiple impacts on other projects (in the same area or different areas) and technology through myriad pathways. However, these indirect long- and short-term impacts and benefits of research are often overlooked. The indirect impacts tend to cross diverse disciplines, which complicates their tracking. The impact sequence is not necessarily linear from basic research to final product; there could be impacts from basic research to technology back to other research before a final product is produced. Finally, the more sophisticated information technology and databases required to systematically track these impacts did not exist in the past. There are two main types of systematic, quantitative approaches available to estimate the magnitude of these direct and (especially) indirect impacts: matrix and network approaches. In matrix approaches, numbers are assigned to the impacts that each of a selected group of research programs will have on each of a selected group of technologies, thus producing a research-technology matrix. These numbers could be generated by many techniques, ranging from subjective individual estimates to sophisticated econometric analyses. The matrix approaches then assign numbers to the impacts that each of the technologies will have on each of the end products (systems, operations, and so forth), thus producing a technology-systems matrix. All the matrices defined for the different steps relating research to the end product are then multiplied to arrive at a final impact relationship between the research programs and the end product. Thus, for a research-technology-systems sequence, the research-technology matrix is multiplied by the technology-systems matrix to produce a research-systems matrix. However, matrix approaches (such as B.V. Dean, "A Research Laboratory Performance Model" in M.J. Cetron et al., eds., Quantitative Decision Aiding Techniques for Research and Development, London, Gordon and Breach, 1972) can account mainly for forward impacts: the impact of a research program on a variety of technologies and subsequent impact of these technologies on a variety of systems. While these forward impacts represent only the tip of the iceberg of total research impacts, even these limited matrix approaches are rarely used. Network approaches assign numbers to the impacts that each research program, technology program, and system has on every other research program, technology program, and system. Network approaches (such as R.N. Kostoff, "Research Impact Quantification," R&D Management, 24 :3, July 1994) can account for forward, lateral, and backward impacts: the impact of a research program on other research programs and other technologies and subsequent impact of these technologies on other research programs, technologies, and systems. Network studies have shown that including these types of indirect impacts in the computational process can result in potential orders of magnitude increase in total value of quantified impacts. The massive increase in value is due to the summation of an extremely large number of modestly sized indirect impacts. The under-reporting of indirect impacts stems from the lack of data needed for the matrices and networks, from lack of a coordinated research tracking system integral to the research execution and transition process. This lack of coordination among all the principals in the national research enterprise contributes to an under-reporting of the full research benefits. This could result (and already may have resulted) in research receiving less funding than is warranted by the full scope of its socially useful benefits and impacts. Research product tracking and monitoring should not be treated as an afterthought, but rather be made an integral part of the funding process, which progresses along the following steps: research planning selection funds outlay execution transition to technology/end product evaluation process What type of research product tracking system should be developed? The system should have the capability of tracking long-term as well as near-term research impacts. It should be able to follow indirect impacts of research as well as direct impacts. The system should be simple to operate, should not require substantial resources from the data providers or the system maintainers, and should cover as broad a spectrum of development categories, sponsors, and users as is possible. For ease of introduction, the system should have some basis in an existing process, where there is a substantial body of operational experience. One very limited prototype of such a system is the Science Citation Index ® (SCI) in Philadelphia. Through its manipulation and tracking of references in papers, it is able to follow the flow of information over time and the evolution and impacts of research. However, for the research product tracking purposes suggested here, the present structure of the SCI has severe limitations. It is focused on basic and applied research only and does not span the gamut of research to technology product. It does not contain sponsor information or funding information, and in many cases it does not contain any mention of potential applications. The SCI dies not contain unique representations for performers and organizations. The name John Smith could appear on a thousand different cited documents, and the analyst has no simple way of knowing whether this represents one John Smith or a thousand different individuals with the same name. Would the credit card companies give identical cards to all the John Smiths in the world? Why should the SCI? This latter problem is more than one of appearances. Much sponsor credit can be under-reported because of the errors and ambiguity of performer and organization information (R.N. Kostoff et al., "Database Tomography for Technical Intelligence: A Roadmap of the Near-Earth Space Science and Technology Literature," Information Processing and Management, 34 :1, 1998). This reference provides examples of the multiple organization abbreviations found in the SCI for the Naval Research Laboratory and the Goddard Space Flight Center.) Equally important, even in the case of examining impacts on basic and applied research, there are severe problems with the SCI. These problems stem from the structure of the basic SCI unit, the published peer-reviewed research paper. The typical paper focuses, in priority order, on research approach, research product, and intellectual heritage (references). This focus derives from performer priorities, not sponsor tracking priorities. The completeness of the references, the adequacy of the references, and the relative importance of each reference are governed by the performer's subjectivity and the limited space available for the paper. In particular, under the present highly competitive climate for research funds, how motivated are researchers to give more credit than absolutely necessary (in print) to the origins of new concepts or paradigms? Thus, the present structure and design of the research paper is not the optimal structure required for tracking. The SCI can be viewed as a beta test prototype for a possible expanded system to address the needs of tracking broader research impacts. The proposed system would cover the range from basic research to product development and testing. It would consist of a science tracking component and a development, engineering, and testing component. It should be viewed as a first step in the improved tracking and documentation of research benefits, not as an ultimate solution. In particular, it is limited to tracking the evolution and technology transfer of that segment of research that has been documented in the open literature, and will therefore not include the tracking of proprietary, classified, and other types of unpublished research. The system could include the following components: Science Component: The science component would be an expanded version of the SCI. It would contain additional journals, sponsor information, and funding information (resource expenditures covered by the paper), and would uniquely and unambiguously identify the performers and their institutions. There could be a code or number uniquely associated with each performer, and there would be one representation for each organization. Some idea of relative importance of the references would be provided. Other useful information could be supplied as well. Modification of the SCI in the manner suggested would require the cooperation of the journals, since they would have the responsibility of requesting this additional information from the authors. The journals would also be requested to have their peer reviewers assign more importance to the completeness and prioritization of the references and would transmit this requirement to the authors as well. Development, Engineering, and Testing Component: This component would consist of one or more databases that would have citations and citation tracking similar to the modified SCI proposed above. The documents in these databases would not be limited to refereed published papers; they could include patents, nonrefereed reports and published papers, book chapters, and other documents that contain references. Each category could have its own database, or there could be combinations of categories in specific databases. Potential Studies: Construction of such an expanded system is possible now because of the advances made in computer speed, storage, and information-manipulation algorithms. Implementation of this expanded citation tracking system would allow long- and short-term impacts of research to be followed. These studies would not be a substitute for expert involvement in retrospective studies but rather would serve as directional maps or guides that allow the experts to identify and probe the different impact pathways. The capabilities inherent in this process would allow the indirect impacts of research to be documented over many pathways, and the full benefits of basic research to be collected and articulated more thoroughly. In conclusion, the full scope of the impacts and benefits of research needs to be articulated better, for researches to receive their fair share of federal and corporate funds. The technology now exists to track and account for many of these benefits. Joint planning between government and industry is required to implement these technologies for research impact accounting. This activity should be started now. Ronald N. Kostoff is manager, data mining, at the Office Of Naval Research in Arlington, Va. The views expressed here are those of the author, not of the Department of the Navy or any of its components.

The Under-Reporting Of Research Impact

Become a Member of

Meet the Author

Ronald Kostoff

Published In

September 1998

Related Research Resources

How Do Embryos Know How Fast to Develop

Research Resources

Podcasts

Webinars

Videos

Infographics

eBooks

The Advent of Automated and AI-Driven Benchwork

Become a PCR Pro

Genetic Insights Break Infectious Pathogen Barriers

Navigating Cold Storage Solutions

Products

Product News

Sapio Sciences Makes AI-Native Drug Discovery Seamless with NVIDIA BioNeMo

New DeNovix Helium Nano Volume Spectrophotometer

Olink® Reveal: Accessible NGS-based proteomics for every lab

Zymo Research Launches the Quick-16S™ Full-Length Library Prep Kit