The Mires of Research Evaluation

All organizations who fund and conduct scientific research are increasingly "under the gun" to better evaluate the performance of their programs. Scientific research is supported by two major sponsors; the federal government funds most basic research and industry supports the more applied research. These sponsors have organizational goals and obligations to their stakeholders. They must account for their expenditures and must justify these investment decisions. How to do so in a viable yet acceptable manner has been an age-old question. In recent opinions and letters in The Scientist, Thomas J. Phelan1 argued for the use of bibliometric measures in the evaluation of institutional performance, whereas George Perry2 criticized the emphasis research universities put on gaining external funding as an indicator of research evaluation. The key issue in research evaluation is the combination of the scientific process being such a complex phenomenon, and our constant desire to quantify its outcomes. This has led to the use of multiple indicators, such as counts of publications, counts of citations, and peer-review. The vast majority of the problems discussed by Phelan, Perry, and other writers are primarily due to the issues inherent in measuring complex phenomena with an array of indicators. Products from research include a wide variety of outcomes, from "nuggets" of knowledge to an array of ideas, concepts, methods, techniques, and explanations. Downstream outcomes include patents, prototypes, and design criteria. Certain metrics (such as bibliometrics) can only capture a portion of these outcomes. Moreover, these outcomes (or products) are differentially prioritized by performers and sponsors and create inherent conflicts. Given that only a fraction of research is documented and captured by bibliometric measures, some, if not much, remains unmeasured. This creates a problem when we consolidate the use of bibliometrics with the broader concept of research productivity. Bibliometrics measure perhaps only about half of the scientific output produced by scientists. Consider the outcomes from research on the human genetic code. Currently we know the building blocks and some sequences, like letters in an alphabet. What we don't know, and are striving for, is how to form words, sentences, and even concepts with these letters. Such an effort is by design multidisciplinary, thus leading to complex and varied types of outcomes. As such, these outcomes cannot be adequately measured by a single type of metric. Bibliometrics as an Indicator of Scientific Productivity Bibliometrics include counts of publications and citation analyses and have long been a favored set of indicators of research outputs. In a recent book,3 I listed nine strengths or advantages of these indicators, grouped in three categories: structure, measurement, and representation. Under structure, bibliometrics can be applied to various levels, such as individuals, groups, institutions, and countries. The cost of collecting data and conducting meaningful analysis is relatively affordable, and these indicators have already been used and tested for validity. Under measurement, they allow for the quantitative assessment of research outputs, and their analysis is a relatively straightforward approach that relies on a small set of assumptions. Similarly, under representation bibliometric measures are helpful in charting the evolution of a scientific discipline, and that of individuals and research organizations. By convention, bibliometric measures have been accepted by the scientific community as valid representation of the outcomes from intellectual and inventive activities. However, bibliometric measures also have a number of problems and weaknesses. These can be grouped under: coverage, measurement, generalization, and biases. In the coverage category, bibliometric measures fail to include a host of other types of scientific communication such as letters and personal communiqués. They also fail to include intellectual contributions in technical outlets and work-in-progress. Bibliometric measures also suffer from the "Pied-Piper Effect;" they measure influence, not quality, and are selective of papers that "toe the line" and don't "rock the boat." Another weakness is the "Apples and Oranges Effect," whereby cross-disciplinary analysis of publications is difficult, because of different structures and procedures of the scientific investigation, ease and rate of publishing, and the nature of the peer-review process--all of which vary by discipline. Under generalization, bibliometric measures lack an inherent standard for validation as a measure of quality. We rely on convention established by a small and elite group of scientists within disciplinary boundaries. This is complemented by the problem of measuring publications over a relatively short time frame, so that citations are generally selective to within a few years of the focal paper. Finally, there are several biases, such as self-citations and citations of "friends," mentors, and students, and the biases inherent in selection and analysis of what constitute "key" papers and journals. All of these are weaknesses of bibliometrics that are built into the metric itself. Behavioral Problems in Applying Bibliometric Measures Like any other quantitative measures used in the evaluation of human activity, bibliometrics tend to suffer from at least two major problems. The first is the readiness of evaluators to ascribe to these measures a life of their own. Before long, these measures become a potent force by virtue of the numbers and statistics they produce. Comparisons are made between sets of numbers, without regard to their meaning, mode of collection, or implications for the person or organization under evaluation. This is the power that quantities seem to possess when they are employed in evaluation--the power to become independent metrics of the activity. A comparable problem involves the phenomenon of "corruption of the indicators." When a measure, particularly quantitative, is used in evaluation, people and organizations to whom the measure applies tend to radically change their behavior to adapt to the exigencies and peculiarities of the measure. If publications and citations indicate scientific performance, scientists will direct most of their effort to produce such quantities as required by the measure, and to concentrate their effort in generating bibliometric outcomes. Of course, this will be done at the expense of other activities. Research that toes the line and produces papers will be preferred to that which rocks the boat and may not be publishable. Back to Peer-Review Even when we consider the positive attributes of bibliometric measures, relative to their weaknesses, the end result is that for any credible quality assessment we end up with subjective evaluation. This leaves us with the only tool thus available: peer-review. Regardless of the many weaknesses of the peer-review process, I view it as "the thin line of control." Subjective assessment of quality by a group of selected knowledgeable people in a given scientific field remains the only operational control process designed to differentiate between poor and good science. The key to effective evaluation of scientific performance is the convergence of peer review and bibliometrics. Specifically, bibliometric measures should be the quantitative compass that guides subjective evaluation, thus lowering the incidence and the harmful effects of biases, jealousies, and intolerance shown by some reviewers, as well as "turf" protection and rejection of change in favor of established disciplinary paradigms. There is also an added value in converging the two metrics. The combination of quantitative and judgmental data offers a synergy, translated into a triangulation of benefits. Blending the two metrics helps to attenuate the biases in both metrics. Another aspect of synergy is the increase in validity, reliability of the metrics, and the confidence evaluators have in these assessments. When combined with quantities from bibliometrics, peer review is fortified with a factual picture of the phenomenon of scientific outcomes. We accomplish this convergence by making bibliometric measures a tool to direct and to enable effective peer review. This can be achieved by improving data mining and similar techniques that allow for a more comprehensive exploration of qualitative patterns and attributes in the bibliometric data.4 A complementary approach is to ensure that the process of peer review is continually audited, studied, and improved. This means that biases in judgment, reflected in an overall manner in bibliometric data, should be brought to the attention of the evaluator, for corrections and modification to the process. The scientific community must accept the fact that to evaluate the quality of research performance, there are no substitutes to a relatively unbiased and honest peer review, combined with bibliometric data. The operational problem is how to make this system less biased and, at the same time, how to weave quantitative data from bibliometric counts into this hybrid mechanism of evaluation. This is where we should concentrate our effort. Eliezer Geisler is Professor of Organizational Behavior, and Director, Center for the Management of Medical Technology at the Stuart Graduate School of Business, Illinois Institute of Technology. References 1. T. J. Phelan, "Evaluation of scientific productivity," The Scientist, 14[19]:39, Oct. 2, 2000. 2. G. Perry, "Research evaluation," The Scientist, 15[1]:6, Jan. 8, 2001. 3. E. Geisler, The Metrics of Science and Technology, Westport, Conn.: Greenwood Publishing Group, 2000. 4. R. Kostoff, E. Geisler, "Strategic management and implementation of textual data mining in government organizations," Technology Analysis & Strategic Management, 11:493-525, 1999.

The Mires of Research Evaluation

Interested in reading more?

Become a Member of