Citation Data Is Subtle Stuff

When starting to compile citation data from the scientific literature over 25 years ago, I aimed to create a new tool for information retrieval—the Science Citation Index (SCI). Out of this came a useful by-product: a large and ever increasing database containing indicators of intellectual connections among scientists and their publications. The SCI attracted the attention of historians and sociologists of science and served as a catalyst to the field of scientometrics, which uses quantitative methods to analyze the process and development of science. The ISI database has facilitated large-scale quantitative studies of the scientific performance of countries, institutions, fields, departments and individuals. In recent years, such scientometric studies have even contributed to public policy decisions in science. However, it is in assessing the performance of individual scientists, especially in the context of promotion or grant decisions, that the use of citation data is most controversial. This controversy has arisen for a number of reasons—none more basic perhaps than a common aversion to the impersonal judgment of numbers. Beyond that, however, much of the criticism of citation measures is a response to "quick-and-dirty" citation-counting, a thoughtless practice that reveals little more than an amateur at work and gives citation analysis a bad name. I have repeatedly warned about the mistaken conclusions drawn from the crude manipulation of citation data. Recently (The Scientist, February 23, 1987), I emphasized the distinct difference between "the simpleminded counting of articles or citations as indicators of quality and the in-depth analyses that can and should be carried out." This statement warrants some expansion. Here, then, is a primer on how to evaluate scientists using citation data. First, you'll need a complete and accurate bibliography of the candidate's publications. The SCI Citation Index lists all cited papers and books under the first author's name. Using the bibliography or full CV will ensure that all co-authored publications are included in your collection of citation data. (If a bibliography or CV is unavailable, a search through successive years of the SCI Source Index, which cross-references secondary to primary authors, can provide a reasonably complete list of papers.) A complete bibliography resolves a second problem: homographs. For example, in the SCI the heading SUZUKI T includes the publications of many authors. Individual works, however, are clearly identified; journal title, volume, year and page information separate citations unambiguously. Having obtained the required data, you have completed only the first, elementary step in citation analysis. You have determined which publications are cited, by whom, and how often—but not why they are cited. Nor does this show what these citations mean. The chief difficulty (and responsibility) resides in the interpretation of these data. To assess the individual fairly, some idea of the comparative performance of his or her peers is needed. Since publication and citation rates vary widely from field to field, you'll need to know what is typical in the candidate's area, as well as for persons of similar experience. Other aspects to consider include: the extent and nature of self-citation; the chronological distribution of the citations; whether citations are concentrated around a few papers (especially those on methods) or dispersed among many; the extent to which citations are cross-disciplinary or international; the quality and impact of journals from which the citations derive; and, of course, the SCI's coverage of the individual's field. An "in-depth analysis" also requires asking why citations have been given. With the citing papers in hand, you can engage in content and context analysis. Content analysis reveals what idea or fact in a publication is being referred to, while context analysis reveals the judgment of the citing author— favorable, critical or whatever. Clearly, such interpretative work is best done by someone knowledgeable in the field of the scientist under review. Citation analysis demands further probing questions. Given the varied nature of citations, many critics have asked, and rightly, what after all is being measured? An in depth analysis such as I have described will go far toward answering that question, but I would add this caution: although many studies have shown a significant correlation between citation analysis and peer judgment, citations are only indicators of influence and impact; they are a partial reflection of the interests of the academic community and the visibility of a person's work. They say nothing about intrinsic value. That is the role of human judgment. Furthermore, judgment is necessary to understand why a publication is relatively or completely uncited. Relatively low citation-frequency may signal that a publication was superseded by another or that its impact was so profound that it underwent "obliteration by incorporation," a process in which the substance of a work becomes part of the common understanding and explicit citation is deemed unnecessary. (See R. K. Merton, Social Theory and Social Structure, 1968, pp. 27-29, 35-38.) Uncitedness, on the other hand, occurs in instances of premature discovery—of work ahead of its time—and not only for work of low utility. "Citation data is subtle stuff," I wrote in 1979. "Those using it to evaluate research performance at any level, but particularly at the level of individuals, must understand both its subtleties and its limitations. The position of those who advocate the use of citation data to evaluate people is not that it is simple and foolproof, but that problems associated with it can be solved satisfactorily with a reasonable amount of methodological or interpretative effort." (Citation Indexing, 1979, p. 241.) Eugene Garfield is Publisher and Editor-in-Chief of The Scientist, and President of the Institute for Scientific Information, Philadelphia, PA 19104.

Citation Data Is Subtle Stuff

Interested in reading more?

Become a Member of