Research Misconduct: Media Exaggerate Results of a Survey

Falsification or selection of research data is an old source of controversy in science. R.A. Millikan's famous "oil drop" experiment to determine the mass of the electron, published in 1913, may have involved some data selection, and the British statistician R.A. Fisher's analyses of Gregor Mendel's genetic ratios from the 1860s suggested that they may have been too good to be true. Contemporary public interest in research misconduct has been stimulated by a popular book by two New York Times editors (W. Broad, N. Wade, Betrayers of the Truth, New York, Simon & Schuster, 1982) and by intensified media attention to several high-profile cases in subsequent years. The attention given to this subject has produced a rich folklore, but data on the actual frequency with which fraud occurs are sparse. In 1993, the Project on Professional Values and Ethical Issues in the Graduate Education of Scientists and Engineers of the Acadia Institute, in Bar Harbor, Maine, surveyed graduate students and faculty to assess their level of exposure to ethical misconduct (J.P. Swazey et al., American Scientist, 81:542-53, 1993). The results were widely interpreted by the media to suggest that such misconduct is common. The researchers themselves, while conceding that their methods do not permit them to measure the incidence of misconduct, do make assertions about its prevalence. The popular media accepted these conclusions, for the most part, without references to limitations reported in the study itself. Conclusions about the prevalence of fraud cannot be drawn from the survey with any certainty, but uncritical treatment of its findings in the media add to the public impression that research misconduct is a frequent occurrence (F. Hoke, The Scientist, 7[24]:7-8 Dec. 13, 1993). In the Acadia survey, questionnaires were sent to 2,000 doctoral candidates and to 2,000 faculty from 99 of the largest U.S. graduate departments in chemistry, civil engineering, microbiology, and sociology. Subjects were asked if they had observed or had "other direct evidence" of any of several specific types of ethical misconduct in their departments within the previous five years. Of the 4,000 surveys sent, about 2,600 were returned, and these may not represent an unbiased sample. Self-selection bias can have serious consequences; its magnitude is difficult to assess, but it is likely to have inflated the percentages reporting exposure. Numerous reports of the Acadia Institute's study appeared soon after the survey was published. These appeared both in the popular press and in publications geared toward scientists. Only one review, in a professional journal (C. Anderson, Science, 262:1203-4, 1993), considered the inability of the survey to control for self-selection bias as a major methodological weakness that undermines the power of interpretation of the results. The general media did not treat the problem of self-selection bias as an important factor, and stories invited readers to believe that the results of the survey accurately represent the views of the academic community at large. The Acadia authors acknowledge that their study could not measure the frequency of occurrence of various types of misconduct: The data provide only information about perceived misconduct. The authors present three main caveats in interpreting the results, of which two bear on the assessment of rates of exposure. The first is that the criterion (having had "other direct evidence" of misconduct) is ambiguous; it is not known how that was interpreted by respondents. The second is that several respondents (graduate students and faculty) in the same department may have reported the same incident. Thus, the survey cannot estimate rates of occurrence of perceived misconduct, but only the proportion of students and faculty who are aware of such cases. The third caveat is that respondents were asked about their experience with perceived misconduct, not about their experience with cases of misconduct that had been confirmed by any sort of investigation. Scientists under investigation by the Office of Research Integrity (ORI) are able to ask for a hearing; ORI has lost several cases on appeal and dropped its charges of misconduct in other well-publicized cases. The experience of others in reviewing cases of alleged misconduct is that the vast majority of cases turn out to be groundless. Thus, there is likely to be a large discrepancy between the number of reported cases of perceived misconduct and the number of actual cases. Despite having identified these limitations, the authors reach conclusions about the amount of actual misconduct that exists in academia. They claim their results "demonstrate that such problems are more pervasive than many insiders believe." They also claim that one could infer from their data that examples of science-related misconduct are not rare. The re-searchers claim that "reports of questionable re-search practices are far more common than reports of outright misconduct." They provide, as examples, the reported percentages of two types of questionable research practices; these happen to be the two highest values of 16 reported percentages of various types of questionable research practices listed in a figure. They also claim that "43 percent of faculty say they know of peers making inappropriate use of university resources for personal purposes. ..." What their figure actually represents is the percent of faculty who report awareness of at least one instance in the previous five years in which they believe that a colleague used university resources for outside consulting work or other inappropriate personal purposes. Criteria for "inappropriate personal purposes" were not defined; the instance presumably could have been a colleague using departmental postage to mail a personal letter. On the basis of such reports, the researchers imply that questionable research practices are common. Swazey and associates conclude with a finding on the respondents' cumulative exposure to misconduct: 44 percent of students and 50 percent of faculty report having been exposed to at least one case of each of two types of misconduct. They assert that these numbers suggest that a great number of researchers have experienced misconduct, and "that there are significant challenges to the integrity of academic science that reach directly into the research enterprise." Consider the following profile of a fictitious student respondent. Four years ago, she observed a fellow graduate student storing alcohol in a bottle, and contrary to standard procedures, failing to date the bottle's label. Two and a half years ago, a friend asked her if he could borrow a computer program she had written that performs a sophisticated statistical test, to save him the time of writing a new program. Last week, she received a questionnaire from the Acadia Institute. She interpreted the alcohol incident as a case of "ignoring research policies (animal care, human subjects, biosafety, etc.)," and she considered her friend borrowing a program that she had written as "trying to get by on the work of others." Other than these two cases, she reported having had no exposure over the last five years to any of the types of misconduct mentioned. The researchers would conclude from this student's questionnaire that she has had "cumulative exposure" to "ethically wrong or dubious behavior." It is worth emphasizing that the fictitious respondent above would fall into the group that has had the greatest exposure to misconduct (two cases). About half of the respondents have had even less exposure to any kind of perceived misconduct in five years. Anderson's report in Science, and an article that cites it (K. Davidson, San Francisco Examiner, Dec. 3, 1993, page A2), were the only news stories to criticize the researchers for making claims about the magnitude of misconduct that occurs in academic science from a survey of the number of respondents that were aware of cases of perceived misconduct. An essay in Society (M.C. LaFollette, 31:6, 1994) emphasizes that Swazey and colleagues' survey measured perceptions, and LaFollette avoids reaching conclusions about prevalence of misconduct. Sharoni Shafir is a postdoctoral researcher in the department of entomology at Ohio State University. E-mail: shafir.1@osu.edu. Donald Kennedy is Bing Professor of Environmental Sciences at Stanford University. E-mail: HK.DXK@forsythe.stanford.edu. Other authors accepted the researchers' conclusions about the high incidence of misconduct and cited the highest percentages from the survey. An article in the Washington Post (M. Gladwell, Nov. 14, 1993, C1-C2), the main thrust of which was that ORI could not defend in court its most recently publicized allegations of misconduct against several prominent scientists, nonetheless cited the Acadia Institute's survey as having "found that many of the nation's scientists have first-hand knowledge of laboratory misconduct." Science News (144:383, 1993) titled its short article "Scientific misconduct: Not so rare?" and stated that the survey reveals that it "may be far more common than previously believed." A Philip Hilts article in the New York Times (Nov. 12, 1993, A22) bore the headline "Misconduct in Science Is Not Rare, a Survey Finds". The article claimed that the survey "has shown that it is not as rare as many scientists have said it is," and a large caption in its center announced that "a gap emerges between what scientists say and what they do." Hilts's short piece in the "Week in Review" (New York Times, Nov. 14, 1993, E2) was headlined "Scientific Misconduct: Study Says Cheating is Not So Rare"; it claimed that the "study of misconduct suggested that it was far from a rare phenomena [sic]." Tommy Denton reported in the Sacramento Bee (Dec. 5, 1993, FO3) on "a sobering body of evidence gleaned from nationwide research that dishonesty in science is widespread." A short piece in the Chicago Sun-Times (Nov. 13, 1993, 8) stated that "misconduct in science is widespread, from overlooking sloppy data to outright falsification, according to a survey of university faculty and students." Pamela S. Zurer reported in Chemical and Engineering News (71:24-25, 1993) that "the researchers conclude that the pervasiveness of questionable research practices is greater than would be predicted from the few dramatic cases that are highly publicized" and does not offer any criticism of arriving at such a conclusion. Finally, Lawrence K. Altman's article in the New York Times (Nov. 23, 1993, C3) was headlined "Her Study Shattered the Myth That Fraud in Science Is a Rarity" and claims that the study "showed that misconduct was more widespread than many scientists have been willing to acknowledge." The issue of research misconduct has become a significant element in the formulation of national science policy, and the Acadia study was thus important enough to merit careful scrutiny by the media. Yet its treatment is a striking example of how a conclusion can become reified through careless media reporting. Most of the popular press ignored the problem of self-selection bias, failed to discuss the discrepancy between perceived exposure and actual occurrence of misconduct, and ignored the problem of multiple reports of the same perceived occurrence of misconduct. The most extreme figures from the survey were reported selectively, and most journalists uncritically accepted the researchers' conclusions. By their failure to address the very serious limitations of the study, the popular media have helped to amplify a negative and poorly substantiated description of ethical conduct in academic science. The consequences are not without cost: Such portrayals shape the public's image of academic science, suggesting that it is a system requiring stricter regulation. But rules governing ethical conduct in science, however well intended, may be detrimental to scientists, the research enterprise, and the public at large.

Research Misconduct: Media Exaggerate Results of a Survey

Interested in reading more?

Become a Member of