Critics Sharpen Assault On Peer Review

Some even recommend abolishing the traditional process used by science journal editors to evaluate submitted manuscripts During the past several years, the practice of peer review of article submissions--accepted unquestioningly by some authors--has come under increasingly harsh scrutiny by others. Many of these authors--joined, in some cases, by the editors of the very publications they are criticizing--are demanding change: Some are suggesting that reviewers' identities be revealed to the author; some, indeed, want to see the traditional peer-review system abolished altogether. Advocates of such changes fear that reviewers either are competitors of the authors whose research they are critiquing--and thus may have a vested interest in delaying the publication of their rivals' work--or are immersed in a subdiscipline at such a distance from the papers they are ostensibly reviewing that they may know nothing about the subject. Some also worry that, because publishing is essentially a buyers' market, there is no accountability on the part of editors. Editors themselves have their share of complaints about the system and reviewers, including their own. Among them: reviewers are not prompt, may be biased, or do not address the major arguments of the papers they are critiquing (see story on page 21). One of the most visible manifestations of concern is a call by some biomedical scientists for research into the peer- review process, which they say is central to all scientific communication, and yet poorly understood. Encouragement has come from the American Medical Association (AMA), which has provided staff support and funding for the First and Second International Congresses on Peer Review in Biomedical Publication, held in Chicago in 1989 and 1993. The first attracted 50 abstracts and the second more than 100, giving impetus to a third congress, planned for 1997. The increasing interest in the subject is hardly surprising, since it seems that almost every scientist can recall a horror story stemming from an experience with a reviewer. The abundance of such tales notwithstanding, it is difficult to get a spurned author to name the journal in question; for fear of harming their future publication prospects, most prefer to keep their experiences off the record. Take Loren Pankratz. His run-in with reviewers happened more than 15 years ago, but for Pankratz, a professor in the department of medical psychology and psychiatry at Oregon Health Sciences University in Portland, it seems like yesterday. Pankratz was a coauthor of a study that described the successful--and surreptitious--reduction of a narcotic administered to a chronic pain patient. He says that when his paper came out--in a journal he declines to name--to his surprise, it was accompanied by four critical commentaries, "some of which were horrified by what we had done [adjusting the dosage without the patient's consent]." The journal subsequently changed its editorial policy, and Pankratz was never allowed to pen a response. "So we thought we had used these wonderful strategies to help this guy cut down his narcotic abuse," says Pankratz, "but instead we got reamed out and we were left looking like manipulative guys." How often do situations like this occur? No one knows, because few studies of the peer-review process have been conducted. To confuse things further, peer review means different things at different publications. Some journals give reviewers explicit instructions, for example, and some don't. Others have statistical consultants, while most don't. And some hide the authors' identities from their reviewers, while others don't make the effort. So it's little wonder that anecdotal complaints abound from both editors and authors. But only in the past five years has biomedicine been encouraged to apply the scientific method to an examination of peer review. Such self-examination is a good thing, according to Drummond Rennie, West Coast editor of JAMA (Journal of the American Medical Association). Rennie, who has presided over the congresses, says he is pleased that AMA has recognized the importance of peer review to the quality of basic science and ultimately to patient care. "Scientific work doesn't exist," he says, "until it is peer reviewed and published." Given his recent experiences and interest in the subject, Rennie probably knows more about peer review than anyone in biomedicine. But, emphasizing his concern, he says that in a scientific sense he doesn't know much at all, because there hasn't been enough research into the peer- review process. Rennie, who is a professor of medicine at the University of California, San Francisco, Institute for Health Policy Studies, has heard all the anecdotes about peer-review abuse. Yet, absent systematic studies to confirm them, he says he would no more place stock in such tales than he would in unsupported claims made in any other area of science. How reliable is peer review? Do the oft-mentioned problems with the system affect the quality of papers that are published? "We don't really know," says Erica Frank, a professor of family and preventive medicine at Emory University and associate editor of Preventive Medicine, "because we haven't done the research." Stephen Lock, editor emeritus of the British Medical Journal, feels that part of the reason is that people were more frightened of editors in the past and didn't want to get on their bad side by criticizing their choice of reviewers. "Perhaps, too," he says, "there wasn't this culture of challenging things." He explains that particularly in the United States, as the competition for funding has increased and the emphasis in tenure and promotion decisions has shifted to the sheer weight of published articles, scientists have recognized that they have a larger stake in questioning the process when it doesn't go their way. In more relaxed times, argues Lock, since most papers eventually got published somewhere, "it was a bit of a lottery. A delay didn't matter too much, so you shrugged your shoulders; it was part of the fun." Frank has seen the problems as a researcher and editor. She thinks it's ironic and a bit embarrassing "that editors hold scientists to all sorts of stringent criteria, and yet we as editors have not held ourselves to any sort of criteria at all." She still steams over an instance in which she feels uninformed reviewers and an arrogant editor--of a journal she prefers not to name--rejected an article that she eventually published here. Her work was torpedoed, she says, by reviewers who didn't know the subject. Later, she says, when she spoke to the editor, he said, "I don't think we've ever made much of a mistake. I've never regretted having rejected a paper." One of the problems, says Frank, is that editors are usually chosen for their research and not their editorial decision- making skills. Another is that, generally, journal publishers have the upper hand in the author-journal relationship, and if something does go wrong, researchers "don't want to risk offending the editors, because they have to submit to that journal in the future." Taking the opposite view is Marcia Angell, executive editor of the New England Journal of Medicine. She thinks peer review works well as it currently exists. She says that it's the responsibility of good editors to keep the process running smoothly: The editor must choose reviewers who know what they are doing and ride herd on them to ensure that they don't subvert the process--for example, by sitting on papers. Angell says editors also have to read reviewers' comments closely to ensure they are not too subjective. And in the final analysis, the editor has to make the decision on whether to publish and not be totally influenced by reviewers' comments. Angell believes the journal marketplace already works well. The quality demands made by editors serve to direct papers to the publications where they belong. If a journal uses statistical consultants, for example, it will reject papers that contain statistical flaws and they will be published in less-selective journals, says Angell. She likes the fact that the process doesn't favor much innovation, too, because she thinks medicine should be conservative. But, she acknowledges, "there is nothing that an editor likes more than a breakthrough manuscript." David Horrobin, editor of Medical Hypotheses, has provided just such a home for a lot of ideas that he says would otherwise not have found one. The journal, which is peer reviewed, "will publish ideas or criticisms of ideas from any person, irrespective of whether any experimental testing of the ideas is then performed by the writer," it notes on its inside front cover. Horrobin thinks most peer review is concerned too much with quality control and not enough with nurturing innovative ideas. Says Horrobin, who heads the Efamol Research Institute in Kentville, Nova Scotia, Canada: "I think we're killing a lot of interesting ideas at birth." The scientific community is also making a mistake, says Horrobin, by fostering the illusion that scientists all review each other's work fairly. "Scientists are just the same as everybody " he says, "particularly in today's competitive environment." For this reason, he doesn't think much of anonymous reviews. "In any situation other than science," he says, "anonymous communications are regarded as worthless." A number of solutions have been proposed to the perceived difficulties with peer review, but as far as Rennie is concerned the only solution is research. He says he'd be willing to shuck peer review completely if research showed that there was a better way of doing science. "Just because it has been done before," he says, "doesn't mean it should continue to be done." That is not self-evident to Angell, who fears that research on peer review may lead to regimentation. She says each journal uses peer review differently and that is the way it should be. Angell thinks that in an attempt to study peer review--which is a subjective, qualitative process-- researchers will use trivial, quantitative measures. "This will tell you almost nothing about the quality of the peer- review process," she says, "but it's the kind of busy work that makes you feel that you are looking at something." Once this research is done, she suspects, the findings will be used to regiment journals--to get them all to do things the same way. She fears that research could lead to a leveling, in which poor journals become better as good ones become worse. Rennie is particularly puzzled by Angell's concern about regimentation. He says all that peer-review researchers want to do is find out what works best. If shielding reviewers' identities from authors, for example, is shown to produce better reviews, maybe more journals will adopt it, says Rennie, but there won't be any rules that require such blinding. "Journals can do what they damn well please," he says. Lock is hopeful. He says that many editors are dissatisfied with the present system and are enthusiastic about improving it. "I think we will get somewhere," says Lock, "but it will take time." At the moment there are a lot of strong feelings, with scientists taking positions for and against such things as anonymous reviewing and grievance mechanisms for authors. "Still, editors are an inherently opinionated lot," says Frank, "but opinion doesn't cut it on this. What we really need is more data." Paul McCarthy is a freelance science writer based in Honolulu.

Critics Sharpen Assault On Peer Review

Interested in reading more?

Become a Member of