Journal Referees Report That Authors Call Many Of The Shots

Publish or perish. It's a dictum drummed into the heads of scientists from the time they begin their studies. Jobs, grants, and promotions--everything of possible import to the budding academic scientist--hinge, to a great extent, on getting one's research into print. But self-advancement also goes hand-in-hand with the progress of science, for publishing papers is the prime means by which important findings are disseminated within the scientific community. To publish a paper in a journal, where it can be studied and cited by other researchers, is to make a permanent contribution to the fund of knowledge. In light of the pressure to publish felt by most investigators, it's only natural for them to view with some trepidation the system by which journals evaluate papers. Cloaked in confidentiality as it generally is, the process of refereeing can seem altogether mystifying, like some sort of cabalistic rite. But in reality, there's nothing particularly mysterious--or sinister--about what happens to a manuscript once the hopeful author deposits it in the mail. Editors and reviewers say they approach their jobs not as authors' adversaries, but as colleagues. Stanley Brown, an editor at the journal Physical Review, speaks for many when he says, "We bend over backwards for authors." Statistics bear this out. While an interdisciplinary publication like Science may reject about 80 percent of the manuscripts it receives, most journals are considerably less picky. In fact, a 1971 study by Columbia University sociologists Harriet Zuckerman and Robert Merton found that major physics and chemistry journals typically accept between 70 percent and 85 percent of submissions. In 1988, another sociologist, Ohio State University researcher Lowell Hargens (then at the University of Illinois), conducted a similar study and came up with roughly the same numbers. Even if a paper is spurned by one journal, the odds are good that it will find a home in another. "Almost everything that's written gets published eventually someplace," according to psychologist Stevan Harnad, visiting fellow at Princeton University and editor of Behavioral and Brain Sciences. When a manuscript arrives at a journal, it's usually examined by the editor, or, at publications with a high volume of submissions, farmed out to a member of the editorial board who is conversant in the paper's subject. The paper may be rejected outright at this stage if it's obviously flawed or deals with a topic clearly inappropriate to the journal. Otherwise, the primary purpose of the first reading is to determine who should review the work. Although cynics may suspect otherwise, editors do not select reviewers by an arbitrary method like sticking pins in lists of names. Nor do they automatically turn to the foremost authorities in a field from a coterie of acknowledged experts they keep on tap. Rather, the decision is typically based on the references cited in the paper as well as the editor's own knowledge of the field. And while the higher-profile scientists undoubtedly get asked to review papers more frequently, editors say they do not consider reputation alone when selecting referees. Sometimes, the fact that certain scientists make good reviewers "has nothing to do with their degree of knowledge in a field," says Allen Bard, editor of the Journal of the American Chemical Society (JACS) and a professor of chemistry at the University of Texas, Austin. "It's simply that they're thorough and careful. And others who can be very well-known figures in science are terrible reviewers because they won't take the time and make the effort." Many journals also maintain some sort of referee database to help editors match manuscripts to reviewers. For instance, JACS keeps online dossiers on more than 8,000 reviewers, which can be accessed via name searches to generate lists of scientists with expertise in any given area. "Most of the time I know from reading a paper who the reviewers should be," Bard says, "but [running a name search] is a good way to jog your memory and provide you with a wider [selection] of reviewers." Suggestions from the papers' authors are also considered and even welcome, according to Brown. "If [the recommendations] are appropriate, we may use them," Brown says. "We offer authors no guarantees, but [their advice] is often useful, particularly for very specialized or controversial areas, where it may be difficult to find the appropriate person" to evaluate a manuscript. By the same token, editors will also try to honor authors' requests not to use a particular scientist as a reviewer, unless the paper's content warrants otherwise--for instance, if the manuscript is extremely critical of the other researcher's work. Even under these circumstances, says Bard, "we wouldn't send it to that person without the author's permission. We usually go back to the author and say, `Look, we would like to have that guy's opinion. We think it's only fair that he have a chance to comment on this.' But we will also send it to some other, impartial reviewers." In most cases, journals strive to expedite the evaluation process and often try to respond to authors within 30 days of a manuscript's receipt. To ensure a fast turnaround, assignments are often made by phone, fax, or electronic mail; referees are customarily expected to deliver their reports within two weeks of accepting an assignment. The number of referees assigned to a given article tends to vary with the discipline. In astronomy and physics, journal editors usually consult one reviewer, while two is the norm in chemistry and biology. As far as conducting the review is concerned, there are no hard and fast rules that the entire publishing community abides by. Although some editors supply reviewers with detailed guidelines and may ask them to complete an evaluation sheet rating the paper in specific areas, such as originality and logic, a more informal approach generally prevails. For the most part, it's understood that editors want about a page's worth of comments on the manuscript's scientific validity and whether it should be published, along with recommendations for possible revisions. Calling Them As They See Them Within this general framework, however, the referee's mission can differ in key respects, depending on the publication's editorial agenda. At journals that cater to specialized audiences, the referee's primary chore is to weed out papers that are faulty or fail to address a meaningful issue. But for a distinguished journal with a broad readership, such as Science, reviewers are required to do more. "Some journals want to get an indication of how exciting or not exciting a paper is," says Erkki Ruoslahti, scientific director of the La Jolla Cancer Research Foundation in California, who holds a seat on Science's editorial board and also edits the journal Cell Regulation. "The Proceedings of the National Academy of Sciences wants to know if a paper is in the top 10 percent of papers in that particular area. Science has a similar [criterion]." By necessity, the scope of the referee's analysis is limited. "You normally can't tell from the paper whether the data itself is right or not," says Robert Bergman, a chemist at the University of California, Berkeley. "You pretty much have to trust the guy's observations to be right." A frequent reviewer for JACS and other chemistry journals, Bergman notes that he will check the experimental section of a paper carefully to make sure he has "a feeling for how the experiments were done and whether they were done properly." JACS' Bard also attaches considerable importance to authors' discussions of their experiments. A paper should provide "enough results to convince the reviewers and the readers that the hypothesis or claims [the author] is making are backed up by experimental detail," he says. Another central concern is that the author adequately references the appropriate literature and gives credit to other scientists where due. "If anything gets reviewers or readers angry, it's that their work, which they think is germane to a given paper, has not been adequately discussed or referenced," Bard says. In his capacity as editor or reviewer, Ruoslahti pays close attention to the introduction. "I find introductions quite important. If they're well done, then one can very quickly figure out what [the paper] is all about," he says. The introduction, he says, should clearly delineate the paper's rationale and also define the questions that it will ask and then answer. "Of course, those have to be the same thing," Ruoslahti adds. "Whatever questions you pose you need to answer; otherwise the paper isn't fulfilling its purpose. Making claims that aren't proven--that's a sure way of getting in trouble." Bergman says that when he judges a paper, he tends not to consider the caliber of the writing. "You should be able to differentiate the quality of the science from the quality of the writing," he says. On the other hand, he points out, a poorly written paper can thwart the very purpose it intends to serve. "Often I had papers in which I thought the science was really good, but I've just been frustrated by the writing; and I'll write a review saying the value of this work is going to be missed because the writing is so confused," Bergman says. In such a case, Bergman adds, he might offer some suggestions on how to improve the clarity and organization of the paper, but the business of rewriting the manuscript remains the author's responsibility. A Twofold Mission This last comment points out the dual function of referees. Besides serving as judges, they are also expected to play the part of advisers, lending authors the benefit of their expertise for the purpose of improving the manuscript. "A lot of authors fail to appreciate that the referee is not there just to give a thumbs-up or -down to a paper," says Brown, who estimates that about 50 percent of the articles that appear in Physical Review are improved by reviewer comments. Bard seconds Brown's assessment. "It's fairly rare for a paper to be accepted without some revision" based on the recommendations of the referees, he says. After a manuscript passes through peer review, it becomes the editor's job to examine the referees' reports and render final judgment on the paper. When referees disagree about whether a paper deserves to be published, it may get sent out to another reviewer, or the editor may arbitrate the matter. Bard acknowledges that there can be a high degree of subjectivity in the editor's final reckoning, particularly when it comes to deciding whether a paper is sufficiently interesting and important to appear in a prestigious publication. "A lot of things become a judgment call, especially in JACS," he says. For that journal, his decision about a submitted manuscript must take into account whether it "would be more suitable for a specialty journal, and vice versa." Sometimes, an article may be accepted only on the condition that all requested changes be satisfactorily completed; in other cases, a paper may have to repeat the review process after revision. Yet authors are not obligated to meekly comply with every change editors and reviewers recommend. Editors realize that researchers may have compelling reasons not to follow a reviewer's advice. Still, Ruoslahti says, nothing annoys him more than an overly argumentative author. "As an editor, I don't usually like rebuttals [from the author]," he says. "People sometimes send in a five-page letter arguing every point. My advice to all authors is to sit down and try to respond to every point the reviewers have made. Occasionally, you may get unreasonable reviews, but even then it's a good idea to try to be responsive, and not just argue." Along similar lines, authors of rejected papers who believe their submission failed to get a fair hearing can usually appeal the decision. At Physical Review, for instance, authors have the right to petition the editorial board, Brown says, adding that editors of his journal are always willing to consult another referee for a second opinion. Yet Stevan Harnad cautions against appealing every rejection one receives, especially when dealing with prominent journals that may turn down a paper for editorial reasons unrelated to the quality of the science. "There's no point wasting time trying to force your way through the doors of a journal that's got plenty of submissions anyway. The sensible thing to do is to move to the next journal in the hierarchy," he says. Reviewing The Reviewers Although it's the editor who ultimately determines the fate of a manuscript, peer reviewers still wield an enormous amount of influence over the decision to publish or reject. And since their identities remain hidden from the scientists whose work they are judging, reviewers also enjoy a certain level of unaccountability for their actions. But editors are not blind to these facts, and most of them make an effort to monitor their referees. For instance, JACS editors grade the referee reports and can enter this information, as well as other comments, into the journal's computerized reviewer files. At Behavioral and Brain Sciences, Harnad goes a step further by asking authors to evaluate reviewers' reports on their papers. Efforts to police the system notwithstanding, many editors and scientists have begun to question the effectiveness of peer review. Critics contend that the value judgments of referees are shot through with biases, particularly against negative results or innovative work that doesn't square with the current dogma. In a study on peer review published in the March 1991 issue of Behavioral and Brain Sciences, researcher Domenic Cicchetti confirmed what skeptics have argued all along: that reviewers lack a clear consensus about what constitutes a publishable piece of work. Harnad acknowledges that the system has its weaknesses. But with so many journals around clamoring for material, he feels that complete objectivity and consistency are not imperatives. "If a paper doesn't get published in one journal, it gets published in a lower-prestige journal. In a situation like this, you don't have to worry a great deal about biases in the peer-review system because if everything gets published, it's not like works of genius are being lost. The evaluation system is simply channeling towards the more high-profile journal." If peer review tends to promote a conservative mindset, Bard says, that's because the scientific process itself is inherently conservative, which is as it should be. "The more controversial and outlandish the claim, the more carefully it has to be looked at," he says. "I think that's why papers that are more routine will be handled quicker and easier than papers which really try to break new ground." Harnad agrees that innovative work may run up against more resistance, at least initially. But eventually, he says, the research will find an appreciative audience "If your work is important," he says, "then every possibility is open to you." Jeff Seiken is a freelance writer based in Pittsburgh.

Journal Referees Report That Authors Call Many Of The Shots

Interested in reading more?

Become a Member of