Opinion: We Must Demand Evidence of Peer Review

Peer review varies in quality and thoroughness. Making it publicly available could improve it.

By Nikolai Slavov | May 21, 2018

ISTOCK, TEMMUZCANHave you read a paper and thought: “How could peer reviews support the publication of such a paper?” I have. More than once. Other times, I have read fascinating papers outside of my field and wondered what the concerns of the experts who peer reviewed the study were. What important caveats am I missing?

Sometimes, I am lucky and find the answers to such questions: A few publications, including those from EMBO Press and eLife, publish the peer reviews alongside the papers. Reading such peer reviews has provided an additional dimension of appreciating and understanding the experiments and the findings, especially when I am not very familiar with the topic. But for most other journals I cannot access the peer reviews that supported a paper’s publication because most journals hide them. 

The exit from this very sorry state is for journals to publish the anonymized peer reviews.

Scientific rigor demands that claims be substantiated by evidence. If I claim that gene A regulates gene B and provide no evidence, my claim will be dismissed. It must be dismissed. Yet, if a journal claims to conduct peer review and provides no evidence of it, the claim is rarely dismissed.

How do we know that a journal conducts peer review? For most journals, the evidence is limited to our anecdotal experiences with the manuscripts that we review ourselves or that we and our friends have submitted. For me this evidence is mixed. I know of manuscripts that have been thoughtfully reviewed and manuscripts that have undergone very expedited peer review or no peer review at all before appearing in the most prestigious journals. This anecdotal evidence is rather weak. If you ask me to substantiate it, I have to refer you to a friend who may or may not be willing to tell you that his or her paper was barely peer reviewed. It is a huge problem that the evidence for such a centrally important process is hidden from public view.        

The evidence for the quality of peer review and editorial oversight is even weaker. How can we evaluate the rigor of peer review at a journal that provides no public evidence that peer review happens? We cannot. The only scientifically justified conclusion is that we must doubt the existence and quality of peer review for any journal that does not publish the editorial and peer-review discussions that support its publishing activity.

The exit from this very sorry state is for journals to publish the anonymized peer reviews. Some journals do this, but they are the exception. The majority of the leading journals provide no evidence whatsoever of their peer review—or the lack of it. Yet peer review is currently the most important function of journals, and I believe that peer-review has much to contribute to the scientific discourse.

I can understand strong arguments for and against signing peer reviews. These arguments form a complex discussion without a simple solution. This discussion must not spill over and complicate the simpler question of whether anonymized peer reviews should be published. I believe they must be published. I see no justification for any journal to hide the anonymized peer reviews.

Nikolai Slavov is an assistant professor of bioengineering at Northeastern University. He is an academic editor at PeerJ.

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: JC States

JC States

Posts: 4

May 22, 2018

I absolutely agree. Too many times, I have read papers that appear not to have been reviewed rigorously. Poorly reviewed publications often add confusion to the literature by spreading conclusions that are inadequately supported by rigorously acquired data,  rather than moving a field forward. Too often, it appears the reviewers never examined the methods sections to make certain that appropriate methods were used and used appropriately! For instance, in biomedical sciences, Western blots are often published without showing proper controls. The upshot is that Western blots are often the root cause of retractions because they are often fabricated or otherwise 'massaged'. Experimental replication is often not clear. Reviewers generally are volunteers and do not get compensated for their work. Additionally, quality review takes time away from one's 'day job'. The pressures of modern research do not support performing quality reviews.  Something has to be done to reward quality review. 

Avatar of: Salticidologist

Salticidologist

Posts: 58

May 22, 2018

I do agree with this essay, but It might be better to drop peer review altogether and just append reader responses (subject to editorial review).  Anyone who really understands a field of study can easily see what papers are well-done, and what papers are poorly-done.  I just read a "peer-reviewed" paper published this year in Nature that was filled with errors and an uninformed discussion from cover to cover.  There is no substitute for an intelligent reader.  Peer review is just an attempt by publishers to show that they are value-added in an era of open-access publishing at all levels.  Now anyone has the resources required to publish, and peer review is just one more archaic concept.  Editorial review is a different matter, and it is a prerequisite for quality.  But, it will never guarantee quality.  Authors who want to improve their work can certainly improve it by consulting with others in their field, and there needs to be more of this (and less secrecy).

Avatar of: kammann

kammann

Posts: 1

May 23, 2018

Peer review must be absolutely kept alive, it is an indispensable element in scientific regour, despite of all its faults. We should intensifiy the review process and openly critizise the many omics journals for maintaining only a pseudo-peer-review. This way, we will have even more nice looking prints of scientifically doubtful papers with questionable production rules in meaningless pseudo-experiments and methodology and faulty printing.

Popular Now

  1. Dartmouth Professor Investigated for Sexual Misconduct Retires
  2. Two University of Rochester Professors Resign in Protest
  3. Theranos Leaders Indicted For Fraud
    The Nutshell Theranos Leaders Indicted For Fraud

    Federal prosecutors filed criminal charges that allege the company’s promise to revolutionize blood testing swindled investors out of hundreds of millions of dollars and put patients in danger.

  4. Laxative Causes Long-Term Changes to Mouse Microbiome