Concerns Raised Online Linger

Users of post-publication peer review forums like PubPeer often question perceived inaccuracies in scientific papers. Are the journals that published those papers paying attention?

By | August 25, 2014

FLICKR, MINISTERIO TIC COLOMBIAWhen its anonymous founders launched PubPeer in 2012, they intended to create a space for open, forthright discussions of the scientific literature. But they were surprised at the volume of posts—from other anonymous users, as well as those posting under their names—pointing out seemingly significant issues with published papers. “The great majority of comments point out some kind of problem,” PubPeer moderators wrote in an e-mail to The Scientist. “And a large proportion of those seem potentially serious to us—e.g. duplication of gel bands that would rarely occur by accident.”

Some papers that have been criticized on PubPeer have since been corrected or even retracted. But in other cases, critiques remain on the site for many months without any apparent resolution.

For instance, Paul Brookes, a mitochondrial researcher at the University of Rochester Medical School in New York, this February commented on a 2011 PLOS Biology paper on gut microbiota and insulin resistance, noting that it had then been around seven months since an anonymous poster first raised concerns about the work in a comment on the journal’s website.

The PLOS Biology commenter said there appeared to be “some errors” in Western blot figures, including similar-looking control bands meant to represent different conditions. The paper has been cited 66 times, according to the journal’s publisher, the Public Library of Science (PLOS). A day after Brookes posted on PubPeer in February, PLOS user “caricilli”—presumably the paper's first author Andrea Caricilli, who completed the work as a graduate student at the State University of Campinas in Brazil—responded to the commenter’s note, explaining that the journal and authors were looking into the issues. Caricilli did not respond to The Scientist’s request for comment.

Brookes most recently posted to PubPeer in late July about the apparent lack of resolution of the issue. “Stuff is just taking way too long to be corrected,” Brookes recently told The Scientist.

“PLOS is actively working toward a resolution, which will be made public when reached,” wrote PLOS’s public relations manager, David Knutson, in an e-mail. “However, this process does take time and is confidential.”

Corresponding author Mário Saad of the State University of Campinas told The Scientist that he and his colleagues have spotted errors in their figures and submitted a correction to the journal around March. “We recognize that there are unintentional mistakes in figures of that paper that we attribute to a not very rigid process of editing the manuscript by the authors,” Saad wrote in an e-mail to The Scientist. “These mistakes do not change one line of our manuscript, which was reproduced many times in our lab, and similar results have been reproduced over and over.”

Of course, in some cases, comments made on papers go unnoticed by the journals that published them.

In a separate thread posted to PubPeer in February, an anonymous commenter pointed out that two sets of data in a June 2011 Nature Protocols paper appeared surprisingly similar. The lead author was RIKEN’s Haruko Obokata, whose work on stimulus-triggered acquisition of pluripotency (STAP) stem cells was retracted earlier this year.

However, it was not until earlier this month that Nature Protocols noticed the comment after University of Tokyo seismologist Robert Geller sent the journal’s editors an e-mail pointing out the apparent irregularities after friends had alerted him to the PubPeer comment.

“Thank you for the note and for flagging up the issue, which we were not aware of and are now looking into,” Nature Protocols responded to Geller on Twitter last week (August 18).

In an e-mail to The Scientist, Veronique Kiermer, executive editor at Nature Publishing Group, explained that while it is not Nature’s policy to monitor PubPeer, “we investigate all serious allegations that are brought to our attention, including anonymous reports.”

Editors at Cell and PNAS also told The Scientist that they do not have policies in place to monitor comments made on PubPeer. “We have found the concerns in PubPeer to be mixed in terms of accuracy,” wrote Daniel Salsbury, deputy executive editor at PNAS.

“Some of the editors may monitor PubPeer,” wrote Science’s executive editor, Monica Bradford. “However, we have not set up a routine, consistent process for monitoring the site.”

Knutson of PLOS noted that his organization does “keep an eye on PubPeer, but we would prefer that readers alert the editors directly about any concerns they may have about PLOS publications.”

Currently, authors of papers discussed on PubPeer are automatically sent e-mail notifications. PubPeer’s organizers said that some journals have enquired about adding a feature to notify them when papers published in their pages are discussed. The organizers plan to add automated notifications for journals as a site feature soon. However, our experience so far has been that journals would rather ignore PubPeer, because it represents unwanted work and negative publicity for them,” they said.

“The rise of sites like PubPeer is giving people a chance to point out problems without having to put their life/career on the line,” Geller wrote in an e-mail to The Scientist. “But there needs to be a better mechanism for getting the journals and the authors’ institutions to pay attention.”

Add a Comment

Avatar of: You



Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo


Avatar of: Opaque


Posts: 2

August 26, 2014

I do wonder if the people talking about it on PubPeer ever bother contacting the journal in question. It's one thing to put it on a site like PubPeer and journals do nothing and quite another if they are getting lots of people (anonymous or known names) emailing them telling them they have a problem and them doing nothing about it.

Avatar of: Hugh-F-61


Posts: 69

August 26, 2014

Journals are much more likely to take a public critcism which has itself been the subject of public debate much more seriously than a direct private letter suggesting that there is  a problem with one of their papers. The latter implies that a referee or editor didn't pay enough attention, and they will respond defensively if  they think it will go away.

Avatar of: Eric J. Murphy

Eric J. Murphy

Posts: 20

August 26, 2014

As an Editor-in-Chief of a society journal, I have never examined PubPeer nor will I do so. First, there is the crowd or group mentality that may over emphasize some point in an irrational manner.  Just as using the marble theory of officiating is bad, one should never base a decision on the quantity of negative or positive comments. Second, if the concerned individual sent an e-mail or letter to me, then I would be duty bound to examine the issue.  It is not my duty to monitor PubPeer or any other such site, but rather to respond to queries sent to me.  So, with regards to Hugh's point, I don't support that position at all.

Mistakes happen, although frankly we try to limit these mistakes and do take steps to prevent publishing papers with FFP, it does happen.  Also, honest mistakes happen in science all the time, so of these result in an erratum, while others go unnoticed by editors and reviewers.  In such a case, someone who does notice should contact the editor to put them on notice regarding the issue so that it may be resolved.  Resolution does not necessarily mean correction, but rather the editor taking a close look at the situation, discussing the situation with the original authors, and then reaching a decision.  Most of the time a correction will be made, but not always.

Popular Now

  1. Thousands of Mutations Accumulate in the Human Brain Over a Lifetime
  2. Two Dozen House Republicans Do an About-Face on Tuition Tax
  3. Putative Gay Genes Identified, Questioned
    The Nutshell Putative Gay Genes Identified, Questioned

    A genomic interrogation of homosexuality turns up speculative links between genetic elements and sexual orientation, but researchers say the study is too small to be significant. 

  4. Can Young Stem Cells Make Older People Stronger?