Advertisement
The Scientist
The Scientist

Post-Publication Peer Review Mainstreamed

The launch of PubMed Commons highlights the pros and cons of re-reviewing published papers. 

By | October 22, 2013

FLICKR, JJACKOWSKIAsk a scientist—any scientist—what irks them most about publishing and they are sure to mention peer review. The process has been blamed for everything from slowing down the communication of new discoveries to introducing woeful biases to the literature. Perhaps most troubling is that few believe peer review is capable of accomplishing what it purports to do—ensuring the quality of published science.  

Indeed, several studies have shown that, in actuality, peer review does not elevate the quality of published science and that many published research findings are later shown to be false. In response, a growing number of scientists are working to impose a new vision of the scientific process through post-publication review, the process of critiquing science after it has become part of the literature.

Reviewing published work is, of course, nothing new. Scientists have always been welcome to publish contradictory findings, for example, contact the papers’ authors directly, or write a letter to the journal’s editor. However, because all are lengthy processes that likely will never be heard or seen by the majority of scientists, most scientists do not participate in formal reviews.

A small number of scholarly journals have launched online fora for scientists to comment on published materials. Uptake, however, has been slow for a number of reasons—chief of which is the inconvenience of commenting journal by journal.  

“If you want to comment on a Nature paper, you have to go to the Nature site, find that paper, and comment. If you want to comment on a PLOS paper, you have to go to a different website, and so forth,” said Stanford University’s Rob Tibshirani, professor of health research and policy and statistics. “It’s a major time investment, particularly when people may never see the comments.”

Likewise, social media platforms, blogs, and other websites—such as Zotero, CiteULike, and Mendeley, to name a few—have also seen only scattershot commenting activities, at best.

Frustrated by these inefficiencies, Tibshirani is one of several scientists behind the development of PubMed Commons, a new post post-publication peer review system housed on the oft-accessed National Center for Biotechnology Information (NCBI) biomedical research database. The Commons, announced today (October 22), allows users to comment directly on any of PubMed’s 23 million indexed research articles, much in the way people review films on Rotten Tomatoes, evaluate restaurant service on Yelp, or grade purchases made on Amazon.

Tibishiri said an organized post-publication peer review system could help “clarify experiments, suggest avenues for follow-up work and even catch errors.” If used by a critical mass of scientists, he added, “it could strengthen the scientific process.”  

David Lipman, who heads up NCBI, said the development of the Commons was a driven by “consistent and increasing requests from PubMed users.” Approximately 2.5 million to 3 million people access the online resource each day, according to Lipman.

Comments made through PubMed Commons will be covered by Creative Commons Attribution license and available for anyone to browse. However, only those who have authored PubMed-indexed comments are able to comment on others’ papers. For those who are eligible to comment, the process is simple: after registering, users can review the article in a comment box that follows each article. The comment box also allows users to post replies to existing comments and rate whether existing comments are useful.

In addition, PubMed Commons will aggregate comments to create a “hot list” that draws attention to papers that are trending.

PubMed Commons has already been privately beta-tested by approximately 250 users and is currently open for additional testing by National Institutes of Health (NIH) grantees. NIH will evaluate comments before officially green-lighting the system. If the decision is to make the system fully public, NCBI will provide a public API so that publishers and other groups can make these comments useful to a wider community.

For now, scientists eager to comment directly through PubMed can do so via PubPeer, a website funded and run by an unknown team of scientists that allows first and last authors of published articles to comment on almost any scientific article published with a DOI or preprint in the arXiv. A free browser plug-in is available for download that that show PubPeer comments directly on PubMed.

Just more than a year old, PubPeer has already demonstrated the power of post-publication peer review in improving the research underlying publications. The most noted example to date is a comment, posted by a person known only as “Peer 1,” who identified several figure-related and typographical errors in a highly publicized Cell paper that reported the creation of human embryonic stem cells through cloning. Though study’s authors maintain their conclusions, they did issue an erratum addressing various formatting errors raised by PubPeer users.

By allowing for anonymous comments, PubPeer aims to create an open, debate-friendly environment, while maintaining the rigor the closed review process currently used by most journals. Its creators, who describe themselves as “early-stage scientists,” have also decided to remain anonymous, citing career concerns. “A negative reaction to criticism by somebody reviewing your paper, grant or job application can spell the end of your career,” a representative from PubPeer explained by e-mail. “We don't want to take that risk and we understand that many commenters do not want to either.”

Detractors say that anonymous comments all too often support destructive discourse. Earlier this year, for example, the Huffington Post and The Miami Herald decided to ban anonymous comments while, just last month, Popular Science chose to disable commenting altogether because of excessive trolling. PubPeer hopes to mitigate concerns stemming from anonymity by reviewing each comment to ensure it directly addresses a paper’s data before it is publicly posted.

PubMed Commons chose to circumvent the downfalls of anonymity by requiring all users to register and identify themselves. Still, Lipman noted, there were “cogent and compelling arguments for both anonymous and identifiable commenting.”

“Those wanting anonymous posts were concerned that many scientists, especially junior researchers, would be reluctant to make critical comments. But those opposed to anonymous comments believed that the quality of interchange would be higher if commenters were required to identify themselves,” he said, adding that the group was “very much split” on the decision.

User feedback will determine whether PubMed Commons continues to require users to identify themselves. “User experience will determine the direction we take [PubMed Commons],” said Lipman. “This is meant to be a community resource.” 

Advertisement

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: Mehrishi

Mehrishi

Posts: 5

October 23, 2013

Pride and Prejudice:Flawed methods used by scientists not spotted by preliminary screeners or referees lead to disastrous, tragic consequences- as shown by HIV, HECPC infected blood.

1. The real problem of 'peer reviewing' in it self a good thing when constructive.

2. A complication is that now- biomedicine, molecular biology, clinical sciences require a wide overarching knowledge.

3. Sadly, it is the preliminary screeners (2,5,10? yr post PhD) for the busy editors wihtout deep and wide knowledge letting the system down.

The authors are at a diusadvantage- the  EIC in general rubber stamps their 'decision' right/wrong.

4. Many papers by biologists, clinicians suffer from a knowledge of basic high school chemistry, undregraduate biochemistry- never learnt or forgotten.

Example: Gibbs adsorption equation written just a year before the French Revolutiuon- if the biologist, clinician paid attention that the surface tension is lowered by organic solutes (increased by electrolytes) is dependent upon temperature, he/she should eb careful about adospprption of impurities in gnadling blood and cell samples. You would see blank faces.

5. Some pprs by well known 'names' usually refereed by buddies are getting thorugh on the nod- by bad technology, careless work 'artifacts are being introduced by the very methods used' while handling cells for transplantation-unawares doing this is inexcusable, doing it knowlngly is criminal neglect and tragic to put such materials into patients.

It is worse when 'patents are involved and the methods are $bn industry driven.

 

Avatar of: Ellen Hunt

Ellen Hunt

Posts: 13

October 23, 2013

In addition, after taking the time to write a cogent comment (which is not always critical, but might be extending) a major journal (that I won't name) got back to me saying that they had lost the connection between my comment and the paper. At that point, I gave it up. Not worth the hassle. 

But - I think that for a comment to be acceptable in science, anonymity CANNOT be allowed!  If you can't stand up to be counted, then you shouldn't be published! Period.  That is not acceptable! 

Avatar of: hgf1980

hgf1980

Posts: 1

Replied to a comment from Ellen Hunt made on October 23, 2013

October 24, 2013

 

 

Couldn't you apply the same notion to peer review?  It is anonymous and accepted, albeit pre-screened.

 

I would think destructive comments could be summarily ignored, flagged, or reported to moderation.

 

Lastly, I'd make two arguments for anonymity. 

One is a reiteration from the article:  junior colleagues are intimidated and fearful of ostracization if they step on the wrong toes.

The second is that anonymity forces the author to respect constructive, useful comments and address them, regardless of whether they came from someone they have never heard of or the person who is on the study section reviewing your grant next month.

Avatar of: shahana

shahana

Posts: 1

October 29, 2013

We are going to space, to Neptune, to Saturn and even going to Alpha Century    ( or already gone)..... But we still didn't  invent any total cure medicine for pigmentation or dark spots on face.... Why ? Its a total shame....

Avatar of: ChriM

ChriM

Posts: 2

October 31, 2013

One has to accept that everything in this world has its merits and demerits: allowing comments on published papers can yield both useful and useless feedback. But the underlying reality for people wanting to make somewhat coherent comments on a publication is that the existing system of publishing one's research has become a rock-hard caste system--the established, moneyed, connected scientists in all countries form a tight coterie that always has access to the better journals, which leaves the struggling middle and lower orders to go elsewhere.

This "elsewhere" could mean lower tier journals or the predatory journals that lurk everywhere, offering to publish work for a fee. Universities are churning out Ph.D. degrees and demand a certain number of publications, impact factor, so much funding etc etc to even consider entering and keeping oneself on the career ladder. On the other hand, the journal editors and publishers form a solid wall, sometimes applying their biases to reject manuscripts that actually fall in their aims and scope but which are authored by the less-glamorous scientists from less-glamorous parts of the world.

This obstacle race makes many "scientists" desperate: the ones from financially strapped laboratories and institutes commit fraud for certain reasons, the ones from well-funded institutes commit fraud for different reasons. How would we know this if this fraud wasn't published? Who's doing the publishing? Who's reviewing? Why? Some admit their wrongdoing, others hide behind their networks and insulate themselves from punishment. Looking at the world does not inspire much confidence in anybody.

So if you knew someone was doing something wrong, would you or wouldn't you report it? Would you do it under your name to prove to the world that you are a righteous hero and then lose your job and go out and beg on the streets to support your family? Trust me, some of these big shots can arrange for this to happen. Whom would you be helping? I have written to many journal editors about problems in papers that I knew to exist because I was familiar with the authors' situation or simply because I came across some information on the Internet. I realized that I got different degrees of acknowledgment depending on the editors and their attitudes towards a critic. Ultimately, they are journal editors employed by publishing houses and their job is to publish.

They don't want to hear anything against their jobs. Of course, their work is not easy and it's always easier to criticize others. Still, except for three occasions, no journal editor warmed to the idea of criticism, some were hostile, wanting to know who I was in general and to question them and their reviewers. Others were milder and tried to say that they were investigating the claims but I knew that the outcome would be nil and it was indeed that.

The only three occasions where I got some acknowledgment for my efforts included a journal editor who promised not to publish highly derivative work that closely resembled other studies published by others and by the authors themselves in other journals, and two who retracted papers due to duplicate publications in two journals and for lifting a figure from an older paper. These three editors were polite but not comfortable with the idea of being told anything about their publications.

Clare Francis or not, ensuring that one's criticism is valid is always the duty of the critic. But people never look at WHAT is being said, it's always WHO's saying it. So anonymous whistleblowers and people who comment on published papers that are suspicious, because of method flaws or interpretations or poor background knowlege, may always be seen as evil trolls, but they are a necessary evil. Especially when journal editors, with all their biases, create these situations where fraud becomes possible and readers lap up the fraud because these "famous" people said so.

 

Avatar of: jjj157

jjj157

Posts: 1

February 22, 2014

i2i Media, Beijing, Alexander Glos

Recently, we planned, and booded a convention through i2i Media, and its CEO, Alexander Glos, for a gay and lesbian rights seminar.

Being homosexual himself, we felt as if we could trust Alexander Glos, and his company i2i Media.  We paid a very large upfront fee to his partner, and lover.  However, when it was time to hold our convention, Alexander Glos, and his Lover disappeared with the funds.

We have let all with the Gay community know not to deal with this man, his company, or his boyfriend.

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
Cisbio
Cisbio
Advertisement