The Cost of Irreproducible Research

Half of basic science studies cannot be replicated, according to a new analysis—to the tune of $28 billion a year in the U.S.

By | June 10, 2015

PIXABAY, OPENCLIPARTVECTORSJust about half of all basic life science research is flawed to the point that it cannot be replicated, according to an analysis published in PLOS Biology yesterday (June 9). These “irreproducible” studies end up costing around $28 billion annually in the U.S.

“While false positives are an inevitable part of scientific research, our study shows that the current level of irreproducibility in preclinical research is very costly,” study coauthor Timothy Simcoe, an economist at the US Council of Economic Advisors, said in a press release.

The analysts used previous estimates of irreproducibility to arrive at their number. Poor quality reagents and materials were the biggest cause of flaws in experiments, mucking up more than one third of preclinical studies, the team found. Poor study design and insufficient analysis or reporting each afflicted a little more than a quarter of studies, while improper laboratory protocols took down one out of every 10 studies. “The four categories are decent, but the estimates are off,” John Ioannidis, a Stanford University epidemiologist, told Nature News. “I would put a much higher rate on the data analysis and reporting component.”

Ferric Fang, a microbiologist at the University of Washington, told ScienceInsider that the analysis may not be accurate. “To suggest that 50% of research dollars are being wasted is ridiculous and unhelpful,” he said.

The authors acknowledged limitations in their analysis. “Our primary goal here is not to pinpoint the exact reproducibility rate but rather to identify root causes of the problem and develop a framework to address the highest priorities,” coauthor Leonard Freedman of the Global Biological Standards Institute said in the press release. He and his colleagues have offered a number of ideas to help address the problem, including having funders require training programs in good study design as well as evidence of reagent validation.

Add a Comment

Avatar of: You

You

Processing...
Processing...

Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo

Comments

Avatar of: JonRichfield

JonRichfield

Posts: 135

June 10, 2015

Hmmm...Improper and unethical pressures and temptations and simple dishonesty didn't even get a mention. That should be very reassuring.

Avatar of: JC States

JC States

Posts: 3

June 10, 2015

Some of the responsibility must fall upon lax peer review. If reviewers paid more attention to the methods section to ensure adequate description of methods and reagents, the situation would improve. Too often we find that missing details make it extremely difficult to reproduce others' results. More stringent peer review also would clamp down on loose interpretations of the results. 

Avatar of: John AB

John AB

Posts: 5

June 10, 2015

Can we have some sense of proportion in reporting here, please?

The headline for this article says "half of basic science studies"...huh?  The article in question addressed "preclinical" studies, and, at best, the work cited in the study covers the area of "biomedical" science.  

So, as an ecologist/aquatic biologist/oceanographer, I'm going to speak for the "basic science" community and cry "foul".  Shame on you, "The Scientist" for sensationalizing, here.

I'm certainly not saying that particular disciplines are any better/worse than others.  But show me the work that demonstrates it before tarring everyone with the same brush.      

Avatar of: QuantumDottie

QuantumDottie

Posts: 2

June 10, 2015

In all fairness, please note that the quote from Ferris Fang to ScienceInsider seems to have come before he read the PLoSBiology paper, as noted in Kaiser's ScienceInsider piece.  Maybe Fang would have even harsher criticism of the study or maybe not; we readers don't know. But perhaps we (journalists/editors and scientists) can improve the science by putting less emphasis on instant reporting and more on high quality work. (Disclosure - as a science consultant, I consult for GBSI, the source of the PLoS Biology paper.)

For an assessment of the PLoS paper by a commenter who actually read the piece, see Derek Lowe's In the Pipeline blog posting http://tinyurl.com/orv2hru

Avatar of: dumbdumb

dumbdumb

Posts: 94

June 10, 2015

just 50%?

According to my experience it is much higher than that! Indeed, unless I have no other choice I prefer not to bother reading literature anymore

But than, why not? Still based on experience, publishing fake data usually assure you a nice career

Popular Now

  1. Thousands of Mutations Accumulate in the Human Brain Over a Lifetime
  2. Can Young Stem Cells Make Older People Stronger?
  3. Two Dozen House Republicans Do an About-Face on Tuition Tax
  4. Insects’ Neural Learning and Memory Center Discovered in Crustaceans