NIH Tackles Irreproducibility

The federal agency speaks out about how to improve the quality of scientific research.

By Jef Akst | January 28, 2014

WIKIMEDIA, NCI, JOHN CRAWFORDLately the scientific community has been talking about the omnipresent problem of irreproducibility—the failure of researchers to replicate results in the published literature. This week in Nature, National Institutes of Health (NIH) Director Francis Collins and Principal Deputy Director Lawrence Tabak weighed in on this discussion, noting several ways in which the agency hopes to fix the problem.

“Reproducibility is potentially a problem in all scientific disciplines,” the duo wrote. “[T]he checks and balances that once ensured scientific fidelity have been hobbled.”

Collins and Tabak cited poor experimental design and a scientific culture that overvalues provocative statements and high-impact publications as contributors to the problem. Misconduct, they noted, is unlikely to be a main driver. “[F]raudulent papers are vastly outnumbered by the hundreds of thousands published each year in good faith,” they wrote.

Among the suggested solutions is better training for young investigators on experimental design, with an emphasis on “frequently ignored” issues such as “blinding, randomization, replication, sample-size calculation and the effect of sex differences.” Starting later this year, such training will be mandatory for NIH intramural postdoctoral fellows, with corresponding materials to be posted on the agency’s website by the end of 2014. Moreover, the NIH officials said that the agency plans to institute a more stringent examination of methodological details during grant review and recommended the inclusion of more thorough descriptions of experimental setups in publications—something that Nature Publishing Group, Science Translational Medicine, and most recently Science, have all pledged to require of submissions to their journals. Also not to be forgotten is the importance of the vast amounts of research that go unpublished, Collins and Tabak added.

“As a funding agency, the NIH is deeply concerned about this problem,” they wrote, adding that “reproducibility is not a problem that the NIH can tackle alone. Consequently, we are reaching out broadly to the research community, scientific publishers, universities, industry, professional organizations, patient-advocacy groups and other stakeholders to take the steps necessary to reset the self-corrective process of scientific inquiry.”

Add a Comment

Avatar of: You



Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo


Avatar of: dan_i


Posts: 1

January 29, 2014

Typical government response: more regulation., more paperwork for the researchers, and zero impact. 

Avatar of: Nikolai Slavov

Nikolai Slavov

Posts: 2

January 29, 2014

Figure 2 in the article below supports the role of the impact factor worship in contrubiting to the problem: 
http://www.nature.com/nmeth/journal/v10/n11/full/nmeth.2697.html ?

Avatar of: Lasha M.

Lasha M.

Posts: 1

January 29, 2014

This would be great and lucrative in some cases, but time consuming in other.

Avatar of: MikeH


Posts: 16

January 29, 2014

Well, duh.  The biomed research system that has been purposely built in the latter half of the 20th century, and kept today without any significant call for change, has many fold over-saturated the "market" with research PhDs making jobs and careers highly insecure and competitive in a time of shrinking funding.  With less than 10% of grant applications funded this is a "winner takes all" field.  And the completely predictable result: irreproducable results and overinflated conclusions.  Absolutely everything is a trigger and controlling some pathway.  It will only get worse until such time as there are less graduate programs, fewer PhDs being cranked out, secure careers provided to researchers, and an end to the myth that we have too few biomed PhDs.

Avatar of: Nejat Duzgunes

Nejat Duzgunes

Posts: 3

January 29, 2014

Drs Collins and Tabak may want to also look at the organization they are running as a contributor to the pressure to publish and get grants with irreproducible data.  It is absurd that NIH funds only 10-15% of applications, with approximately 80,000 applications per year.

NIH needs to change its supposedly perfect peer review system that has never been tested scientifically against an alternative system of funding, looking at "success" criteria like publications, citations, patents, and cures achieved by two different funding systems over 5 and 10 year periods, as I proposed in 1999 (The Scientist 13(8), 13; 1999).


In the alternative world I have proposed (The Scientist August 1, 2007) (https://www.the-scientist.com/?articles.view/articleNo/25269/title/A-New-Paradigm-for-NIH-Grants/), we would have 96,000 NIH grants compared to the current 26,000.  In this world, 40,000 young scientists starting their first independent position would each get a $50,000/year grant for 5 years.

96,000 minds who can put their ideas to work to understand and cure diseases are a lot better than 26,000 minds who spend half their time working on grant applications.

Wake up, NIH, and its leaders!

Popular Now

  1. Estonia Offers Free Genetic Testing to Residents
  2. Human Brain Organoids Thrive in Mouse Brains
  3. New Ovarian Cancer Vaccine Shows Promise
  4. The Second March for Science a Smaller Affair