Menu

Latest Effort to Replicate Psych Studies Yields 62 Percent Success

Among the experiments that came to the same result as the original, the effect size was smaller on average.

Aug 27, 2018
Kerry Grens

ABOVE: © ISTOCK, TONIFLAP

A large undertaking to reproduce 21 psychology studies published in Nature and Science came to the same conclusion as the original papers 62 percent of the time, according to a report in Nature Human Behavior today (August 27). When the results of the original and replication experiments agreed, the effect sizes were smaller the second time around, indicating to the authors that both false positives and “inflated effect sizes” are part of the reproducibility problem in the field.

“A false positive result can make other researchers, and the original researcher, spend lots of time and energy and money on results that turn out not to hold,” study coauthor Anna Dreber, an economics professor at the Stockholm School of Economics, tells NPR. “And that’s kind of wasteful for resources and inefficient, so the sooner we find out that a result doesn’t hold, the better.”

In addition to the replication attempts in the latest study, Dreber and her colleagues also tested a method for predicting which of those studies would indeed be reproducible. They asked around 200 people, none of whom were involved in any of these experiments, to place bets in a stock-market–like scenario on which results would pan out. Remarkably, the participants anticipated the very studies that replicated the originals.

“I did a sniff test of whether the results actually make sense,” Paul Smeets, a participant from Maastricht University, tells The Atlantic. “Some results look quite spectacular but also seem a bit too good to be true, which usually that means they are.”

See “Studies Unable to Reproduce Results of Two Diabetes Papers

The 21 original papers were published between 2010 and 2015 and chosen from Nature and Science because of the journals’ prestige. All of the authors were alerted to the replication project and asked to give feedback. Among them is Will Gervais, a psychology professor at the University of Kentucky whose 2012 study examined participants’ religiosity before and after looking at Rodin’s sculpture “The Thinker.” It didn’t replicate, just as the prediction study expected. “Our study in hindsight was outright silly,” Gervais tells The Washington Post.

“We should not treat publication in Science or Nature to be a mark of a particularly robust finding or a particularly skilled researcher,” Simine Vazire, a psychologist at the University of California, Davis, who was not involved in the study, tells The Atlantic. These journals “are not especially good at picking out really robust findings or excellent research practices. And the prediction market adds to my frustration because it shows that there are clues to the strength of the evidence in the papers themselves.”

See “Replication Failures Highlight Biases and Ecology and Evolution Science

January 2019

Cannabis on Board

Research suggests ill effects of cannabinoids in the womb

Marketplace

Sponsored Product Updates

FORMULATRIX® digital PCR technology to be acquired by QIAGEN
FORMULATRIX® digital PCR technology to be acquired by QIAGEN
FORMULATRIX has announced that their digital PCR assets, including the CONSTELLATION® series of instruments, is being acquired by QIAGEN N.V. (NYSE: QGEN, Frankfurt Stock Exchange: QIA) for up to $260 million ($125 million upfront payment and $135 million of milestones).  QIAGEN has announced plans for a global launch in 2020 of a new series of digital PCR platforms that utilize the advanced dPCR technology developed by FORMULATRIX combined with QIAGEN’s expertise in assay development and automation.
Application of CRISPR/Cas to the Generation of Genetically Engineered Mice
Application of CRISPR/Cas to the Generation of Genetically Engineered Mice
With this application note from Taconic, learn about the power that the CRISPR/Cas system has to revolutionize the field of custom mouse model generation!
Translational Models of Obesity, Dysmetabolism, Diabetes, and Complications
Translational Models of Obesity, Dysmetabolism, Diabetes, and Complications
This webinar, from Crown Bioscience, presents a unique continuum of translational dysmetabolic platforms that more closely mimic human disease. Learn about using next-generation rodent and spontaneously diabetic non-human primate models to accurately model human-relevant disease progression and complications related to obesity and diabetes here!
BiochemAR: an augmented reality app for easy visualization of virtual 3D molecular models
BiochemAR: an augmented reality app for easy visualization of virtual 3D molecular models
Have you played Pokemon Go? Then you've used Augmented Reality (AR) technology! AR technology holds substantial promise and potential for providing a low-cost, easy to use digital platform for the manipulation of virtual 3D objects, including 3D models of biological macromolecules.