Gambling on Reproducibility

New research finds that observers placing bets in a stock exchange–like environment are pretty good at predicting the replicability of psychology studies.

Bob Grant
Bob Grant

Bob started with The Scientist as a staff writer in 2007. Before joining the team, he worked as a reporter at Audubon and earned a master’s degree in science journalism...

View full profile.


Learn about our editorial policies.

Nov 10, 2015

Can science publishing benefit from the establishment of a stock exchange of sorts?WIKIMEDIA, NEW YORK STOCK EXCHANGEThe poor reproducibility dogging psychological science has been well documented in recent months. Now, an international team of researchers—some of whom were the first to sound the alarm regarding the field’s replication problems—suggests that solving the predicament may lie in setting up futures markets in which psychologists acting as traders place bets on whether or not a particular study’s findings will be replicated in future attempts.

“The results show that a collection of knowledgeable peers do have a good sense of what will replicate and what won’t,” Caltech behavioral economist Colin Camerer, who was not involved with the study, told Nature. “This information is in the judgments of peers but has never been collected and quantified until now.”

The study’s findings, published yesterday (November 9) in PNAS, further indicate that...

Dreber and her colleagues instructed 92 psychologists participating in University of Virginia psychologist Brian Nosek’s Reproducibility Project—designed to test the replicability of 100 psychology studies published in 2008—to place bets on whether 41 of the studies would succeed. Each participant was given the equivalent of $100 and told to buy or sell stock in studies based on their perception of their reproducibility. Each participant could see the trading activity of all the others, as is the case in stock-market trading. A trader could double his money if a study was reproduced successfully, but would lose his stock and money if it wasn’t. “If you believe the result will be replicated, you buy the contract, which increases the price,” Dreber told FiveThirtyEight. “If you don’t believe in a study, then you can short-sell it.”

The markets correctly predicted 71 percent of the replication outcomes. When the participants were asked to predict the replication odds for each of the studies, they were right only 58 percent of the time.

So could a similar system be used in addition to peer review to vet studies in psychology and other disciplines? Perhaps, said Nosek. “We may be able to use prediction markets to be more efficient in deciding what needs to be replicated, and making estimates of uncertainty about studies for which replication is not likely or possible,” he told The Atlantic.

Others aren’t so sure. “It would be unwise to use prediction markets to draw conclusions about the replicability of individual studies,” psychology researcher Simine Vazire of the University of California, Davis, told Nature. “But I think they could provide useful information about moderators of replicability—what types of studies do prediction markets anticipate to be more likely to replicate?”