Using a mathematical model that estimates false positive rates among published p-values, two researchers have found that only 14 percent of statistically significant results may be false, despite claims by some critics that a vast majority of biomedical research is erroneous. The study, which was posted on an open-access study archive, arXiv.org, follows a series of claims beginning with a 2005 essay in PLOS Medicine that “most current published research findings are false” due to small study sizes and bias.
“Our results suggest that while there is an inflation of false positive results above the nominal 5% level, but [sic] the relatively minor inflation in error rates does not merit the claim that most published research is false,” the authors wrote in the study’s discussion.
They conclude that their study, which extracted 5,322 p-values from 77,430 published biomedical study abstracts, upholds the reliability of biomedical literature and scientific progress. They do acknowledge, however, that it is not the last word on the matter. The results “do not invalidate the criticisms of standalone hypothesis tests for statistical significance that were raised,” the authors wrote. “Specifically, it is still important to report estimates and confidence intervals in addition to or instead of p-values when possible so that both statistical and scientific significance can be judged by readers.”