The COVID-19 pandemic has led to a cultural shift in the way that science is communicated and shared. Traditional scientific publishing is a slow process and so, needing a faster route of disseminating vital new findings, scientists turned to preprint platforms, which host non–peer reviewed articles on specialized servers. My colleagues and I reported this month that researchers shared more than 35 percent of the early COVID-19 literature as preprints. But scientists aren’t the only people who have turned to preprints; we found that members of the general public and journalists have been sharing and accessing preprints at unprecedented levels.
With this sharing and use of preprints by nonexperts, their coverage by news outlets, and the fact that they have been cited as direct influences on contentious public health interventions, it becomes crucial that we assess the quality of the preprint literature and ask: Can we trust preprints?
Prior to the pandemic, some groups were very vocal about the perceived shortcomings and dangers of preprints. These voices have become louder during the pandemic and, given that preprints are being used in ways they never have before, valid concerns need to be addressed.
Strikingly, these studies all have the same conclusion: preprints should be considered valid scientific contributions that are comparable to the peer-reviewed literature.
One danger is that non–peer reviewed science could misguide public health decisions. In an analysis of policy documents, we found that outside of a global pandemic, preprints have not often been used to inform policy decisions.
One area that has, for me, been most egregious is the use—often by right-wing politicians such as former President Donald Trump—of flawed science to propagate conspiracy theories and policies not supported by evidence. Preprints have been hijacked to provide inaccurate or even false evidence in support of theories that have no scientific merit. Some of the most highly shared preprints during the pandemic have been used in this way, including a flawed preprint focused on seroprevalence and a preprint linking the SARS-CoV-2 spike protein to HIV, which was retracted within two days of posting. It’s important to note that preprints aren’t the only culprits in disseminating such questionable and flawed research; perhaps the most notorious cases are those of Didier Raoult, whose published paper on hydroxychloroquine caused a wave of misinformation, and the now infamous Surgisphere scandal (which encompassed both preprints and articles published in high-profile journals). Together, these incidents have damaged public trust in science, at a time when trust was needed perhaps more than ever.
In addition to these concerns, scientists have reservations about the use of preprints in the media, and opponents to preprints often cite perceived quality issues due to the lack of peer review. However, there is no direct evidence to support claims that preprints are lower quality than peer-reviewed papers are.
Trust in preprints
In assessing the quality of a scientific study, the gold standard would be repeating the experiments to see if the results are reproducible. However, this is expensive, time consuming, and it is difficult to find a journal willing to publish the results. In the absence of replication, we must rely on the peer-review process. Therefore, by comparing preprints to their published versions that have undergone peer review we can assess how much any given article changes and, therefore, how reliable that article was originally.
This has been the approach taken by a number of researchers recently, including us. Other groups have examined the wider preprint corpus, whereas we focused on the early COVID-19 preprint literature. One study, itself first posted as a preprint in 2019, assessed the quality of reporting between preprints and published articles. Its reporting metrics included key components underlying scientific articles such as data availability and conflicts of interest statements. Overall, the authors found that although peer-reviewed articles had a higher quality of reporting than preprints did, the difference was small. These data were subsequently supported by an independent team on a larger sample size.
More recently, a separate research team performed a comprehensive natural language processing comparison of preprinted and published versions of articles. Again, this team concluded that the changes upon publication are minimal.
Following these studies, we performed a direct analysis of the scientific content on a small subset of 200 preprints shared within the first four months of the pandemic. We focused our study on the key conclusions given and found that for 85 percent of the COVID-19 articles we assessed, there was no change in the conclusion upon publication, and more than 94 percent of non-COVID articles had no changes in their conclusions.
Strikingly, these studies all have the same conclusion: preprints should be considered valid scientific contributions that are comparable to the peer-reviewed literature. This conclusion comes with an important caveat: so far no studies have investigated the comparability of preprints that are never published with those that are subsequently published. With more than 70 percent of the preprint literature eventually being published, this represents a minority of preprints; nevertheless, we are planning future work to directly address this remaining issue.
There are always exceptions and bad players, whether they share their work as a preprint or published paper. But can we trust preprints as a mechanism of quickly disseminating vital science during a pandemic? The answer appears to be a very clear yes—at least inasmuch as we trust the peer-reviewed literature.
Jonny Coates is a postdoctoral researcher at the William Harvey Research Institute and Queen Mary, University of London and can be found on Twitter @JACoates91. He is a member of preLights and the ASAPbio community.