The COVID-19 pandemic has had a profound effect on the growth of preprints. For scientists and health professionals, the pandemic has driven the need to share research findings about COVID-19, however preliminary, as quickly as possible to better understand its pathology, find new therapeutic options, and inform policy. Public demand for information about COVID-19 has driven the increase in media reporting of research posted as preprints. Even before the pandemic, there was a call for scientists to better understand the work of journalists, and the pandemic has made bridging the gap between the scientific and media communities, particularly in relation to communicating scientific uncertainty, an imperative.
Last year, ASAPbio conducted the #bioPreprints2020 questionnaire, an online survey of a wide range of stakeholders, including librarians, researchers, journalists, and funders, to understand perceived benefits and concerns about preprints. ASAPbio (Accelerating Science and Publication in biology) is a scientist-driven nonprofit working to promote innovation and transparency in life sciences communication through the productive use of preprints and open peer review. Not surprisingly, the possibility of premature media coverage of preprints was the top concern for respondents. This was the case for both respondents who had posted research as preprints and those who had not.
The media community faces strong time and competitive pressures that make conducting independent verification and validation of research posted as preprints a challenge.
To address this concern, ASAPbio launched and coordinated the Preprints in the Public Eye project, with support from the Open Society Foundations, in September 2020. As far as ASAPbio is aware, the project was the first step toward bringing together stakeholders from the scientific and media communities to agree on common principles around media reporting of research posted as preprints in order to mitigate the risks of publicizing research that is potentially flawed or too preliminary.
The project involved representatives from preprint servers, publishers, and research institutions (including institutional press offices), along with journal editors, researchers, and journalists. The project is now complete, and the outputs are a series of documents making recommendations for preprint servers on labeling preprints and for institutions and researchers on communicating about research, including with the media. For journalists and science writers there are tips and links to resources intended to aid the discovery of research posted as preprints where the science has undergone some form of assessment, such as by a journal-independent peer-review service.
As part of the preprints labeling arm of the project, we conducted a user experience investigation to determine whether readers notice the labels that currently exist on preprint servers, whether they can distinguish between a preprint and a peer-reviewed article, and what would make the labels more noticeable and easier to understand. The users were five laypeople, three journalists, three researchers, and one clinician. The findings showed a need for noticeable labeling and clarity about the screening criteria used to determine what is posted on a preprint server.
While the document for preprint servers focuses on transparency about screening criteria and the absence of traditional peer review carried out by the preprint server, there was much discussion among the project stakeholders about the nuances around peer review of preprints. For example, some preprints will have undergone traditional peer review and be published in a journal, some will undergo peer review via journal-independent peer-review services, while others will undergo community review, either via direct commenting on the preprint or by other community-run peer-review or curation activities.
The flawed nature of traditional peer review has been iterated many times, and the rise in the rate of retracted peer reviewed articles challenges the notion that traditional peer review alone is a reliable form of validation. The stakeholder discussions highlighted the need to move beyond attitudes that focus on the presence or absence of traditional peer review toward an approach that encourages a standardized and responsible way of talking about all research for all stakeholders. While this discussion is not directly reflected in the recommendations for preprints, the project stakeholders agreed that the quality of research should not be determined by its peer-review status, but by the level of scrutiny the research has undergone and whether that feedback (which may be from a number of different sources) is positive or critical.
Institutions and researchers
For institutions and researchers, project stakeholders made recommendations around how they should talk about research with the media community. The document for institutions encourages transparency about the level of independent scrutiny and limitations of research, and encourages the provision of guidance and support for researchers on communicating about their work in various forms of media.
One point that was and remains contentious is whether institutions should actively promote to the media research that has been posted as a preprint and has yet to undergo independent scrutiny. There was a strong view from some stakeholders that they should not, and that, in cases where active promotion of information was in the interest of public health or safety, an internal risk assessment should be performed before any such promotion. While this is understandable for medical and biomedical research, promoting research posted as preprints in the media is the norm for some fields and some institutions. We have acknowledged this in the guiding principles document as a point that is still open for debate.
The document for researchers is along similar lines, encouraging researchers to talk about the peer-review status and limitations of their work, avoid overhyping their findings, and work in collaboration with their institutional press offices.
Journalists and science writers
The media community faces strong time and competitive pressures that make conducting independent verification and validation of research posted as preprints a challenge. To address this, the guiding principles aim to make it easier for members of the media to identify research that has been scrutinized. The guidelines expand on the existing document Making Effective Use of Preprints: Tips for Communicators, and include resources that journalists can use to identify preprints that have undergone expert evaluation. In addition, by encouraging scientists, via the guiding principles for researchers, to talk about the limitations of their research, and by encouraging preprint servers to include important information in their labeling, the aim is to make it easier for anyone to understand the level of scrutiny a piece of research has undergone.
Jigisha Patel is an independent research integrity specialist. She was employed by ASAPbio as the Project Coordinator for the Preprints in the Public Eye project.