ABOVE: © Istock.com, Olivier Le Moal

If a set of newly finalized research assessment reforms is accepted, European scientists may find themselves freed of the “publish or perish” model that’s long been the law of the land. The document, called the Agreement on Reforming Research Assessment, was facilitated by the European Commission and includes input from research institutes, researchers’ associations, and funders, who together have produced a roadmap of guiding principles for reforming how scientists and their work should be evaluated for funding, awards, research priorities, and recruitment across Europe. 

There is a broad consensus that the current system isn’t working, according to Stephane Berghmans, the director for research and innovation at the European University Association (EUA), a group that was involved in drafting the agreement. He adds that this is the case “not just in universities, but in the [scientific] community overall,” and that with these reforms, “researchers and the universities are looking, in a sense, to claim it back.”

Presently, in order to get research funded, scientists—including tenured and tenure-track faculty, research scientists, and early career academics—are frequently assessed based on the number of papers they publish, and in which journals. 

The newly proposed criteria for evaluating research will instead recognize a broader range of contributions to science and the diversity of careers in science; reframe assessments to include evaluations by peers; abandon what the document refers to as the “inappropriate” use of journal and publication metrics as a measure for success; and avoid using international rankings of research organizations when assessing individual researchers. 

This framework is the result of a long period of negotiations. Following a 2021 scoping report by the European Commission (which involved many stakeholders from across Europe), representatives of EUA and Science Europe, an association of European funding agencies and research organizations, joined together with independent policy advisor Karen Stroobants to draft an agreement on how to reform research assessment. While the European Commission facilitated the drafting, in all, about 20 stakeholders gave feedback on the text as part of a “core group,” while an additional 350 stakeholders from more than 40 countries voiced their opinions in assembly meetings. “The agreement in itself is a collective effort, going beyond just saying we don’t want this anymore, and rather defining where it is that we want to go,” says Berghmans.

The ten commitments of research assessment reform

The final agreement, which was published on July 20 and can be signed by those interested in adopting the reforms from September 28 onwards, puts forward four core commitments. The first includes a recognition of the diversity of contributions and careers in research, rewarding activities such as teaching, supervising, and mentoring instead of solely focusing on journal publications. The second commitment is to base research assessment “primarily on qualitative evaluation for which peer review is central,” the document states, putting the focus on the quality of research rather than the quantity. Third, organizations commit to abandoning the “inappropriate uses” of journal- and publication-based metrics for research assessment, such as impact factor and h-index. And finally, the fourth core commitment is to avoid assessing individual researchers based on the rankings or prestige of their research organizations. 

In addition to the four core commitments, another six supporting commitments aim to assist the signatories in their work by pledging resources, raising awareness of the need for such reforms, and evaluating and communicating the proposed changes. One commitment also details the formation of a coalition of signing organizations, which is intended to serve as a space for mutual learning and collaboration that will help organizations implement the core commitments. 

Similar initiatives already exist, such as the 2012 San Francisco Declaration on Research Assessment (DORA) in the US or the international Leiden Manifesto for research metrics from 2015. But the new agreement goes even further, according to Toma Susi, a physicist at the University of Vienna who was involved in drafting the new agreement as a representative of the Initiative for Science in Europe (ISE). “DORA . . . is the most important statement, but it’s still just a statement,” Susi tells The Scientist, noting that the new agreement “sets out principles…following very much what has been the discussion for a long time, but then augments those principles with concrete commitments.”

For Stroobants, the most important commitment is the first one, which recognizes the variety of contributions from other types of researchers, including data scientists and software engineers. “At the moment, researchers are in a very competitive environment and, at the same time, the definitions of success are very narrow. . . . For me personally, the key thing is to start to recognize much more the wide diversity of contributions that researchers make, to change what success means.” 

Vinciane Gaillard, the Deputy Director of Research and Innovation at EUA, sees the agreement as the start of big changes within science. “The long-term vision is about culture change,” she tells The Scientist. “Let’s not forget that in the end, the contribution [of research and researchers] to society is way more than the number of publications and where they have been published.”

Risks and opportunities for signatories

Going forward, the organizations that sign the agreement will be asked to help develop the Coalition, first by establishing organizational structures and then by coming together in a constitutive assembly towards the end of the year to agree on the Coalition’s organizational form. “The commitments are the frame, and within this frame, institutions, funders, [and] all the research community can define how [they want] research assessment to be done,” says Berghmans. “It will not be just one system, . . . there could be variation depending on the type of [organizations], differences between disciplines, even between countries.” 

Despite the progressive ideas being put forward by the agreement, there are still areas of debate—particularly about the impact that the proposed changes could have on early-career researchers, according to Sebastian Dahle, the vice president of the European Council of Doctoral Candidates and Junior Researchers (Eurodoc), an organization that helped draft the scoping report and has stated its interest in signing the agreement. 

“Having a whole change of the entire system poses a quite tremendous risk, specifically for early-career researchers who are signed into the system,” Dahle tells The Scientist. In particular, activities currently rewarded in assessments may no longer carry the same weight in a reformed system, and the full impact of these changes on their career path remains “an open question.” Therefore, Dahle says it is important that early-career researchers have a seat at the table in the Coalition. “Having the flexibility to really take a look at that, having the mechanisms to observe what’s happening . . . and then adapt—that’s a really good way of approaching this.” 

Gaillard also acknowledges that early-career researchers are in the most difficult position in this transition, as such assessments carry more weight in setting the course of their careers.  “The idea is to have them frontline in the Coalition, to draw attention to what it is that should be done and how to do it, to avoid any unintended consequences,” she writes in an email to The Scientist. And because researchers are the ones being evaluated, Susi says that it makes sense that “they should be central in deciding what the reform looks like in practice.”

Elizabeth Gadd, a research policy manager at Loughborough University and chair of the International Network of Research Managers (INORMS) Research Evaluation Group—which was involved in the assemblies—tells The Scientist in an email that the effectiveness of the new agreement will now depend on whether institutes actually adopt it. “Once adopted, I think it has the advantage that it guides signatories through to successful implementation through a series of ‘supporting commitments’ and the development of communities of practice and mutual learning events,” she says. “Assuming they get the critical mass, and continue to engage internationally, it has the potential to have a significant impact.” 

Several researchers tell The Scientist that the moment feels ripe for such an initiative.        

“I’m hopeful because there is a lot of momentum. . . . I don’t think there has been a moment in my career where I felt so strongly that people were really willing to move on this,” says Stroobants. For Susi, it’s now or never. “From my point of view, this is the only chance. I’ve never seen a chance like this, and I don’t think there is any realistic alternative in the cards.” 

Correction (August 22): A previous version of this story stated that the agreement was published on July 22, but it was published on July 20. The Scientist regrets this error.