ABOVE: Novo Nordisk Foundation
The Novo Nordisk Foundation, one of the largest private scientific research funders in the world, announced last month that it would begin employing a partial randomization system to fund some types of research projects. For the next three years, the Copenhagen-based funding agency will use a combination of committee selection and a lottery system to choose some of the awardees of its $500,000 Project Grants in the fields of biomedicine, biotechnology, and natural and technical sciences, as well as its $800,000 Exploratory Interdisciplinary Synergy Grants. Together, these grants comprise roughly 10 percent of the organization’s total research project funding, says Lene Oddershede, the senior vice president of natural and technical sciences at the Novo Nordisk Foundation, who oversees the grant funding process. She says she hopes that the partial randomization system will reduce conscious and unconscious bias in the committee selection process and improve funding inequities.
“I think most researchers want to see that the applications are treated in a fair manner and in a transparent manner,” says Oddershede. Scientists also want to ensure “that the best research is funded, of course,” she says, “but what is best research?”
In Denmark, 90 percent of the funding goes to just 20 percent of researchers, and a similar concentration exists in many countries. According to recent studies in the United States, for instance, funding inequities have increased in the past decade. In 2020, the top 1 percent most highly funded investigators received 10 percent of the National Institutes of Health’s Research Project Grant (RPG) funding, an increase of 2 percent from 2013, while the least-funded half of investigators received only 20 percent of the total RPG funding, a number that has stayed static for the past decade. And last year, white, cisgender, late-career researchers and men received funding more often than women, early career researchers, and members of other racial groups or Latinx researchers.
See “Biomedical Innovations from Women Less Likely to be Adopted: Study”
A partial lottery may alleviate the tendency for committee members to favor established researchers, Oddershede says. Novo Nordisk also hopes the randomization will lead to funding more high-impact projects and increase the diversity of their selection pool. The foundation plans to collect information on how the partial randomization system impacted the types of projects funded over the efforts first three years.
The Scientist spoke to Oddershede about the new partial randomization step and why the foundation decided to institute it.
The Scientist: What is partial randomization of funding and how are you employing it?
Lene Oddershede: A partial randomization means that it’s not a full randomization, which would be a pure lottery. We don't do that because we still need to have quality in the applications which we support.
If we receive a pile of applications, then we have an expert committee consisting of professors with relevant expertise . . . read the applications. They grade . . . and prioritize the applications. Then, normally, they agree that a certain fraction of applications should be funded, no doubt. For simplicity let’s say we get 100 applications. And let’s say that the committee agrees 10 of them should get funding. The committee also agrees that another, say, 20 percent have a high quality and should be funded, if there were enough funding. But usually, the situation is that there are more fundable projects than there is funding. Then the committee needs to prioritize between these 20 percent, which is what we call the “gray zone.” So in that gray zone, they argue. This is usually where the committee spends most of its time at a funding decision meeting. . . . In that gray zone, we are now making the experiment.
If the committee has decided that these applications [that] are in the gray zone are high enough quality to be fundable, then we make a lottery. So suppose we are able to fund 20 percent [of the total applications], then, in this case, [for] those which enter the gray zone, there will be a 50 percent chance that they will be chosen by a lottery. That’s the idea behind partial randomization.
We have a trial period of three years. And in that three years, we run two processes in parallel. One of them is a partial randomization, which I just explained. The other one is a completely normal process. We will have two datasets, one selected by partial randomization and the other one selected by the normal method of the committee. And after three years, we’re going to compare those two datasets and ask: what did we achieve with this? Was partial randomization better in maybe eliminating some of the unconscious biases? Did we have more high-risk, high-gain projects?
TS: Which grants are you planning to fund with partial randomization?
LO: Our smaller types of research grants, those valued at about half a million US dollars [in total over three years], and in particular, those which are interdisciplinary. If you do interdisciplinary work, then it’s very difficult for committee members to prioritize these projects. It turns out to be really hard for a researcher to judge an application which has a large fraction of the science rooted in another discipline.
TS: Why is the Novo Nordisk Foundation partially randomizing the funding process?
LO: The reason we are doing this is to decrease bias. . . . We also seek to be less conservative and more courageous in the decisions we make.
We have an obligation to constantly strive to be better at awarding grants and constantly strive to award grants without biases, conscious or unconscious. There can be gender biases, age biases, there can be biases against certain institutions or certain citizenships. These biases could be in terms of scientific area—a typical professor tends to be most excited about the type of research that he or she does. There can be all kinds of biases. Some of these are conscious and some of these are unconscious. I would anticipate that the partial randomization process would also mitigate racial biases as well.
TS: What research supports the use of partial randomization to fund grant applications?
LO: It’s partially a push from the research community. For instance, the Royal Danish Academy of Sciences and Letters published a white paper [pdf download] where they recommended funding institutions consider some degree of randomization in grant awarding. But in addition, there [are] already similar schemes, say from the VolkswagenStifTung in Germany. They don’t do it exactly like we do, but it’s quite similar, and they have had a very good experience so far. And the research communities are really happy about it. Also, the Swiss Research Council has implemented a partial randomization in many of their instruments.
Now, you’re from the US, you probably would ask, what about US funding agencies? And the answer is not that I’m aware of.
TS: Do you think this process could be adopted by other funding agencies?
LO: Definitely. So far, there’s not much data on this process. That’s why we also try to increase the amount of data on partial randomization. That’s the reason why we make these two independent data sets, which we could then compare. We actually implement this scheme on programs where we receive, say, 500 applications. We receive a large number of applications, and because we have the large statistics, because we have the two independent data sets, then after that trial period, we can actually . . . be quantitative about the quality of partial randomization. This is a scientific approach to say is it really better or not?
TS: What are the metrics that you’ll use to compare your two samples?
LO: The impact measures which we are going to look at would be gender distribution, career level, institutional affiliation, the field which is funded . . . that’s another goal: we also try to fund new fields of research. . . . Of course, we’re going to use the more typical metrics, like citations, number of publications, the impact of journals in which the [publications appear], and social media, et cetera. But of course, it would be difficult [to see much in those metrics] after three years, because much research takes a little bit longer before it has a major impact, say, on citations, et cetera.
TS: What challenges did you encounter in rolling this out?
LO: There’s been a lot of worry about whether the research that would be funded in this manner would have a high enough quality. The major worry is how do we ensure a process by which we are certain that we can actually, really stand behind every single application that is funded? So the committee is still necessary to make a decision on whether a project has a high enough quality. That’s really crucial.
TS: What problems that exist right now could be solved with partial randomization?
LO: One major problem is conservatism: the tendency of funding more safe research and not being brave with respect to moving into new fields, or maybe trusting that a younger researcher who doesn’t yet have a very strong CV is capable of really contributing significantly to research.
There’s this effect in research, which is called the Matthew effect, which is actually described in the Bible: it says, to those who have more shall be given. And it’s definitely a Danish issue. And it’s a European issue. But in Denmark . . . 20 percent of the researchers have 90 percent of the funding. If you are a really successful researcher, if you have a fantastic CV and lots of citations, it’s easy to get funding. . . . But not funding the other 80 percent means that a lot of ideas are never realized.
Another issue is the leaky pipeline—the fact that the further you progress along the academic ladder, the fewer female researchers there are. The third is what we call the hourglass problem. If you look at the career level, when you just received your PhD, if you’re willing to move, it is possible to get a postdoc stipend. If you are at the top, say you are a very successful professor, you have a very strong CV, then life is easy. But if you’re in the middle, where the hour glass becomes most narrow, it is difficult because you haven’t yet established yourself with a strong enough CV to really go out and get the big grants. These are the three major issues in research which we are trying to solve through better grant awarding processes.
Correction (February 28): This article previously included an image of Novo Nordisk A/S, not the Novo Nordisk Foundation. The Scientist regrets the error.
