Predicting Scientific Success

On the other hand, Milojevic added, the finding that a researcher’s impact is stable, “is a bit more negative,” meaning that if an early-career scientist’s Q value is low, they are unlikely to have a really big impact over the long term. “They are not going to be Madonna,” she said.

The number of papers produced and citations garnered by a scientist are commonly used metrics—often distilled into a measure called the h-index—for assessing that person’s research performance. As such, these measures influence decisions such as who to hire, who to give tenure and, increasingly, who to fund.

“But there is a flaw,” said Roberta Sinatra of the Central European University in Budapest, who led the new study. “If I hire a scientist because of her h-index, I am implicitly making an assumption that somehow this is a promise of future performance,” she said. But in fact little is known about how a scientist’s impact changes over her career.

Sinatra and colleagues therefore wanted to understand how an individual’s success or career impact evolves. “In particular, we were interested in big discoveries,” she said. “Can we predict when a scientist will make the best discovery of her life? And how big will that be?”

The team gathered publication data on thousands of individual scientists in seven disciplines with careers spanning at least a decade. For each of these scientists’ papers, the impact was determined as the cumulative citations obtained after 10 years.

Sinatra’s team discovered that the highest impact paper for any scientist, regardless of discipline or career length, is randomly distributed within that person’s list of publications. That is, it has equal probability of being published first, last, or anywhere between.

By averaging the calculated impacts of all publications from individual scientists, the team discovered that some scientists tend to have an overall higher Q value than others. This Q value shares similarities to the impact indicator, µ, described last year in PLOS ONE by Luís Nunes Amaral of Northwestern University and colleagues.

The stability of a given scientist’s Q value means that papers published early in a career can predict the success of those published later. Such early Q values could also predict subsequent Nobel Prize-winners, Sinatra and colleagues showed.

“This constant level of quality [Q]. . . is a fairly profound finding,” said John Walsh of Georgia Tech who did not participate in the study. “It basically argues that there is no learning. You’re just as good as you are.”

It is not yet clear what determines Q. It could be where the person studied, his technical abilities, his communication skills, IQ, gender, or something else. “My intuition is that it is a combination of many factors,” Sinatra said.

It is also important to remember that Q is a measure of impact or success—defined by citation records—and not necessarily of scientific prowess, said Sinatra. To make an analogy, she said, “Kim Kardashian, she’s successful, certainly, but we cannot say that that’s down to the quality of her performance.”

R. Sinatra et al., Quantifying the evolution of individual scientific impact,” Science, doi:10.1126/science.aaf5239, 2016.