Can Publication Records Predict Future PIs?

Researchers present a tool that uses a scientist’s PubMed data to estimate the probability of becoming a principal investigator in academia.

By | June 2, 2014

FLICKR, MOONLIGHTBULBThe odds of a scientist becoming an academic principal investigator (PI) can be predicted with publication data, according to Lucas Carey from Spain’s Pompeu Fabra University and his colleagues. The team developed an online tool, dubbed PIPredictor, which uses a machine-learning approach to analyze a user’s PubMed data and that has already churned out more than 800 career-success estimates to date. Carey and his colleagues describe their tool in Current Biology today (June 2).

“We show that becoming a research professor is highly predictable, [and] we analyze the features that are predictive” of success, Carey told The Scientist in an e-mail. The algorithm can predict who might become a PI and how long it could take for them to do so with an area under the curve (AUC), a measurement of accuracy, of 0.83 and 0.38, respectively.

While he was a postdoc in Eran Segal’s lab at the Weizmann Institute of Science in Rehovot, Israel, Carey and two then-PhD students, David van Dijk and Ohad Manor, decided to apply the machine learning-based approaches they were using to predict molecular mechanisms from gene expression data to guess who among them might one day become a PI. “There was quite a bit of debate at the beginning over if it would work, or if becoming a PI would turn out to be entirely non-predictable,” said Carey. “I was quite skeptical that we would be able to predict much, and we are all quite surprised [by] how predictable the entire process turned out to be.”

While having papers in Nature and Science can certainly help, it turns out that high-profile publications are not the only factors that determine whether an early-career scientist will one day lead her own academic lab, the team found. Rather, it’s the total number of publications, the impact factors of the journals in which they’re published, and whether each paper meets or exceeds the average number of citations for a given manuscript in that journal that seem to matter most. In other words, quantity and quality count. Overall, the researchers noted, higher h-indices—metrics that attempt to quantify the productivity and impact of a scientist’s publications—are predictive of a greater chance of academic career success, lending support to a concept first proposed in 2012 by Rehabilitation Institute of Chicago’s Daniel Acuna and his colleagues in Nature.

“However, both the scientist’s gender and the rank of their university are also of importance, suggesting that non-publication features play a statistically significant role in the academic hiring process,” Carey and his colleagues wrote in their paper. The researchers found that, given the same publication record and all else being equal, male authors are more likely to become PIs than their female counterparts. Their model controls for both gender and institution rank.

Randall Ribaudo, CEO and cofounder of the career training firm, spent five years as a PI at the National Cancer Institute’s Laboratory of Immune Cell Biology in Bethesda, Maryland, before moving on to work in industry. According to PIPredictor, he currently has a 59 percent chance of becoming a PI, Ribaudo told The Scientist. He questioned whether the team’s tool could account for factors such as the academic job market, which has changed considerably during the last few decades. “Over the past 20 years, hiring for tenure-track positions has gone down a lot,” he wrote in an e-mail. “If [the authors] are using longitudinal data and not considering that downward trend over time, the better statistical likelihoods in earlier years could be giving artificially high numbers.”

Paula Stephan, who studies the scientific workforce at Georgia State University, agreed. “The times are changing, and with that, the underlying probability of becoming a PI,” she wrote in an e-mail to The Scientist.

Stephan added that while the model is “based on a well-constructed bibliometric database,” publication records alone cannot account for “factors that reflect the scientist’s ability to produce the type of research that may be funded, such as innovativeness [and] creativity.”

Even so, in an increasingly competitive job market, it could be beneficial for a young scientist to get a feel for where her publication record stands. “The tool provides one benchmark that early career scientists can use to see their relative position based on publication metrics and reputation of university,” said Stephan. “I suspect that many young scientists already have a good idea of where they are, and some a more accurate idea than this [tool] can provide within their narrowly defined research world.”

D. van Dijk et al., “Publication metrics and success on the academic job market,” Current Biology, 24(11), 2014.

Add a Comment

Avatar of: You



Sign In with your LabX Media Group Passport to leave a comment

Not a member? Register Now!

LabX Media Group Passport Logo


Avatar of: John Salerno

John Salerno

Posts: 10

June 3, 2014

 Re: “factors that reflect the scientist’s ability to produce the type of research that may be funded, such as innovativeness [and] creativity.”

I've been a PI for over forty years, and those factors have increasingly become negatives in most panels and study sections, despite their prominence in current evaluation metrics. The funding mechanisms are risk adverse, and many program offficers will advise applicants of this. 

Avatar of: Doug Easton

Doug Easton

Posts: 14

June 3, 2014

This is just one more driving force behind the rise in fabrication and falsification of experimental data by graduate students and post-docs. .

It is obvious that getting into the right lab as a grad student or a post-doc is critical to later sucess. A prospective student should look at the number of publications where grad students and post-docs are included as authors and the average number of papers on which an individual grad student is an author before finishing a degree or a post doc. Many years ago I should have chosen my post doc more carefully in that respect.

Becomming a PI depends a lot on the quality of the institution and department in which a post-doc takes a first  tenure track job. A long list of publications and a seminar based on work done in a good lab lands one a good job.

Avatar of: Andrea Stierle

Andrea Stierle

Posts: 1

June 3, 2014

I am trying to understand why this is revelatory.  This has basically been the mantra of graduate schools and academia for many years - attend a high profile university and  publish multiple papers in high profile journals. 

Avatar of: Salticidologist


Posts: 51

June 3, 2014

Trying to get a permanent job in academia is like driving a car into a crowded parking lot.  Good luck!  In this era of automation, there is no job that is safe.  But for Ph.D.s, there is usually no job.

Avatar of: Chin, Chien Ting

Chin, Chien Ting

Posts: 2

Replied to a comment from Andrea Stierle made on June 3, 2014

June 3, 2014

Quote Andrea Stierle

I am trying to understand why this is revelatory. 

Indeed, was Carey playing dumb when claiming he didn't expect PI probability to be predictable?  Anyone who've entered into PostDoc/tenure track knows full well the "game rules".  Machine Learning has simply made a mathmetical predictor out of the recruitment criteria of the typical higher insitutions.  Next up, Machine Learning will predict the precise times of sunrise and moonrise?

And it has done nothing to provide insight.  So men are more likely to be PI then women, but are men better and more competitive than women?  The predictor simply reveal the (unsurprising) pattern in the status quo, it does not help us understand the pattern nor how to improve it.  And it has all the prime ingredients to be misunderstood and misused.  Should PIPredictor be used in hiring and promotion decisions?  To the effect that, e.g. white males will be given priority in scholarships, projects and grants since their prospects are brighter, statistically proven. 

Unfortunately, this paper will likely be highly cited, so I predict. 

Avatar of: Borealbob


Posts: 3

June 16, 2014

After reading this- can anyone wonder why so many motivated students elect never to even try to gain a career in academia?  Risk averse is the paradigm of the day. Do the research, publish it and then apply for the grant to do it. Sad state of affairs. Note- the US is the only nation on the planet to foster such a system of pedigree and false productivity.

Avatar of: Singh


Posts: 2

June 27, 2014

The key question is that whether this program takes in account of other factors such as who do you know? Who is your boss? How much funding you have ? Etc etc. for the record my " chances" to become PI are 91% but I only received one interview call and now I work  in a crappy industry.  My opinion- it's just an article that will likely make the author famous. 

P.s. My research proposal, evaluated by tenured professors, is " original and great".

Avatar of: Singh


Posts: 2

Replied to a comment from Salticidologist made on June 3, 2014

June 27, 2014

Well said. There were never many. May be US should halt awarding PhDs? 


Avatar of:

Posts: 2

Replied to a comment from Salticidologist made on June 3, 2014

July 26, 2014

The saddest thing is that most of those parked cars are either clankers or pimped cars! COL :-(

Popular Now

  1. Secret Eugenics Conference Uncovered at University College London
  2. Like Humans, Walruses and Bats Cuddle Infants on Their Left Sides
  3. How Do Infant Immune Systems Learn to Tolerate Gut Bacteria?
  4. Scientists Continue to Use Outdated Methods