Q&A: How to study scientists

Scientists spend their time trying to uncover the most effective and efficient techniques and their impact on research, but what about the most effective and efficient scientists, and how they impact the field? linkurl:Pierre Azoulay;http://pazoulay.scripts.mit.edu/ of MIT's Sloan School of Management talked with The Scientist about his work on what influences the productivity of scientific researchers, and how productive scientists can, in turn, influence the scientific community.

Image: Wikimedia commons,
Robert Scoble

The Scientist: How do you measure scientific productivity? Pierre Azoulay: Badly. The [measures] are proxies -- things like counts of publications, of patents, of citations to those publications or to those patents. [But] those measures are really proxies for what we care about, [which] are basically important ideas. So we're often multiple steps removed from innovation, and even many more steps removed from actual economic growth. TS: In a linkurl:recent study;http://pazoulay.scripts.mit.edu/pubs/extinction_qje.pdf published in the Quarterly Journal of Economics, you looked at "scientific superstars" -- which you defined using criteria such as amount of funding, number of citations or patents, and whether the researchers were Howard Hughes Medical Investigators or early career prize winners. You focused on 112 life science superstars who died suddenly, and the effect on their collaborators. What did you find? PA: Basically what we found is that the [productivity] of those scientists who [had lost a collaborator] declined after the death. So what we think is going on in some sense [is that] the field associated with the star atrophies upon his or her passing. For a long time people have talked about the invisible college -- scientists that are linked...in this "invisible college" of ideas. In some sense, the natural experiment of those scientists dying revealed exactly the boundaries of this invisible college. TS: What are the implications of your findings? PA: Facetiously, I could say that if you're a superstar you probably shouldn't fly a glider plane. Because it's not only about you; it's about those around you. [But] there are some ideas that come out of this. In particular, it might be very important for eminent scientists to be able to circulate their ideas more efficiently. How do we make sure that all the possible opportunities for collaboration are exploited? When you have someone really famous that comes to a campus, how do you structure those short-term visits in a way that maximizes opportunities for meaningful intellectual exchange? And that might fall short of actual collaboration, but might be more than just having coffee and shooting the breeze. TS: How does funding influence a researcher's productivity? PA: If we want to motivate people to "work hard" on a particular task, we sort of know what to do -- just pay them piece rates, as opposed to giving them a guaranteed salary no matter what. The prototypical example is a very good empirical study from the late '90s, [which] focused on windshield installers. It turns out that when you switch windshield installers from a flat salary structure to a piece rate, where they get paid by the installation, their productivity completely goes through the roof. So that's very interesting, but windshield installation is sort of a routine and boring task. What about creative tasks where the deliverable is not preordained? I thought, I know exactly the kind of setting linkurl:[to test this idea]:;http://pazoulay.scripts.mit.edu/docs/hhmi.pdf the ways in which NIH funds scientists -- [standard R01s last 3 to 5 years and recipients are monitored quite closely] -- versus the ways in which HHMI funds scientists -- [who are given 5 years of funding and an additional 2-year grace period]. What we found is that, in terms of impact, the HHMI [funded scientists] really produce a lot more high impact papers than the controls. Is this because of incentives or because of other things? We can't be quite definitive. Maybe [what] matters is just the increased resources. And we can't really solve that out with our data. TS: So it seems like you're saying that scientists might work differently from windshield installers -- that more open funding may be better. What are the next steps in your research? PA: What we want to do now is convince scientific agencies that they actually need to run experiments. We're groping in the dark. We're doing the best we can with the data that we have, but if you share the belief -- and I think lots of scientists share this belief -- that how we spend money is just as important as how much we spend, then if we could just devote 0.5 percent of the funding at experimenting in a formal way with different funding models and evaluate them to see what works better, that would be a very worthwhile exercise.
**__Related stories:__***linkurl:An Economic Gamble;http://www.the-scientist.com/article/display/53302/
[July 2007]*linkurl:Evaluation of Scientific Productivity;http://www.the-scientist.com/article/display/12068/
[2nd October 2000]*linkurl:Gender Differences in Research Productivity;http://www.the-scientist.com/article/display/18691/
[27th September 1999]