GAO to NIH: Keep your house in order

The Institutes that make up the National Institutes of Health are funding significantly more applications that are scored below the institutes' paylines, and the Office of the Director should be doing more to monitor those funding decisions, according to the Government Accountability Office.

In a recent report, the GAO notes that more than 18 percent of the funded R01s in 2007 were scored below the payline, while institutes made similar exceptions in only 10 percent of grants funded in 2003. One of the most frequent reasons institutions say they choose to fund applications below the payline is to give a boost to young investigators -- a practice that garnered controversy in this month's issue.

The GAO, for one, wants the Office of the Director to keep closer tabs on its institutes' decisions to fund applications that did not score above the payline.

Does it bother you that nearly 1 in 5 grants go to investigators who did not score above the payline? Should the NIH continue to support young investigators at this level?

You can read more about this in the report in the New York Times and the on the Medical Writing, Editing & Grantsmanship blog.

Alison McCook, Deputy Editor

Of course it does not bother me. Why should I care that at a 6% pay line NIAID is choosing to ignore paying grants from investigators across the country with proven track records (and a grant at 11 percentile), and instead funneling the money to cronies at Harvard and new investigators? I am sure the 94% of the investigators not making that pay line don't care one whit.

This bothers me a great deal. I was once the victim of something simlar at the NSF. I was told by two members of the panel that I had the highest score of the group and was congratulated by them. I didn't get funded. A former student of the section head received a similar amount of money out of *discretionary funds* to do the exact work that I had applied to do. I'm left wondering how common this sort of thing is.

Common Lisa, very common.

It would seem that if the reason was to provide opportunities for new investigators to be supported, why not create a program for new investigators with those goals explicitly? NIH has previously funded programs like the Comprehensive International Program of Research on AIDS (CIRPA) with the previously expressed goals of expanding clinical research capacity. That way, paylines can be kept for established researchers, and sloppy research application from researchers with pull or name recognition will not get funded. And if paylines are truly honored, then it opens the field to less recognized/politically connected proposals, and perhaps would push some "established" researchers to get back in the game and legitimately compete for funding. It would also allow for a more transparent process for the funders, and would reduce the ability for political sway to be a part of the process.

Since the CIPRA program is no longer adding new researchers, there is a definite void in the "pipeline" of developing opportunities for international researchers and their sites for HIV and AIDS. It is definitely a concept worth revisting, although a new system must learn from the previous systems and gauge the realities of true staff and site development timelines, which ~ if done well ~ may take 3-5 years.

Helping new investigators is good; that's why there should be some funds set aside for them.

Cronyism is bad, and to the extent that some new investigators benefit from cronyism (they come from the "blessed" labs), it's bad.

Of course, if paylines were realistic, as they are in many European countries (I review grants for a few EU countries, where paylines are 50 to 75th percentiles), who would care? Here in the US, the scientific differences among grants scored 5th %ile and 12 %ile are probably nil, but the second grant still won't be funded in these fiscal times.

Lisa,

The system of funding R01s and R21s and thus providing fair critique through the study section system is corrupt and rife with petty-narcissism. Until the peer-review system and protocols for reviewing grants is conducted FAIRLY AND OBJECTIVELY in an open format so that we now who is reviewing YOUR grant, the system is open to corruption, abuse and self serving protectionism. The 2007 review by THE NIH of the peer-review system was meant to get around the problems that are being discussed in this thread, but to all intentions, they are failing miserably.

The biggest issue that concerns me is the Juggernaut or elephant who has a research program that parallels yours. He/she sees it as undesirable competition and kills your grant during the review process and study section. One bad score and two good ones, means nothing. One is all it takes to kill your grant. Thus, by limiting the competitive pool, applications wither from the said juggernaut PI or his/her minnions stand a much greater chance of either making the payline and being noticed or being funded.

The ORI and office of the inspector (if indeed they have a vested interest in tackling the problems of corrupt and unobjective peer-review head-on, which they are not of course) must instate a system of audit and greater accountability of the review system or the top tier Universities will forever more, perhaps unfairly (but certainly not always) cream off the dwindling resdearch dollars available to so mnay deserving new investigators and established investigators who are seeing their labs destroyed in the aftermath of the economic meltdown and the vicious cycle of default negative peer-review system that is hurting the US research base right now.

Let's be clear about those "Paylines" -- they're based on numerical scores that are in term based on the personal opinions of one, two or at best three study-section members. Neither the numerical scores nor the "paylines" represent an objective assessment of either the quality or the importance of the research. The issue of "cronyism" raised by some of the commentors in this forum applies far more often to study section members than it does to NIH staff (or NSF staff! -- disclosure: I used to be an NSF program officer myself, and I deeply resent the insinuation that either I or any of the wonderful, dedicated colleagues that I worked with while I was there would award grants based on "cronyism;" in fact, conflict of interest is taken very seriously at NSF at all levels). The problem with scores and paylines is that study section members are notoriously and documentably averse to new ideas, ideas that conflict with their own beliefs or with the dogma in their fields, and to newcomers. As a result, the applications that get the best scores tend to be the ones that propose experiments that will incrementally fill existing knowledge gaps in currently-hot areas of research that are well-explored by other scientists. Yes, it is important to incrementally fill existing knowledge gaps. However, it is also very very important to push the envelope of knowledge in new directions, possibly even potentially paradigm-shattering directions, or in directions that will open up not only new "facts" but also new ways about thinking about the problem and new ways of thinking about previously-uncovered facts. Some may argue that it is even MORE important to push the envelope in this way than to systematically and incrementally fill in gaps. Not all scientists have the creative ability to push the envelope of science, and I would even venture to say that most scientists do not even appreciate the envelope-pushing work until after the "new" has become the "new hot thing" and everybody else is jumping on the new bandwagon. But how can envelope-pushing get to the point of "bandwagon" if the work isn't being supported? And THAT, in my humble opinion, is why it is imperative that wise grant program officers look carefully not just at the numerical scores but also the texts of the reviews, and use their unbiased judgment to select some (20% may not be nearly enough) projects that may (or may not -- that's the definition of "risk") lead to real breakthroughs instead of just filling in knowledge gaps. That's what program officers try to do at DARPA, and to a lesser extent (unfortunately) at NSF, and to a much lesser extent (even more unfortunately) at NIH. I am pleased to learn that program officers at NIH are "reaching" more and "covering their asses (via priority scores)" less than they used to. They should do so more.

Eve, you are quite correct that figuring out how many angels can dance on the head of a kn own pin has a much higher priority on study sections than someone proposing to study a whole new pin.

But you speak from the point of view of an insider, not an applicant.

Applicants want to know that if their grant is a 10% and the pay line is 9%, that someone getting a 17% is not jumping over them and getting money. It used to be that study sections had a sympathy vote for new investigators. With the disparity in funding lines between new and established, that charity is gone, baby, gone.

The issue is how to address the review process such that the scores mean something and that they institutes stick to the instead of making up their own rules. If they continue acting in a capricious manner, then why should established investigators stay in the business much less want to train students to take their place?

Eve, there is "cronyism" in the NIH system, and it is not just limited to study section members. For example, I know folk who seek out program officials at meetings, in order to wine 'em and dine 'em.
In fact, at a grant writing course I attened years ago, this was encouraged. I won't disclose more details so as to retain anonymity.
NSF may be different, I wouldn't know (my research is not eligible for NSF consideration). DoD and VA certainly are different (cronyism won't help because the systems fund very differently from NIH).

Beverly -- I certainly would not equate "getting to know your Program Director" with "cronyism." At NSF, would-be PIs are strongly encouraged to contact their Program Directors -- generally by telephone, but also in person. I don't know if it's still recommended on the NSF website but it certainly used to be, and before the website era, it used to be encouraged in official brochures. Sometimes Universities send delegations of would-be PIs to the DC area, often in the company of a university outreach official, to meet with program officers at NSF, NIH, and whatever other agencies might be appropriate for the science involved. I often met with such groups of scientists (usually Assistant Profs but not always), sometimes by myself and sometimes in a group-grope with other Program Officers, depending on the circumstances, and I think it was beneficial to the scientists to learn what the agency was looking for and what kinds of criteria the agency used to evaluate proposed projects during the decision-making process. Of course, the scientists would do a show-and-tell of their work as part of the meeting, and sometimes program officers would offer advice and suggestions. WE WOULD DO THAT FOR ANYONE WHO ASKED, so long as we were not asked directly to critique a draft proposal (if we did, we would have to recuse ourselves from the review and decision-making process because the proposal would in that case obviously be in part our own work). The scientists would leave with their impressions of the program directors they met and of the agency as a whole, and the program directors would be left with their own impressions of the scientists -- both positive and negative, of course. This getting-to-know-you process was always considered a desirable (albeit timeconsuming!!!) part of the outreach process. Similarly, several Federal agencies routinely rent exhibit booth space and/or arrange for alternative modes of interaction between program officers and scientists at major scientific meetings -- the purpose being not only to peddle brochures and answer questions, but also to put "a human face" on the cast of characters. All of these outreach activities do not amount to "cronyism." And keep in mind that a brown-nosing scientist would be equally likely to make a good impression as a bad one.
In the case of currently-funded PIs, it was critical for program officers to be in touch in order to learn what the PI is doing, and if the PI is doing something really exciting, to be able to use this information within the agency for purposes of programmatic progress reviews and, if it's really really exciting work, to brag about it in reports to OMB and Congress as a rationale for requesting more funds.
And as for "wining and dining" -- the implication here being bribery -- wow, uh-uh, no-no-no, that's a bigtime Federal no-no with dire consequences for any Federal employee (whether permanent or temporary) who is caught accepting "wining and dining" or any other legally inappropriate gifts.

I realize that you genuinely believe the worst-case scenario you described, and I don't know exactly how to dissuade you of your belief, but I genuinely believe that you have misinterpreted something along the way.

Peer review has taken a heck of a beating recently in The Scientist Online. If it's so bad, maybe the NIH is justified in using other funding criteria. I'm just saying.

Eve, I purposely left out some details, which are salient.
I don't wish to disclose them in these fora.

Beverly -- I certainly respect discretion. Thank you. But I do wish to add that a few anecdotes do not a pattern prove. Even a few genuinely bad actors do not prove that all actors are bad, or even the vast majority. And misinterpretation always remains a possibility whenever human beings are involved (heck, everything remains a possibility whenever humans are involved).

Rather than focus on conspiracy theories involving study section members and NIH officials, let's bring this back to reality. There are more people desiring to conduct science than society has the means or desire to support. That is creating stress on the system which, if we do nothing to favor new investigators in some capacity, will result in the loss of an entire generation of scientists.

Study sections advise on the merit of the science for which there is precious little difference between an 8 or 15 percentile score. Someone has to make decisions about whom to support. That is based very heavily upon study section scores but, as the article indicates, with some discretion must be allowed. That can include the necessity of the Program Officer to balance a research portfolio to ensure that science does not become too heavily weighted in the latest fad. Sometimes, geographical considerations are balanced which I personally dislike more than anything mentioned previously. And yes, established programs and even relatively new programs sometimes are provided interim support by a program officer to keep them from falling apart. This discretion is needed as fractional differences in scores are not strict numeric indicators of relative worth.

So, it is good that there is a second level of decision-making by someone seeing the big picture. The person in that position is in a thankless, but necessary, job. Some of you may be following the ongoing revisions to the review process. One of them, in very exploratory implementation , includes the use of 'editorial boards' as part of that post-study section decision-making process. Would you rather see those decisions made by an editorial board of your established peers? Would you rather have a career program officer making those decisions? Again, tough decisions need to be made by someone and the major question is what is the best way to accomplish that. If you have opinions on that, then I recommend that you contact the NIH with your opinions.

Fred -- thank you for your comments. I agree 100%.

BTW, even at NIH, the "career program officers" are, for the most part, themselves reputable published scientists with expertise in the research area for which they are responsible. This was not necessarily the case 20 or 30 years ago, but it definitely is the case nowadays. And yes, they can and do see a bigger "picture" than can be seen by most members of study sections or individual PIs. AND -- they don't act capriciously or in a vacuum; any "stretch" that they want to make has to be justified in writing and approved by higher-ups.

Again, thank you for your comments.

Cronyism, NIH, NSF, 10%, 18%, XYZ, ABC...! WOW! I'm tired. How about, as a researcher, you show me you are capable of researching. You have discovered, created, or invented in the past or better yet, recent past. Show-PROVE credibility in your actions and scheme(s) both present and past. And by the way, if you happen to exhaust your grant without completing what you set out to do, or at least show me (possibly prove) why you were unable to meet your outline deadline, then don't bother trying to dip into this well again (NSF, NIH). I am not interested in hearing you ran out of time and especially money. Budget properly, and stop making excuses.

The above should be set down as a new set of completion standards. Show me (sort of).

The issue at hand is that it is not your money. You apply and someone hands you a check because you can write well. How about we have researchers put up their own money, say 20% of the grant, to receive the other 80%. I will bet you dimes to dollars the research is precise and will, come to a conclusion. Frivolous research will be swept aside for meaningful projects.

Put your own dollars into the research as I have done, set the budget, and demand the results from those that stated they could deliver. Don't badger, help them achieve. Bring the resources to them and keep them focused.

Now how do you propose to put in your own money?
My university wouldn't allow that. Nor would the VA.

Have you served yet on a study section?

Please excuse the generalities of my statements. We would be here for months on end discussing the necessary details. There absolutely must be an overhaul of the requirements and components of the grants system. This came from a very nice woman in Washington associated with the potential overhaul. Difficult perhaps, not impossible.

Your University wouldn't allow it or won't allow it?

It all comes down to structure. Anything can be structured through documents stating responsibility and potential outcome. As for the VA (Veterans Administration?), if you go high enough, you find someone who will take the time to assist you in whatever you want or need. The issue at the VA, is lack of competant personnel that understand proper business and/or regulations, and if you do find that person, they are usually burned out from running into walls because of bureaucrats.

No, I have not served on a study section.

To Christopher: I don't know what you mean by "frivolous research," nor do I understand how anyone can do "meaningful research" in the manner you describe. Some might even question, if a project can be KNOWN to be completed exactly as described in a proposal in a given amount of time then why bother doing it? (or worse, one might even suspect that the investigator has actually already done the work). Such "guaranteed outcome" research could not possibly stretch the envelope, or uncover significantly new knowledge or understanding. What you are describing is what some call "grind and find" work, providing some incremental progress and filling-in some highly specific knowledge gap. Don't misunderstand -- this kind of work is extremely important for applied research and for translational research. But the OTHER kind of research -- the kind you deplore as "frivolous" -- provides the breakthroughs that lay new ground, open new ideas, new ways of thinking about things, and is the root of real progress for the long term as opposed to incremental progress for the short and intermediate terms. That, to my mind, is "meaningful."