The emperor's new… cancer grants?

An article in the New York Times this weekend revealed something that most biomedical researchers already know: The system for obtaining grants through the National Institutes of Health encourages safe projects likely to yield incremental results and actually discourages innovative work.

The article specifically focuses on cancer research -- going so far as to call NIH a "jobs program" more than a granting agency -- but it's an assessment most researchers have undoubtedly heard in other areas as well.

Well, the cat's out of the bag now. Still, I found the admission by Raynard Kington, NIH's acting head, rather shocking. Didn't the agency just go through a massive peer review assessment that was supposed to address this issue, among others?

-Alla Katsnelson, news editor, The Scientist

While it is in the minority (in terms of % of funding), there is a program at the NCI which sponsors high-risk, highly innovative technology development and has been around for over 10 years. It's called the IMAT program, the Innovative Molecular Analysis Technologies program, and its basic pilot-stage support does not require preliminary data. It's peer review process is also unique - in that study section reviewers are asked to focus on technology-development (or the application of biological concepts for building technologies), instead of R01-hypothesis based research.

The biggest problem with the NIH peer-review system is the peer-review system itself of assessing the worthiness (or not) of grants.

There is a curious disconnect between the grants that are funded and quality of the publications as well as quality of the PI of the submitted grant. I am not and have never been convinced that the best science or best qualified researchers are funded. There are many cases, myself included, where the publication base and knowledge of the PI is insulted during the peer-review process.

If a PI of a grant has published in high-quality journals, and has a long track record of published works and small or pilot grant funding, I think it is irrelevant for peer-reviewers of grants to nit-pick the minutia details of the more technical and methodological focus of grant applications with petty and nonsense critiques. Yet, this always seems to be the trend where methods and technical aspects of experimental protocols are targeted.

Clearly, it is a requirement of peer-reviewers to be objective and specific in their critiques and provide helpful feedback – this is good and competent peer-review. However, in so many instances, the feedback is negative, derogatory and frankly insulting to the PI of a grant who after all has a formal education, post-doctoral fellowships completed and a technical knowledge base perhaps equal to or better than the reviewer.

All too frequently, when reviewers put on their peer-reviewers cap, they become super critical of items in the grant that are not wholly relevant. Aspects of the grant that should receive focus are innovation, hypothesis-driven aims, experimental design, feasibility and the investigator. This rarely seems to be borne out in the critique reports however. I would submit that peer-reviewers of grants are themselves peer-reviewed for objectivity, clarity and fairness in their critique of grant applications.

I have personally received reports written in poor English, with barely comprehensible critiques that are frankly insulting and petty. One of the biggest obstacles in the peer-review process is to avoid obvious conflicts of peer-reviewers who are conducting the same research as the applicant. An objective critique is less likely in my mind when that conflict occurs. I am also highly suspicious of cronyism and cartel-like systems where misplaced favoritism is given to colleagues "friends", collaborators or past consultants of grants. There has to be unbiased objectivity.

The NIH peer-review system fails. I would like to see the following fairly basic principles put in place for peer-reviewers to qualify as expert in the field to review:

1) The peer-reviewer must have a proven ability to speak and write in English. Because they have been funded by the NIH does not qualify them to sit on a study section on this basis. There are many agencies out there that can stylize and edit a grant application of applicants who do not have competent command of the use of spoken and written English. It does the applicant no good at all to receive shoddy and frankly incomprehensible critiques.

2) The peer-reviewer should ideally not be personally known to the applicant. This is fairly simple to screen by answering a peer-reviewer conflict of interest questionnaire. The peer-reviewer may have personal scores to settle with an applicant, and this must be avoided.

3) The peer-reviewer must not have any competing, present or future pending funded research in the same area as the applicant. While the peer-reviewer should be an expert in the field, the reviewer has the power to crush or destroy an applicant's grant on purely selfish and unprofessional grounds. This is a very touchy and sensitive subject that is difficult to control or regulate. Multiple peer-reviewers go some way to avoiding the impact of this. However, if a reviewer wishes, the scoring of a grant can be distorted to the extent that it could be unscored or unfunded because of peer-reviewer biases and not necessarily because of any defects in the quality of the science or grant application per se.

4) There needs to be publicly available auditing of peer-reviewers. Criteria to be assessed should include the scoring habits of the reviewer, any conflicts of interest as described above, the competence of the reviewer in the use of written and spoken English and an assessment of the credentials of the peer-reviewer to qualify him or her to make judgments of the fundability of any grant application.

There are many other issues I would like to see investigated. For example, the success rates of MD qualified versus Ph.D. qualified candidates. I would be interested in determining whether there is any evidence for bias in the NIH grant awards system of those that are physician investigators as compared with those that are basic or translational Ph.D. scientists. I suspect the results of such an audit could be compelling.