When does oversight overstep?

When vascular biologist linkurl:John Cooke; of Stanford University received a grant in 2007 from the linkurl:California Institute for Regenerative Medicine; (CIRM) to launch stem cell research in his lab, he never expected the agency to linkurl:take back the money; -- especially not when his research was just starting to take him in some exciting new directions. Human embryonic stem ce

By | November 16, 2009

When vascular biologist linkurl:John Cooke; of Stanford University received a grant in 2007 from the linkurl:California Institute for Regenerative Medicine; (CIRM) to launch stem cell research in his lab, he never expected the agency to linkurl:take back the money; -- especially not when his research was just starting to take him in some exciting new directions.
Human embryonic stem cells
Image: Wikimedia commons,
Nissim Benvenisty
Within a year of starting the experiments on human embryonic stem cells (hESCs) he had outlined in his grant application, Cooke's group developed a protein-based strategy for generating induced pluripotent stem cells that they hope "will be a paradigm-shifting approach to vascular regeneration." The problem was that this was not what he had originally proposed to do. Cooke's award was a SEED grant -- Scientific Excellence through Exploration and Development -- CIRM's attempt to jump start research in hESCs, and by nature, fund exploratory basic research. The SEED grant Request for Applications (RFA) called for "new ideas and new carry out studies that may yield preliminary data or proof-of-principle results that could then be extended to full scale investigations." But after reading about this change of direction that Cooke clearly -- and proudly -- laid out in his first annual progress report, CIRM officials terminated the grant. "I anticipated that they would be happy with that [new] proposal," Cooke recalled. "[But] they weren't happy." In January 2009, after a second, more detailed progress report, follow up phone discussions, and a petition for reconsideration from Cooke, CIRM revoked his second year of funding -- nearly half of what he had originally been awarded -- citing the new directions his research had taken. "I can understand their reasoning," Cooke said. "I just wish I had understood that that applied to the SEED grants." Cooke wasn't the only one. The annual progress reports, required of all CIRM grantees, spurred the agency to revoke two other SEED grants of the 74 awarded: linkurl:Hari Reddi; of the University of California, Davis, and linkurl:Eric Verdin; of the J. David Gladstone Institutes in San Francisco linkurl:also suffered; the linkurl:same fate.; All three grants were pulled due to "inadequate progress," CIRM's former chief scientific officer linkurl:Marie Csete said; at a CIRM board meeting last June. (Both the grantees declined to comment, and CIRM officials refused to provide further details regarding the specific circumstances surrounding any of the revoked grants.) Clearly, funding agencies need to offer some oversight of grants to avoid misuse of funds. But can there be too much of a good thing? Agencies that fund basic research generally expect the unanticipated shifts that come with exploratory science. "Our belief is that some of the biggest discoveries happen by twists and turns and serendipity," said Robert Finkelstein, associate director for extramural research at the linkurl:National Institute of Neurological Disorders and Stroke; of the linkurl:National Institutes of Health; (NIH). Under an R01 grant for basic research, "it's ok if an investigator goes off on all kinds of tangents based on his interests. The PI is pretty much in the driver's seat in terms of the way the project goes." But the SEED grants walked a fine line -- on one hand, they undeniably targeted exploratory science, but on the other hand, their specific aim was to build up the field of human embryonic stem cell research. Arguably, that may have left Cooke's switch to pluripotent stem cells out of their purview. Interestingly, of the 14 Disease Team awards CIRM announced last month -- which support collaborations moving research to the clinic within four years -- only four focused on human embryonic stem cells. (Cooke ultimately got funding for his new work -- an initial $150,000 from the American Heart Association that kept his research going until he was awarded more than $11 million in two different NIH grants.) There's also CIRM's short timeline, which may force the agency to monitor awardees more closely and be harder on those who veer off-target. While the NIH will exist for many years to come, CIRM has a 10-year lifespan, as approved by California voters in 2004. "CIRM has very defined goals," said the Burnham Institute for Medical Research's linkurl:Huei-Sheng Vincent Chen,; another SEED grant recipient. "[They] wanted something within 10 years so they have to be more aggressive." In some ways, said linkurl:Keith Yamamoto,; a molecular biologist at the University of California, San Francisco, who cochaired the NIH's peer review working group last year, these grants sound more like a contract granting mechanism, in which there is a close working relationship between the funder and the investigator in pursuit of very specific goals. There's nothing wrong with such goal-driven research, he said, but it is "critical" for CIRM to make its intentions explicit from the get-go. Basic research grants awarded by both NIH and CIRM require annual progress reports that summarize the previous year's achievements and any changes made to the research program, but the level of scrutiny that each CIRM report appeared to receive was, to some, a bit unexpected. linkurl:Ellen Robey; of the University of California, Berkeley, also a 2007 SEED grant recipient, said she was surprised when she got a phone call from her project manager requesting more detail on her annual progress report. "I was used to the NIH system where the progress reports are really kind of a formality -- you get the sense that no one even looks at [them]," she said. The CIRM project manager was satisfied with the additional information Robey provided, and she was able to keep her grant. But, she admits, "I wasn't aware that they were revoking or taking away grants." As for the NIH, program managers thoroughly review each annual report submitted, but only a minority of the projects require additional follow-up, and this only "rarely" results in the termination of the grant, said Finkelstein. A spokesperson at the NIH Office of Extramural Research said the agency makes decisions about terminating grants on a case by case basis and does not keep track of the numbers of revoked grants. There are mixed reviews among the scientific community about whether CIRM's close watch of their grantees is a good thing. To some, it is an important practice for public funding agencies such as CIRM to show the tax payers that their money is going towards productive and fruitful research. "I think the oversight is outstanding," said John Simpson, the stem cell project director at the advocacy group linkurl:Consumer Watchdog; in California. "It shows that they're not asleep at the switch. CIRM is functioning as both a grant making agency [and] also something of a steward of the funds it hands out." But others say this kind of intense supervision can burden investigators -- and the science itself. "In theory, it's a terrific thing," agreed linkurl:David Kaplan; of Case Western Reserve University in Ohio, who has written about the peer review system at NIH. "To have the granting agency being involved enough to be helpful to their grantees, I think that is a terrific idea. The problem with that kind of a system is that you can be too intrusive. That eliminates that kind of serendipity [in scientific discovery]." In addition, such vigilance can imply a lack of trust that "is really important to the progress of science," said Yamamoto. "If the penalty [to not meeting specific goals] is to take [the money] away, the agency [is basically saying] 'We don't really trust you.'" CIRM is not alone in its struggle to find the balance between "thoughtful oversight" and "micromanagement," said linkurl:Larry Goldstein,; director of the stem cell research program at the University of California, San Diego, and a recipient of a number of CIRM grants. "I think any agency confronts this problem," he said. "You want to have enough oversight [to ensure] that people make progress during the term of the grant, but you also want to leave room for changes of direction that make sense given what you discover."
**__Related stories:__***linkurl:How to spend the NIH stimulus;
[11th February 2009]*linkurl:CIRM grants delayed;
[2nd February 2009]*linkurl: CIRM cuts ten grants;
[7th December 2007]


November 17, 2009

\nDear author,\n\nI agree with the **theoretical** arguments described in this article. I am not in California and my knowledge of the CIRM tale is limited to what has been reported at The Scientist and California Stem Cell Report.\n\nIt appears as if the issues are: \n\n1. Is oversight a fundamental requirement for competitive public science? Why is oversight so important?\n2. How I am going to oversight?\n3. What if oversight does not work? What are my resorts and resources?.\n\nIt sounds like I am preparing an R01. I will not send one more application because the chances are that I will be either triaged or deemed not competitive for funding at this time (blah, blah, blah). Instead, I am going to elaborate (contest) on the comments of my reviewers:\n\nMr X, from Consumer Watchdog, writes: It is critical to ** not asleep at the switch. CIRM is functioning as both a grant making agency [and] also something of a steward of the funds it hands out**. \n\nYes Consumer Watchdog. You?re absolutely right but if you don?t ** make your intentions explicit from the get-go**, your oversight cannot be **outstanding**. Neither innovative nor efficient but unfair. Because ambiguity does not help. It may well help ?MORE OF THE SAME? to keep going.\n\nDr A from Case Western Reserve University: **This proposal on Oversight is a terrific idea. The problem with that kind of a system is that you can be too intrusive. That eliminates that kind of serendipity [in scientific discovery].**.\n\nDr A, would you consider **too intrusive** the following case:\n\n1. Departmental and School space coordinators arrive one day to your lab saying that you?re moving to the space of your senior colleague PI, who is also moving to a much smaller space, but you will have your own bench. You find out that your official **mentor** is in that School Space Committee which, by the way, is an undisclosed arbitrarily membership selected committee that has not had the approval of the full department?. Would you consider too intrusive taking away your lab space without any prior discussion?. Would you consider the event coincidental if, in your previous Summary statement, one of the reviewers had questioned whether you really had your own lab space? (in your grant application it was clearly described your lab space and resources)\n\n2. While writing your grant application, to be sent in a week, your senior colleague PI comes and asks: **Listen did you withdraw your grant application?**. No, I did not, who said that?. **Go and ask Y, Division Chief. He?s been at the Departmental Business Office and said that, in the computer, it looks like you withdrew your application?.\n\nI did not ask Y because I knew, from years of experience trying to understand him, what his methods and approaches were about. He could have asked me directly but this is not his way. I simply called my SRA and asked for an explanation. Obviously, my SRA did not have a clue of what was going on, neither did I and I continued writing my application.\n\nDr A, I will not go on with examples. I consider these to be examples of intrusive and deliberate harassment. These are facts and a consequence of impaired oversight, promoted by the power and influence of multimillion NIH dollars that sustain high salaries of senior researcher administrators.\n\nDr B, University of California San Francisco writes: ** such vigilance can imply a lack of trust that "is really important to the progress of science," said Yamamoto. "If the penalty [to not meeting specific goals] is to take [the money] away, the agency [is basically saying] 'We don't really trust you.**. \n\nGreat comments Dr B, except that you have forgotten the premise in which your statement should be based, COMPETITIVENESS. The Peer review system, based on competition, awards public money to the best proposition(s) of scientific objectives and attainable results by researchers. How do we know that those objectives are being achieved over time?. \n\nIn the sports world, players are competitively judged and rewarded in the public arena and by their public performance. They are not penalized when they do not reach the finals; they are penalized when they break the rules of competition (cheat). In the scientific arena, that judgment is restricted to a very limited sector of the scientific community (reviewers of research reports and publications in any specific area). The public trusts scientific agencies (government or private) and their administrators to ensure that their money is productively used for the public good. They are not legislators but they have received a ?delegate? government function (from the public) that should transpire good judgment and a timely introduction of improvement/corrective measures when funding priorities and research objectives are not met. \n\nIt seems very clear and laudable to me that the three CIRM researchers have reacted like good citizens and outstanding scientists. They did a great job but did not reach the finals. They could have not been penalized because they had not cheated. They were requested to come back for next tournament. (That is my understanding on the assumption that the rules were clear: 10 years for the best job).\n\nDr L from University of California San Diego writes: ** "I think any agency confronts this problem," "You want to have enough oversight [to ensure] that people make progress during the term of the grant, but you also want to leave room for changes of direction that make sense given what you discover**.\n\nDr L, I totally agree and I would add that there are almost always contingencies that arise and could have not been anticipated. That?s how science appears to work most of the time. A good oversight system ought to provide ways to overcome those contingencies on the researcher?s provision of a rigorously defined plan. \n\nBut you know Dr L, you are the last reviewer and your vote does not count!.\n\nThank you.\n\nPS. \n\na) The events described here are part of a true story. Since I believe that perfection does not exist, I feel happy about my contributions to academic science. But, in no way, I desire to do science any longer as I had done in the past. I am raising my voice as an expression of a personal commitment to help prevent undesirable, yet avoidable experiences in academic science. I agree that termination should be last resort. Universities ought to hear that.\n\nb) The Scientist had deleted this message because reviewers are by definition pseudonyms and we cannot give away names when describing work experiences.\n
Avatar of: David Jensen

David Jensen

Posts: 7

November 17, 2009

A central question in this discussion is whether the expenditures by the NIH and the California stem cell agency generate genuine value for taxpayers. Of course, there is the question of what is meant by value. \n\nI think it is fair to say that that most taxpayers generally expect a concrete result that has real impact on their lives. However, I am not sure researchers agree that is the primary mission of either NIH or CIRM. More than one observer has commented about a sense of entitlement in California on the part of researchers. I suspect that many scientists also view the NIH as their creature. \n\nYou can read more about the affairs in California at the California Stem Cell Report -- \n\n
Avatar of: Anand Rajan KD

Anand Rajan KD

Posts: 4

November 17, 2009

Oversight: - the act of introducing management 'theory' into science.\n\nIf you, as an overseer, can provide oversight over a project - i.e., get a sense of where things are headed before they actually happen - why do the project at all?\n\n'Taxpayer money' is just an euphemism for people to poke their nose into what is not their business in the first place. If at all - like the previous poster noted - there can be financial administrative oversight. You can look out for any scientist buying a BMW with his grant money, but if he is doing experiments with the money you gave him, where is the problem?\n\nIn general, what is the meaning of scientific oversight? It seems, to be an oxymoron.\n\nIf taxpayers were actually made to sit down and listen to all the inconsequential data and minutae their 'taxpayer money' has produced, all research would come to a grinding halt. Remember the literal meaning of the word 'academic'?
Avatar of: Terry Iorns

Terry Iorns

Posts: 1

November 17, 2009

Part of the job of research scientists is to sell their ideas to granting agencies to obtain research grants. When you get the grant, what is the obligation to pursue the original intent of the application?\n\nWhen an agency comes up with a pot of gold, whether it be from taxpayers or other sources, the natural tendency is to chase it. However, in this case, the agency had a fundamental charter to advance human embryonic stem cell research. When there is such a charter, I think it is good policy to have considerable oversight.\n\nA fundamental question that should be asked, is, would the agency have funded the research if the new direction was actually the primary direction in the application? If the research goes off on a tangent that is of no interest to the agency's charter, it should be cancelled!\n\nConsider the example of a researcher applying for a grant to a pharmaceutical corporation to look into drug targets in a specific therapeutic area. If the researcher finds something interesting that may have application in polymer chemistry, should the drug company be expected to support that research. I say no.\n\nI agree with others in the areas of be able to pursue serendipity and whether proper terms of oversight were apparent to the researcher, but I don't think getting the grant is a blank check to do any kind of research.\n\nI see no difference in using the funds to pursue unrelated research and using the funds to buy a BMW.
Avatar of: Fred Schaufele

Fred Schaufele

Posts: 52

November 18, 2009

I agree with Terry. There is a difference between a research grant (emphasis on project supported) and a research fellowship (emphasis on individual supported). I am pleased that the NIH actually supports a hybrid of those two activities since it allows us to pursue the inevitable, more insteresting tangents that come out of our studies. One could argue that, for five year grants, such flexibility is prudent and that an agency would be wasting funds in demanding a strict adherence to the original goal.\n\nDespite the positive attributes of the NIH model, we should not expect that all agnecies adhere to that. If I were granted, or contracted, funds to pursue a specific line of research that was peer reviewed and deemed an appropriate way for the agency to spend those funds, then I must follow that line of work. That is even true for NIH funding. \n\nHow to remain true to the completion of the orginal goals of a thoroughly vetted project while maintaining innovation is one component of effective PI management. When faced with a new and exciting direction, the role of the PI is to find the ways to make that happen IN ADDITION to the original goals. The PI should be evaluating personnel and projects to ensure that all can be done within budget. Most of the time, the PI should be immediately thinking 'new grant proposal' as this represents not only a research opportunity but a funding opportunity. If the argument is that the project is not sufficiently developed for funding, then I think that answers the original question about why some agencies, like CIRM, take a dim view on jettisoning the original study Aims for the new direction.
Avatar of: David Jensen

David Jensen

Posts: 7

November 24, 2009

Given the difficult state of governmental finances, most responsible citizens expect concrete results from activities financed with their dollars. How the money is spent is certainly their business.\n\nIf researchers accept public money, they should expect to be measured publicly.
Avatar of: Dawne Shelton

Dawne Shelton

Posts: 1

January 20, 2010

With regards to the idea of taxpayers wanting some "value for their money", I'd like to point out that a grant is money to pursue a plausible research idea/hypothesis, usually by several different methods/aims. If the money is spent on further research which shows the original hypothesis/idea is false or improbable, then the remainder of the money should not be spent on a dead hypothesis. This is one of they key reasons that researchers shift their research. This is not waste. Shifting the direction of research is often a mechanism of following the positive data. It would be more wasteful to continue spending the grant money on a hypothesis that has proven faulty. And this is certainly nothing like spending the money on a BMW.

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Life Technologies