Advertisement

Is the NIH budget saturated?

Why hasn't more funding meant more publications?

By | September 11, 2007

Editor's Note: This piece from our November issue ran online ahead of print to spark discussion. What's your hypothesis for why the doubling of the NIH budget doesn't seem to have doubled productivity? Please post a comment by clicking here. Complaints by scientists about the flat NIH budget have grown louder in recent years. For scientists to effectively lobby Congress for increased funding, however, we need to show that increased funding increases productivity. Given this need, I decided to examine scientific productivity as a function of the budget. Since the NIH budget doubled from $15 billion to $26.4 billion from 1999 to 2003 (Figure 1), I reasoned that there should have been a corresponding jump in productivity. The test was the simplest measure of productivity: the number of publications. (continued below)
Here's what I found: The number of biomedical publications from US labs did in fact increase from 1999-2004. However, so did the number of publications from labs outside the US where the research budget did not double. Figure 2 shows a parametric plot of the number of scientific papers indexed in the ISI Web of Science database by the keyword "biology" that were published each year from US labs and non-US labs. There is no upward jump that you would expect to see with a sudden increase in productivity. (continued below)
Perhaps the lack of correlation between funding and the publication rate was an artifact of choosing the wrong keyword. So I repeated the search, this time using the keyword "neuron" instead of "biology." Again, as shown in Figure 3, increasing funding had no effect on the publication rate. The key words "RNA," "DNA," and "disease" produced similar results. (continued below)
The lack of correlation between the publication rate and the budget is further supported by the fact that after 2003, when the budget flattened, the publication rate did not decrease. The analysis is not insensitive to changing publication patterns; during the same period of time there was an enormous increase in publications originating from China. For example, using the keyword "neuron," from 1996 to 1999 publications from US labs increased 2.9% annually, while those from China increased 42.7%. From 1999 to 2005 publications from US labs increased 3.6% annually, while those from China increased 38.9%. The trend to publish more articles seems to reflect the activity of journal publishers rather than scientists. The number of indexed journals increased from 4500 to 6500 between 1994 and 2006. Still, I reasoned, perhaps publication rate is not a good index of productivity, and the papers are twice as good as they were in 1996. That is hard to judge, but analysis of the literature suggest otherwise (Ref 1, Ref 2). An NIH institute director suggested that the absence of correlation may reflect a significant latent period between funding and stimulated productivity. Given this data, however, that latency would have to be greater than ten years, and we are talking about a net excess influx of more than $50 billion since 1996. Presumably, the NIH does not expect a ten-year latency between the granting of funds and the delivery of new data, since they award grants for only 3-5 years. The best data on US research quality should be available from the NIH, where review panels judge the scientific quality of grant applications more than 50,000 times a year. The panel's grades of research quality are averaged to a precision of at least three significant figures; a person may be denied funding if their grant quality is judged to be 0.1% below the current funding levels. The ability of a review panel to decide whether a person's ideas are not worth a grant signifies that they are a good judge of quality. I encourage the NIH to analyze the available data and see if current research is twice as good as older research. The lack of correlation between funding and publication rate does not mean the NIH is funding bad science. It isn't. It is funding some great science, but it is not spending its budget efficiently. According to the director of NIH, doubling of the NIH budget led to a paradoxical decrease in the grant funding rate (Ref. 3). What happened to all the extra money that flowed into the NIH? Was it used for unproductive clinical trials (Ref. 4, Ref. 5)? Was it absorbed by inflation? Wherever the funds went, they left no clear scientific record. Frederick Sachs is a professor of physiology and biophysics at the University of Buffalo. References: 1. E.N. Brown and S. Ramaswamy, "Quality of protein crystal structures," Acta Crystallogr. D. Biol. Crystallogr, 63:941-950, 2007. http://www.the-scientist.com/pubmed/17704562 2. J.P. Ioannidis, "Why most published research findings are false," PLoS Medicine, 2:696-701, 2005. http://www.the-scientist.com/pubmed/16060722 3. E.A. Zerhouni, "Research funding - NIH in the post-doubling era: Realities and strategies," Science, 314:1088-1090, 2006. http://www.the-scientist.com/pubmed/17110557 4. A.R. Marks, "Rescuing the NIH before it is too late," J Clin Invest, 116:844, 2006. http://www.the-scientist.com/pubmed/16585950 5. A.R. Marks, "Rescuing the NIH: the response," J Clin Invest, 116:1460-1461, 2006. http://www.the-scientist.com/pubmed/16585950
Advertisement

Comments

Avatar of: Andras G. Lacko

Andras G. Lacko

Posts: 2

September 11, 2007

The roller coaster nature of funding for medical research is not new. The first major upheaval occurred in 1968 when the Nixon administration reduced the appropriations for the NIH. Subsequently, during the Reagan years there was an attempt to sequester a substantial portion of the budget, earmarked earlier for NIH research funding. These reductions were usually followed by upswings, making long term planning and sustained development of research programs progressively difficult. The latest euphoria about increased availability of NIH funds at the turn of the 21st century has now been followed by a dismal, period threatening the careers of particularly talented young investigators.\n\nThe question is what to do? Clearly, the answers are not easy to come by. however, here are a few ideas.\n\n1) Demonstrate good will toward Congress by decreasing reliance on Federal funds. This may be accomplished by gradually scaling back indirect costs and investigator salaries so that the funds within the budget could be directly channeled to projects at the bench and at the bedside.\n\n2) Based on these principles, attempt to communicate with Congressional Representatives to protect the funding for extramural NIH support by insulating it from other appropriations and convince Representatives to develop a long range financing strategy for NIH.\n\nIt should be easy to argue as to how the roller coaster nature of the funding damages the infrastructure as well as the talent base for biomedical research. Hopefully, common sense would prevail and solve this age old problem. There is clearly enough money in the system to support sustained growth of biomedical research. The task is to apply a seemingly simple rationale to execute this task.
Avatar of: Larry Mulcahy

Larry Mulcahy

Posts: 1

September 11, 2007

If the rate of papers is not increasing with the budget increase, then the cost per paper must be increasing. It therefore seems logical to assume that access to increasingly expensive technologies is an important part of getting papers published. Microarray data, proteomics, and other expensive techniques are being used more routinely. That has to play some role in this. \n\nBut I also don't think using publication rate as a metric of productivity is the best way to go.
Avatar of: Anil Wali

Anil Wali

Posts: 3

September 11, 2007

I agree with the analysis to some degree but we have to take into account the quality of the publications and not the quantity. There are several other factors that NIH study sections have to take into consideration to improve scientific productivity threshold levels. First, are we funding renewal applications of PIs with third tier list of publications. Second, has the previous funding cycle produced important new strides and moved the field forward i.e., has the data generated any clinical/translational relevance. Third, we emphasize lot on novelty and innovation at the begining of the careers in assessing scientific merits of the applications but become complacent when the same scientifc endeavor comes up for renewal.\nWe all have to ponder on this issue and give preference to some out of the box solutions rather than following the conventional idealogy in funding scientific proposals.\n"No pains no gains"

September 11, 2007

Much of the increased support has been misdirected to two clsses of inadequately reviewed, inadequately monitored, consistently underperforming, projects: i.e., (1) multi-investigator center and program projects and (2) NIH-initiated, NIH-directed projects.\n\nThe most egregious example is the biodefense research program, in which NIH has misdirected more than $1 B each year from FY2003 onward to work of marginal scientific and public-health significance. Much of this support has gone to mediocre and sub-mediocre investigators (investigators whose proposals routinely had been triaged prior to FY2003, but who whose proposals could be recast as biodefense and reviewed by bidefense special emphasis panels with relaxed standards after FY2003). Much of this support has gone to mediocre and sub-mediocre multiinvestigator centers and programs, including multiple redundant university-based regional biocontainment laboratories with initial construction costs of >$10 M each and multiple redundant national biocontainment laboratories. The faillure of these expenditures to have significant impact on science or on public health should be unsurprising.\n\nAlthough the biodefense research program is the most egregious example, there unfortunately are numerous other examples. Common themes exhibited by the failed and failing programs of the last ten years are as follows:\n(1) Support was provided to multiinvestigator centers and programs, as opposed to individual- investigator programs;\n(2) Support was provided through NIH-initiated, NIH-directed mechanisms, such as RFAs, RFPs, and the NIH Roadmap, rather than through investigator-initiated mechanisms.\n(3) Support was provided following non-standard, "special," review processes (review by special emphasis panels and special study sections that consider only multiinvestigator center and program proposals or that consider only NIH-initiated proposals, setting aside these proposals and ranking them only against only each other, rather than ranking them against all proposals).\n(4) Support was provided in large blocks (>$1 M US direct costs per year).\n\nIt is time for the NIH to scrap these failed and failing approaches, and to return to a focus on individual-investigator programs, investigator-initiated programs, rigorous competitive review, and appropriate budgets.\n\n
Avatar of: Rickye Heffner

Rickye Heffner

Posts: 1

September 11, 2007

It is likely that more elaborate regulations covering animal care and protection of human subjects contribute to the reduced productivity. For minimal (if any) actual benefit to animals or humans the cost in time and money to gain approval for research with animals or humans and to carry out that research have increased greatly over the time span in question. \n\nA second factor is that increased resources attracts increased consumers. These added consumers of NIH funds are largely drawn from the pool of people who would not have applied, or perhaps not even pursued a career in science, had additional funds not attracted them.
Avatar of: Dr. Raam, Shanthi

Dr. Raam, Shanthi

Posts: 43

September 11, 2007

We all have to agree that in our country USA during the past two decades tremendous strides have been made in technology. As a cancer researcher, I can speak for strides made in diagnostic techniques, tumor marker identification, therapeutic advances, and identification of environmental carcinogens.NCI has used its money wisely, even initiated long-term project support ( SPORE-10 years)for selected most prevalent cancers. Productivity measurement, especially for clinical subjects and biomedical engineering should not be restricted to publications. Most of the new drug developments, new diagnostic assay developments are patented first before publication. Some patented materials are never published. Including number of patents ensued from NIH funded projects may be a more balanced and a better approach. Another point to consider: Non-US scientists in many countries do not often have combined teaching and research responsibilities. Many of the productive scientists are just reseachers based in Universities or research institutes. This may make a lot of difference in productivity in terms of publications. How about collaborations between several NIH funded scientists resulting in a single publication? This is the reality of the day. Multi-disciplinary collaborations are becoming essential part of scientific productivity. All these factors would influence publications per funded investigator per annum.
Avatar of: Withheld

Withheld

Posts: 199

September 11, 2007

Careful examination of US and European sponsored research versus non will show a large volume of papers published that have nothing to say. For instance, biostatistical methods that nobody would ever want to use, which are worse than undergraduate exam question answers, microarray analysis results that are utterly meaningless, and on and on. I have found students that have published papers based on theses which they cribbed from patent office applications. (I satisfied myself of that. Interestingly, every single one was a foriegn student from what we call now the "developing world", much as I hate to say it.) Advanced degrees are churned out of India, Russia, Egypt, and other nations, and for a significant percentage, those PhD holders are less qualified than your average bachelor's student in the USA, Germany or Switzerland. In such nations, the wealthy/privileged get through regardless of merit. (Some of them have even gotten plum positions in the USA. I am thinking of the chair of at a US University whom I have satisfied myself is far less knowledgable than the students reporting to that chair. But scheming political skills honed in the developing world are hard to beat, since they have no boundaries they respect.) Advancement by application of money has occurred a few times in the USA also, and at some fancy universities. It is not unknown for rich foriegn students to have a cadre of people paid to ensure they get through with good grades. And "donations" can go far. But this stuff becomes farcical in India, Russia, or certain other nations. Certainly, there are highly qualified people coming out of such systems, but there are also quite a number that are not in the least. And yet, mass drivel is getting published, and in large volume. \n\nSo the real question is what the quality is of the papers in the "other" column. I doubt very much that most are anything but disinformation of the most obnoxious sort.
Avatar of: Ellen Hunt

Ellen Hunt

Posts: 199

September 11, 2007

Particularly in the biosciences, it is hard to do anything now without use of quite expensive equipment such as flow cytometers. Complicating this is the fact that much of this equipment is sold to people with little understanding of it, with much hocus-pocus. \n\nThis leads to all sorts of logistical problems when, for instance, on flow cytometer gets results that another one doesn't get. What, then, does it mean? Many labs just go with the one they like. But delving into the physics of the instruments, one realizes that these extremely expensive machines are made with cheap components that have different characteristics from machine to machine, and all sorts of problems. \n\nSo, when a lab tries to duplicate results and they can't, what does that mean? Something? Or nothing? \n\nThe list goes on and on. That's one reason that today's science is more expensive. Everything is more expensive, and fewer and fewer of us understand even whether what we are being sold as kits is actually doing what it claims.
Avatar of: Josh

Josh

Posts: 1

September 11, 2007

Due to the belated realization that Postdoc positions in biology lasted more than one or two years, the NIH raised its guidelines for compensation during this period. Wage increases, therefore, may have played a role in the increase of the dollars per paper ratio. This is not to say that Postdocs are getting rich. On the contrary, Postdocs were grossly underpaid to begin with.

September 11, 2007

Although the total number of papers can be a good indicator for the productivity, it is not appropriate for the total population. In total, the number of scientific publications has more to do with the number of journals available than anything else. To get a more fair assessment, one should also take into account of the rejection rate. Does the overall rejection rate grow faster than linear? Another factor to consider is the number of conference abstracts/papers. What one really should measure is are the number of new breakthroughs, but this measure is very hard to evaluate.
Avatar of: Martin Zane, MD

Martin Zane, MD

Posts: 2

September 11, 2007

It is necessary to ask if a budget increase corresponds to an increase in investigators or to an increase in expenditures by a fixed number of investigators. Moreover, it is appropriate to question whether an increase in funding does correlate to an increase in oversight. Another question to consider is the possibility of increased control over the nature of the investigations.\nThank you for you continued monitoring of the investigative world.\n
Avatar of: Michael Morris

Michael Morris

Posts: 1

September 11, 2007

The author expects that a single parameter, the number of publications per year, should map very closely to the increase in funding? And that there should be an obvious relationship between these two things with the annual levels of funding and publication in other countries? \n\nIf his simplistic expectations are not met, he concludes that there must be funding inefficiencies. Well, I think he is drawing a VERY long bow. The logic is too simple, even child-like. I'm glad that the article has at least stimulated some thoughtful responses from readers.
Avatar of: Andrew Currie

Andrew Currie

Posts: 1

September 11, 2007

Surely in the last decade we have seen an increase in the number of publications working on large multi-national projects , with extensive author lists. If the number of published NIH-funded authors (on all publications) was plotted over the same period, would this be seen to increase with the rising budget?. The rise in quality means that larger projects are tackled, with more authors, and hence fewer publication.
Avatar of: Keith Cheng

Keith Cheng

Posts: 1

September 12, 2007

Some additional considerations deserve mention: \n\n1) There is more going into top publication now than before (supplementary data). Numerical measures are not enough. \n\n2) Significant time is being wasted on rewriting grant applications that arguably should be funded. It is a common impression that, on average, the top 15-20% of applications are sufficiently meritorious to deserve funding - when funding levels diminish below that level, few substantive suggestions are made, and system efficiency diminishes - valuable time that could be spent on research is spent instead rewriting applications. \n\nTo amplify on others' comments:\n\nClearly, a stable mechanism of funding - a growing endowment, is needed. Intelligently-designed investment of a small fraction of the monies spent on the war in Iraq on an endowment would maintain a growing NIH budget for the foreseeable future.\n\nIt would seem wise to diminish the fraction of politically manipulable funding, since it is absolutely clear that the US has built its strength of research on the merits of the individual investigator. \n\nAs far as the plagiarism issue, anyone with such history should, in my opinion, be barred from participation in science. It is otherwise unfair to society and to those who play by the rules. Scientists have a responsibility to maintain and earn society's trust, since they are, after all, spending taxpayers' money for society's benefit.\n\nWe should speak to such issues with solidarity, so that positive change can be made.\n
Avatar of: Frederick Sachs

Frederick Sachs

Posts: 1

September 12, 2007

Dr. Shanthi Raam suggested that much of the research money might have been spent on "secret" research applied to patents. In fact, the number of patents in biology decreased strongly over the period the NIH budget increased. I can't plot it, but here is the data for 1992-2006 for the number of biology related patents issued by the US. Apparently the confidential patent related research was either unproductive or non existent and perhaps discouraged by the increase in NIH funding.\n\n2006 45\n2005 54\n2004 75\n2003 95\n2002 113\n2001 136\n2000 149\n1999 168\n1998 183\n1997 191\n1996 199\n1995 208\n1994 212\n1993 217\n1992 219\n

September 12, 2007

[The following is a clarification of my previously posted comments.]\n\nMuch of the increased support has been misdirected to two clsses of inadequately reviewed, inadequately monitored, consistently underperforming, projects: i.e., (1) multi-investigator center and program projects and (2) NIH-initiated, NIH-directed projects. \n\nThe most egregious example is the bioweapons research program, in which NIH has misdirected more than $1 B (~5% of the NIH budget) each year from FY2003 onward to work of marginal scientific and public-health significance. Much of this support has gone to mediocre and sub-mediocre investigators (investigators whose proposals routinely had been triaged prior to FY2003, but whose proposals could be recast as bioweapons research and reviewed by bioweapons-reasearch special emphasis panels with relaxed standards after FY2003). Much of this support has gone to mediocre and sub-mediocre multiinvestigator centers and programs, including multiple redundant university-based regional biocontainment laboratories with initial construction costs of >$10 M each and multiple redundant national biocontainment laboratories with initial construction costs of >$100 M each. The faillure of these expenditures to have significant impact on science or on public health should be unsurprising. \n\nAlthough the bioweapons research program is the most egregious example, there unfortunately are numerous other examples. Common themes exhibited by the failed and failing programs of the last ten years are as follows: \n(1) Support was provided to multiinvestigator centers and programs, as opposed to individual- investigator programs; \n(2) Support was provided through NIH-initiated, NIH-directed mechanisms, such as RFAs, RFPs, and the NIH Roadmap, rather than through investigator-initiated mechanisms. \n(3) Support was provided following non-standard, "special," review processes (review by special emphasis panels and special study sections that consider only multiinvestigator center and program proposals or that consider only NIH-initiated proposals, setting aside these proposals and ranking them only against only each other, rather than ranking them against all proposals). \n(4) Support was provided in large blocks (>$1 M US direct costs per year). \n\nIt is time for the NIH to scrap these failed and failing approaches, and to return to a focus on individual-investigator programs, investigator-initiated programs, rigorous competitive review, and appropriate budgets. \n
Avatar of: Harrissi

Harrissi

Posts: 1

September 12, 2007

A federal research agency such as NIH requires all data to be made public and available (there is nothing called secret science that need not be reported in a usual NIH grant). What other metric Morris and some other readers are proposing ? secret, unreported science? So increased funding should have translated into more papers/patents. Unless of course the increased funding solely went to increase the grant size of existing grantees. That did not happen. Another possibility is that the increased budget went to fund largely the same pool researchers but in the form of multiple grants. In this case a PI with multiple grants is responsible for productivity of each of his/her grant, and not at the expense of productivity. This story explored several metrics and it seems to me that publications/patents is the most reasonable one. If the increased funding went to increase the pool of PIs one would have surely see a quantum jump in papers/patents. If a large pool of money went to existing PIs then data simply proves that PIs are human and they cannot be three, four or n-times more productive ? law of diminishing returns. A small additional note, using cute words like ?child-like? in Morris' story simply lower the quality of debate on this important topic and suggest they be left alone. We are all adults here and can surely read divergent viewpoints without feeling like reading a sensational magazine.

September 12, 2007

Dear Professor Sachs:\n\nI read with interest your note on "Why hasn't more funding meant more publications?" and I wish to add a piece of data from Mexico in terms of number of publications vs budget. It turns out that from 1997 to 2006, the Mexican number of Scientific articles increased by 84.7 % with a decrease that runs from 12% to 21.5 % in the country's budget for Science. The range of decrease depends on who analyzes the government's investments in Science.\n\nI send you this note, because if you are intrigued by the correlation between publication and budget in the United States, I am more intrigued by our correlation between scientific productivity and budget in Mexico.\n
Avatar of: MW

MW

Posts: 1

September 13, 2007

I have observed first hand many of the situations raised by individuals in this line of discussion that seem to have arisen as the result of the doubling of the NIH budget.\n\n1) Taking advantage of the sudden windfall, established investigators have applied for and won multiple RO1s, with little or no increase in publication rate. It is easy to avoid the risk of the progress report these days, all you need do is submit grants with new titles. Some of this will obviously be self correcting in the current climate.\n\n2) My institution has taken part in the biodefense histeria, and with dollars in hand built their expensive version of the field of dreams, only in this case, nobody came. It is amazing how NIH can award these dollars without ever asking whether there are sufficient investigators or plans to hire investigators that will use the facilities. I expect in a decade these new facilities will be redesigned or collect dust as a monument to the great histeria of 9/11\n\n3) We certainly have our share of large >$10 million multi-investigator, let's spread science into places where it will disappear as soon as the NIH funding is gone. University administrators love these things, one grant is worth more in IDCs than ten individual researchers combined and for that reason you are a heretic if you oppose these initiatives. However, it is unclear to me whether NIH will ever honestly assess these investments. If they did a fair accounting, I believe they would reach the conclusion that there wasn't one decent chance of success from the get go. \n\nThis is the message that I will leave. Telling anecdotal stories as we all have will see little impact. What is critically needed if scientists are to capture the attention of policy makers and solve the NIH funding inefficiencies is a carefull system wide quantitative analysis of the impact of how NIH funds are distributed. Are there real diminishing returns from having too many grants? What is the cost to basic science from the shift of funds to small business in the form of SBRIs? Are special funding initiatives to selected states wise investments, do they accomplish their intended goals, and how does this benefit overall US health? Has there been too much power shifted from the study sections into the hands of program officials-with the corollary, is there a difference between the quality of science from research funded by established study sections versus special emphasis panels?

September 13, 2007

My guess is that the increased budget allowed and promoted a widened scope. For example, studying expression of one gene may not be interesting enough, one must study the expression of 20000 genes to place it into perspective. The higher ambition level is more costly but also more scientifically healthy. In this respect, NIH and the US are leading global biomedical science towards more definite and relevant knowledge.
Avatar of: Kevin

Kevin

Posts: 1

September 13, 2007

When the NIH budget doubled there was a doubling of the number of postdocs but very little increase in faculty. Alluded to here, Nature 422, 354-355 (20 March 2003). If you hire twice as many postdocs then you need to go deeper into the talent pool. Since all postdocs are paid the same, no matter their ability, you get alot of waste of money to support poor quality postdocs. Ironically from personal experience bad postdocs not only waste money, but they also waste time. Better scientists, both graduate students and postdocs, end up hand holding the bad postdocs thereby preventing them from getting their own work done. Additionally as the labs explode in size the faculty are more ineffective in managing the lab. Also since NIH funds caused an increase in lab size (more postdocs) instead of the number of labs (more faculty) you prevent diversity in the research enterprise since postdocs have to hew reasonably closely to their mentors interests. Independent scientists on the other hand need to distinguish themselves from their previous mentor and so explore different avenues of study. Also traditionally older faculty tend to increase depth in a field while younger scientists tend to establish new fields. I believe establishing more young independent scientist instead of quartering them as postdocs reduces scientific redundancy. For example it is less likely to have the problem of papers being scooped (big waste of money) because two 55 year old faculty members with 5 ROI's and 30 postdoc labs, that have competed in the same field of study for the last 25 years are stepping on each others toes. Lop off one ROI, lop off ten bad postdocs, redistribute the money to one outstanding postdoc turned into faculty per 55-year old run mega lab and there might be more bang for the buck.
Avatar of: Sam

Sam

Posts: 34

September 14, 2007

Anyone who actually applied for federal funding knows the term "percent of effort". If the budget is tight, less effort and money. So the extra NIH funding will be obsorded by the effort increases, more salary, without changing a thing.
Avatar of: Pavel Goberman

Pavel Goberman

Posts: 1

September 16, 2007

The former USSR was destroyed because most people tryed to get from government than to give. \n Here is worse: so many government employees have huge salaries many years after years, and have no any responsibility, no pay back to taxpayers / society. It is worse than Communism system, where people are trying to give back more as they could.\n Our nation is in big debt, but the NIH is trying to milk taxpayers as more as could. The NIH is funding many so named "research" organizations many years, even these businesses have no any result, no any productivity. It is a common sense: stop waste taxpayers money. But: who cares? Many researches are donating corrupted politicians, and ... they are geting more funding.\n Blame US government, blame President Bush for wasting taxpayers money. Stop Communism System!\n\nPavel Goberman - Candidate for US Senator, OR\nwww.getenergized.com/vite.html
Avatar of: John Kounios

John Kounios

Posts: 1

September 18, 2007

One factor that could explain, at least partly, why publications have not kept pace with total funding is that, under increasing pressure, researchers have been spending more of their time writing grant proposals and less time writing journal articles. Also, as the best journals have become more competitive, increasingly, articles have to be submitted serially to multiple journals, slowing down the publication process. And finally, as research pushes technology to its limits, larger and larger research teams have to use increasingly expensive equipment in order to learn new things. What all this means is that research funding will have to increase further in order to yield an increase in journal articles.
Avatar of: DANIEL JONES

DANIEL JONES

Posts: 1

September 18, 2007

I would like to see the US publication data presented here confirmed by scientometrists at Thomson Science, the producers of Web of Science. The number of biomedical publications originating in US labs on an annual basis is not so simple a search of WoS as textwording for biology, RNA, DNA, OR disease.\n\nConfirming the data should be the first step before drawing conclusions that NIH "is not spending its budget efficiently."
Avatar of: Steven

Steven

Posts: 1

September 18, 2007

As some other posters have suggested, I think that role of journal space may be playing a limiting factor in publications. In order for the number of articles to increase there has to be some place for them to be published. I assume that there is only limited flexibility in the number of articles published per issue in most journals. This would mean that a substantial increase in the total number of publications requires an increase in the number of journals. \n\nI expect that the market forces driving the number of journals in a field are much more complex than just the productivity of that field. The comment from one of the previous posters that publication in existing journals is getting more competitive also suggests that available publication space may be the limiting factor here. An analysis of submissions rates during the same time period might confirm that productivity has increased the number of manuscripts being written. If this is the case it might be the journal system that is saturated.
Avatar of: Dr.Raam, Shanthi

Dr.Raam, Shanthi

Posts: 43

September 18, 2007

Dr.Sachs seems to have misinterpreted my comment. I never directly or indirectly implied that NIH is funding "Secret Science". The focus of my comment is inclusion of both publications and patents to assess the productivity issue of NIH funding. NIH does not fund patents! It funds variety of projects, a few of which result in patentable product or method. Every publication be it an abstract, full length article or patent (applied for, or issued) contains statements of acknowledgement for funding support, with details of grant number and funding agency. Issued patents are published in the USPTO web site for everyone to read(not a secret science). I agree with some of the comments made regarding the fallacy of selecting the words biology, DNA or RNA to scan the publications.Same is true for scanning patents. When I scanned my own past publications (seven of them, easily accessible to me)in Journals Endocrinolgy, Clinical Chemistry, Journal of The National Cancer Institute,Cancer research none contained these three words. All acknowledged grant support from NCI or other agencies. Therefore, a better approachit seems to me, is to scan the acknowledgement section of the publications where funding agency details are stated. NIH may already have the data on publications resulting from their grant suppport every year; patents resulting from investigations which were supported by the agancy; so would the USPTO. Will they not supply that information (which is public information) to The Scientist?

September 19, 2007

The problem with the analysis that this article is based on is that it uses, essentially, a ratio to make a general comment about a multifactorial trend.\n\nThe analysis compares US and Non-US publications. The assumption is that because non-US publications ALSO grew over time, ostensibly without the benefit of increased funding, ignores a number of very important factors. (1) Non-US investigators apply for, and win, NIH dollars. (2) The EU countries have in fact make substantial increases in funding dollars in the areas of genomics and proteomics; (3) The rest of the world (non-US) includes many countries whose scientific research community matured and grew over the time period.\n\nThe ratio approach does not work here for the same reasons that fold-change is an unreliable measure of differential expression in microarray studies.\n\nJames Lyons-Weiler\nDirector\nBioinformatics Analysis Core\nUniversity of Pittsburgh\nhttp://bioinformatics.pitt.edu/\n
Avatar of: F.M. Johnson

F.M. Johnson

Posts: 1

September 20, 2007

Why hasn't more funding resulted in more publications? Hypothesis: Because the increased funds have been spent on proliferating bureaucratic functions such as ?communications,? ?translational research,? ?liaison,? ?oversight,? ?education and awareness? and ?outreach,? that have little or nothing to do with scientific productivity. Each function will typically have a well paid director, often with an associate or deputy director along with a staff of workers, travel and supply budget and occupying valuable space, all of which combine to take away from scientific productivity.
Avatar of: Mark Boothby

Mark Boothby

Posts: 1

October 4, 2007

This article, at least as written and presented, is an impressively pernicious example of non-rigorous, poorly analysed data with dubious analytic approaches being passed off as findings.\n\nThe topic is an important one (US Congress members explicitly ask, "What have we gotten from the doubling?"). Unfortunately, even though the basis for an interpretation is not supported by the evidence in the piece, the flaws are so numerous as to need an entire separate article. \n\nA few particularly egregious flaws:\n1. rigorous and accurate measurement of 'productivity is extremely hard, but the metric used herein does not try very much to get even close to something 'right'. {journals during the same time period have become vastly more demanding in terms of the amounts of data needed to achieve acceptance. Moreover, data that previously could be 'recycled' now more and more gets used as online supplemental material. Finally, during a more optimistic time, groups may hold out more for 'Big splash' articles of greater impact (with far more data) rather than slicing and dicing. Personal view is that number of figure panels of primary data is a far better metric to start to get at 'productivity' than 'number of articles'.\n\n2. The process of comparing to non-US sources renders the data meaningless, as does the failure to take into account any changes in the aggregate number of awards from non-NIH sources. (If, with the rapidly expanding investment of the PRC into biomedical science and their push to publish in international literature, there was a parallel increase in the number of articles from non-US sources, the graphs in this article would be the expected result.\n\n3. The point about not (yet) seeing a falloff right at the beginning of the slowdown is a bit naive, too, since the appearance of an article commonly lags generation of the bulk of its data by over a year.\n\nIncidentally, a separate analysis previously released in recent months showed that articles being published by US researchers ARE flattening / starting to taper off as the ongoing crisis created by NIH and governmental mismanagement stretch to more years.\n\n4. Finally, perhaps the author is immune to inflation but most of biomedical science is not so over the 5 year term they chose to show, the 'cost of doing our business' went up some 25%\n\nDon't get me wrong - I think it is entirely possible that because the NIH chose during the double to throw a lot of money at inefficient "Big Science" and mega projects (SPORES, SCORs, PPGs) that I would bet are inherently less productive (per $1,000,000) - the amounts of publication from the doubling (adjusted for inflation - which makes it more like the 50% increase in real $$$ over 10 years) may not have increased as much. But non-rigorous pseudo-analyses are not going to bring much clarity to the issues.

October 5, 2007

I thought that Dr.Sachs was asking this rhetorical question as a way of determining whether the NIH was spending its budget effectively. In this regard, the number of publications is a short-term proxy for productivity. But the bigger picture question is whether we as investigators are moving the goal posts forward? We are in a Kuhnian crisis, because the paradigm shift forced by the publication of the Human Genome shifted biomedical research from descriptive to mechanistic. Unfortunately the reductionist molecularists think that by describing a mechanism they have determined how genes translate into phenotypes, and visa versa. We desperately need to generate an algorithm that will facilitate that effort so we can take advantage of the tools we have been given. I believe that this is the core problem, and that publications, patents, peer-review are all symptoms.

November 9, 2007

Cut lines for funding of NIH grant applications have been reduced to about 10-12% of scored applications for many institutes. Given the 50% rejection outright (no score) an application has to rank within the top 5-6% of submissions to be funded. Full-time researchers cannot fund themselves entirely on one grant, and often participation in at least 3 grants is required. One can readily estimate that the number of applications (including revisions for resubmission) has to be sustained at 1 per month to garner full support under 5-year grants. Combined with a typical non-negotiable budget cut of 20-30% for every award (yes, after finally getting funding approval!), it is no suprise that NIH-sponsored research is terribly inefficient at these funding levels, given the number of applicants. Either more money, or fewer researchers, or a combination of the two is necessary to stabilize this unsustainble situation. It is a wonder that paper count is as high as it is, and quality has certainly suffered, in my opinion.

November 19, 2007

I'd like to add something to Dr. James Lyons-Weiler's note ("Flawed Analysis"). The NIH often funds large projects whose findings and results are publicly available to all. A case in point; during the budget doubling period, the NIH funded the human genome project. Surely this project has contributed to the increase in research papers by non-US labs, especially in the field of bioinformatics. \n\nAs said above, assuming that number of publications is a useful measure of productivity is naive. Perhaps an analysis of numbers of citations would show the difference in research quality a bit better. \n\nThe sad thing is that the flawed analysis in this article will probably be used to justify future flat NIH budgets... you'd expect a more scientific analysis from "The Scientist"

November 30, 2007

Contemporary medical research in the world is based on the clinical and translational research. The translational research is a paradigm, it is based on the interdisciplinary collaboration clinical investigators, basic scientists and statisticians to facilitate the testing a novel therapeutic strategies. Repeteadly I have pointed out methodological problems when using this paradigm in medical research. Several times I have written about this problem not only to different journals (at first to JAMA in 2002 without any publication success ), but I also informed the EU, WHO and many US senators in a written form without any effect or even an answer.\n\nThe translational research versus the basic research.\nOn the necessity of the basic research in medicine.\nJaroslav Smidl\nI think, nobody will doubt, that the classification of science according to the movement of mass into:\na)physical level of the movement of mass\nb)chemical level of the movement of mass\nc)biological level of the movement of mass\nd)psychosocial level of the movement of mass\nare correct.\nIf we want to classify medicine according to the above mentioned levels of the movement of mass, we have to state that this classification is impossible! Medicine has always had an integral character, i.e. all levels of the movement of mass are a branch of medicine.\nThis integral character of medicine has been well known from the times of Hippocrates, who required the following study process: at first, the student of medicine has to know all fundamental sciences on all levels of the movement of mass and later on, the student of medicine is able to study medical disciplines, as the highest knowledge necessary for the medical practice.\nThe integral character of medicine gives rise to fundamental questions in medical research:\nWhat is the basic research and what is not the basic research in medicine?\nAnd another question:\nIs it necessary to speak about the basic research in medicine since the basic research undergoes on all the above mentioned levels of the movement of mass?\nThe scientific research, generally speaking, falls into two parts: the basic research and the applied research, but this dividing of research is artificial. The basic and applied research are two sides of one coin. The dividing of the research into the basic research and applied research is based on:\n1)the different aims of the research,\n2)the different methodological approach to the solution to the problems\nduring the research,\n3)the different economic utilization of the results of the research.\nThe aim of the basic research is to reveal the fundamental and unchangeable causes of phenomena. From the methodological point of view, the information about the fundamental and unchangeable causes of phenomena are processed by inductive judgements concerning the fundamental laws of the theory i.e. to the principles. Therefore the principles can describe the objective reality by means of minimal exact defined concepts, with the respect to its universality. Theory is then able to connect different experiences, rules and laws ( which have been judged separately up to now ) into a logic unity. From the economic point of view it is impossible, a priori, to estimate the expense on basic research and assess the economic utilities of the basic research.\nOn the other hand, the applied research does not usually bring a new fundamental information, but provides suggestions to the fundamental research. From the methodological point of view the new knowledge and theories come from the basic research which are processed by deductive judgements, with the aim to gain the highest economic profit.\nThe hope of the economic profit is greater and faster in the applied research than in the basic research.\nThe strictly economic consideration leads up to the conclusion that it is advisable to invest only to the applied research and the basic research should gradually be suppresed. In the course of the time this situation leads to a delay in the economic development in such a country.\nOn the other hand, the experience shows, that an exactly solved problem on the level of the basic research can lead to a great economic profit as e.g. the development of the new industry. The history of the control of growth of the semiconductor monocrystal confirms it. This achievement in the basic research leads not only to the development of semiconductor industry but also to the development of computer industry and development of the space technology.\nFrom the general point of view concerning the question of basic and applied research in the science we have to say that medicine was accepted as an art from the time of Hippocrates. This conception led to fanciful therapeutic processes in therapy which were based on the most fantastic ideas about the substance of diseases.\nThese voluntary experiments in the patients were stopped during the first half of the 19th century. Everything started by Lamarck's finding that the body of all animals consists of cells. During the 30's of the 19th century this finding was proved also in plants. At that time, the philosophical conception of Schelling's philosophy of nature, based on the deductive speculations, was overcome too. Therefore the time of observations and experimental verification of the findings of observation in medicine started.\nThis approach in medicine led to the definition of the scientific medicine and physicians created by R. Virchow:\n"According to our point of view it is self-evident that medicine involves the art of healing-although the most recent developments in medicine may make it appear as if this had hardly anything to do with the matter. Only those who regard healing as the ultimate goal of their efforts can, therefore, be designed as physicians.\nScientific medicine, for its part, has as its object the investigation of those altered conditions which characterize the diseased body or various ailing organs, the identification of abnormalities in the phenomena of life as they occur under specifically altered conditions, and, finally, the discovery of means for abolishing these abnormal conditions. It presupposes therefore a knowledge of the normal course of the phenomena of life and the conditions under which this course is possible. It is therefore based on physiology. Scientific medicine is compounded of two integrated parts - pathology, which delivers, or is supposed to deliver, information about altered conditions and altered physiological phenomena, and therapy, which seeks out the means of restoring or maintaining normal conditions."(1.)\nTo the defense of the conception of cellular pathology Virchow said: \n"I formulate the doctrine of pathological generation and neoplasma in the cellular pathological sense, in simple terms: omnis cellula e cellula."(2.)\nThis statement became the fundamental statement of the cell theory, and the cellular pathology is based on the conception of the basic biological theory- -the cell theory.\nThe cell theory is based on the following statement:\n1)The bodies of all animals and plants are composed of cells.\n2)All cells include the cytoplasma, as a carrier of living process.\n3)Omnis cellula e cellula - all cells arise from other cells.\nFrom the point of view of our contemporary knowledge, all statements of the cellular theory are false. We know that viruses and prions are a subcellular elements of life, we know that metabolism, as a carrier of living processes, occurs in all structures of cells and not only in cytoplasma. The third statement of cell theory is the tautology - the proof by the cycle. This above mentioned cogent arguments make us consider the cell theory as paradigm.\nNevertheless, the conception of scientific medicine has been accepted until now. The resolution of the genetic code led to the use of terms as biomedical research or biomedical sciences with the aim to designate the use of knowledge from the molecular biology in medical practice. \nThe accumulation of knowledge in the field of molecular biology leads to the conception of the so called translational research(3.) or large scale biomedical science(4.)\nThe aim of the translational research is an effective translation of the new knowledge, generated by progress in the basic scientific research, namely in molecular biology, into new approaches concerning prevention, diagnosis and treatment of diseases.\nIn my opinion, the terms "biomedical research" or "biomedical sciences" or "translational research" or ?large scale science? are only a certain magic formula by means of which we hide the theoretical deficiency i.e. cell theory and cellular pathology! This serves us to avoid the solution of the paradigm of the cell theory and the following paradigm of cellular pathology!\nOn the basis of the general theory of the development of science, medicine is at the end of the stage of the so called descriptive science. It is characterized by phenomenological theories. The aim of phenomenological theories is to define the relationship between the observable phenomena. At this stage of the development of sciences methodological implement is the solution to the question between enter and exit information of the black box. At the stage of descriptive science the criterion of truthfulness is only the experiment. The truthfulness of the result of the experiment is not proved by theory!\nWe can also say that medicine belongs to experimental sciences and the development of experimental sciences leads to the so called exact sciences. This transition is characterized by the shift of phenomenological theories to the non-phenomenological theories. Non-phenomenological theories are characterized by the introduction of unobservable or hardly observable terms into theory. At this stage of the development of science the methodological implement becomes the black box again. Now , i.e. at this level of exact sciences, we study the inner structure of the black box. The criterion of truthfulness is the experiment, but the result of the experiment must be proved by theory!\nThe necessity of the connection of the biological level of the movement of mass with other levels, i.e. physical and chemical ones and the shift of the phenomenological theory i.e. cell theory to non - phenomenological theory could be observed in the 19th century, when Virchow wrote: ?In order to comprehend the essentially cellular phenomena of life we must understand the composition, the mechanical characteristics, and the functional changes of cell substance; as far as the course of investigation is concerned, there can be no disagreement regarding the fact that chemical and physical investigation is primary, and the anatomical or morphological investigation secondary.?(5.)\n?All diseases are in the last analysis reducible to disturbances either active or passive, of large or small groups of living units, whose functional capacity is altered in accordance with the state of their molecular composition and is thus dependent on physical and chemical changes of their contents. Physical and chemical investigation has a very great significance in the respect, and we can do no more than wish a prosperous development to the school which is striving to form itself.?(6.)\nJ.E.Purkine arrived at similar conclusions:??the universal physical science developed in the progress of the natural sciences which describes the laws of all phenomena living and non-living on the mathematical basis.?(7.)\nOn the basis of the above mentioned facts we have to make things clear. Now, after the general analysis of the development of sciences and analysis of the present situation in the medical research and also in medical practice, we can go back to the questions appearing at the beginning of this article and find corresponding answers.\nAs described above, medicine involves all levels of the movement of mass. This specifity of medical practice undoubtedly indicates, that medicine requires its own basic research on all levels of the movement of mass. This requirement suggests that the field of basic research in medicine must be also a vertical connection of knowledge from different levels of the movement of mass and their application into medical practice.\nTo clarify the necessity of the basic research in medicine and its range we have to define the aim of it.\nThe aim is to establish the branch theoretical medicine, which would seek the basic and unchangeable causes of living processes, not only under the condition of health, but also in pathological states and their treatment.\nThe term ?theoretical medicine? is restricted only to the area of interdisciplinary collaboration between those, who are interested in the theoretical and clinical aspects of medicine and application of mathematics to the solution of problem arising from the present biomedical research.\nIn my opinion, we have to consider the term ?theoretical medicine? to be an integral of knowledge of all levels of the movement of mass ( physical, chemical, biological and psychosocial ) which are incorporated to the knowledge of clinical medicine.\nNow to the question of the fundamental characteristic features, which we will require from the general theory of medicine on the biological level of the movement of mass with the aim to substitute paradigm of the cell theory and substitute the paradigm of cellular pathology.\nOn the level of the biological movement of mass the basic building unit of the biological system has to be not only compatible with the chemical, physical and psychosocial movement of mass, but the fundamental unit of biological systems also has to accept the following criteria. The fundamental unit of biological systém has to be capable of:\n1)development\n2)reproduction\n3)selection\n4)creation of information.\nAfter we have found the exactly defined fundamental building unit of biological systems, we can define the primary pathological impulses, which lead to pathological states.\nAs far as the question of characteristic features of the primary pathological impulses is concerned, the condition of vertical connection on all levels of the movement of mass and contemporary requirements of medical practice must be kept:\n1)The basic pathological impulses must have identical physical and chemical characteristic features on the microlevel and at the same time on the macrolevel.\n2)The reaction to basic pathological impulses must have identical physical and chemical characteristic features as well as the analogic reaction on other levels of the movement of mass.\nThe first condition comes from the medical practice, the physician knows about the primary mechanism of pathological states because the initial impulse has the same character on macro and micro levels.\nThe second condition involves the demand of the vertical connection of different levels of the movement of mass.\nI am convinced that only this approach can build up a new theory of pathology and establish the branch of theoretical medicine, so that the best science could be a part of the best care for the patients.\n\nAbstract\nPractical medicine makes use of the knowledge concerning all levels of the movement of mass, i.e. the physical, chemical, biological and psychosocial levels of the movement of mass. The aim of the basic research in medicine must be the integration of knowledge from all levels of the movement of mass into medical practice. The conception of the translational research was based on the following facts:\n1)The disproportion between the new knowledge from the molecular biology and medical practice, which is oriented onto the idea of the cell theory till now.\n2)A small economic profit of the basic resarch as far as the aimed biomedical research is concerned.\nThe translational research solves the questions of the translation of the new knowledge from molecular biology to medical practice, i.e. only on the biological level of the movement of mass.\nTherefore, the translational research is only a non systemic solution to the problems of the basic research in medicine. And thus, the translational research is characterized, by onesidedness, which may lead to many mistakes in medical practice, especially as far as the damage to the patient, caused by therapy, is concerned.\nThe translational research absolutely avoids the solution to the fundamental problem of the basic research on the biological level of the movement of mass, i.e. the cell theory is not a theory anymore, but only a paradigm on the biological level of the movement of mass. The cellular pathology is based on the cell theory and therefore the cellular pathology is a paradigm, too. The concept of scientific medicine is composed of two parts: pathology and therapy. The cellular pathology is a paradigm and therefore the concept of scientific medicine is not scientific and it is wrong!\nThe author finds the solution to these problems in establishing the branch theoretical medicine and defines the basic properties of the fundamental unit of biological system and the properties of the fundamental unit of pathology.\nThe exact description of the fundamental unit of the biological system and fundamental unit of pathology, defined during the 80´s of the 20th century, led to the general theory of medicine and to the theoretical and practical aspects for the treatment of not only bacterial and viral infections ( AIDS ), but also for the treatment of cancer.\n\nBibliography \n1.Virchow,R.:Standpoint in Scientific Medicine.( 1847 ). In: Disease, Life, and Man. Edited by Rather, LJ. Standford: Standford University Press;1958: 26-27.\n2.Virchow,R.:Cellular Pathology.( 1855 ). In: Disease, Life, and Man. Edited by Rather,LJ. Standford: Standford University Press;1958: 88.\n3.Fontanarosa,PB. and DeAngelis CD.:Basic Science and Translational Research in JAMA.JAMA April 3, 2002-Vol.287,No.13,p.1728.\n4.Large-Scale Biomedical Science:Exploring Strategies for Future Research (2003). National Academies Press.\n5.Virchow,R.:Cellular Pathology.( 1855 ).InNational Academies Press: Disease, Life, and Man. Edited by Rather, LJ. Standford: Standford University Press;1958:84-85.\n6.Virchow,R.:Cellular Pathology.( 1855 ). In: Disease, Life, and Man. Edited by Rather, LJ. Standford: Standford University Press;1958:100.\n7.Janko Jan and Strbanova Sona:The Science of the Time of Purkinje. Praha: Academia; 1988.[in Czech].\n\n\nWith respect to the use of the statistic method I have to draw attention tn the article: I.Prigogine: Time, Chaos and the Laws of Nature in: Law and Prediction in the Light of Chaos Research pp. 3-9, eds.: P. Weingartner & G. Schurz, Springer Verlag, 1996, where as far as I know, the disproportion between statistical and individual description was first described.. From this point of view, the use of statistic methods in medical research is too limited. \nIn present medicine, exactly defined basic terms concerning the non-phenomenological level at all levels of the movement of mass are missing. See CORDIS Partner Search under my name.\nIn my view the lack of exactly defined basic terms on the non-phenomenological level at all levels of the movement of mass is strictly caused by the decrease of the scientific and research productivity in the medical area. \nsmidlj.md@seznam.cz
Avatar of: anonymous poster

anonymous poster

Posts: 31

February 13, 2008

Perhaps it is time to re-examine what the metric for NIH-supported research should be and also to determine whether the money is and has been spent wisely. Is a published paper what the taxpayers really want, or is it progress in curing or managing disease? We have several problems in science. Among these are:\n\nIncreases in funding are like providing nutrients to a bacterial culture. The number of scientists competing for funds rapidly increases with an increase in funding thereby leading to the current situation of instability. Concurrent with this is excessive indirect costs. Aside from siphoning funds away from actual research, the rate is so high that research actually is a money-making enterprise for institutions. Thus, facilities expand with the budget thereby requiring scientists to staff them and increasing the competition among investigators. Reduce indirect costs so that institutions are not making a profit.\n\nMany areas of research may churn out extensive numbers of papers, but yet have little or no impact on disease. An example is much of the chemoprevention research. Billions have been spent in this area, yet nothing of practical import has come from it. No drugs, very little cancer prevented. I only mention this field as an example and do not mean to disparage the many dedicated scientists who are sincerely devoting their efforts to this field. There are many other areas of research equally unproductive.\n\nClinical trials and RFAs are prematurely issued, often in response to political pressure. An example is work on interstitial cystitis. Few people actually working on the disorder are funded because grants are routinely trashed because the etiology is unknown and therefore the definition of the disorder is strictly clinical. Yet, "hypothesis-driven" grants from people working on related areas that probably are unrelated to the disorder itself receive the funding, or else it has gone to clinical trials of drugs that have proven ineffective. A more discovery-based or inductive science model would be more useful than the deductive model so favored. Unfortunately, this approach tends to follow what has gone before, so more and more is learned about less and less while less defined and perhaps more productive pathways remain uninvestigated because "the evidence for the hypothesis is weak." \n\nMention was made of megalabs and poor-quality postdocs. Yes, I think that is a very real problem. Perhaps the number of R01 grants that one individual can hold should be capped. After all, just how much attention can one person really devote to guiding the science in his or her group? I would think that as the size of a lab increases, real productivity tends to decrease because the leader devotes so much time to managing postdocs, filling positions with bodies and then trying to cover for the poor quality ones, writing grants and the like. \n\nIn summary, perhaps it is time to completely re-examine the system of how money is parceled out and how results are assessed. It is not clear that papers in the literature should be the metric.\n\n\n\n

Follow The Scientist

icon-facebook icon-linkedin icon-twitter icon-vimeo icon-youtube
Advertisement

Stay Connected with The Scientist

  • icon-facebook The Scientist Magazine
  • icon-facebook The Scientist Careers
  • icon-facebook Neuroscience Research Techniques
  • icon-facebook Genetic Research Techniques
  • icon-facebook Cell Culture Techniques
  • icon-facebook Microbiology and Immunology
  • icon-facebook Cancer Research and Technology
  • icon-facebook Stem Cell and Regenerative Science
Advertisement
Advertisement
Mettler Toledo
BD Biosciences
BD Biosciences