Opinion: Rethinking Scientific Evaluation

The quad of King's College Medical School Guys Campus RICHARD NAFTALINThe upcoming Research Excellence Framework (REF) facing British universities in 2014 is a major threat to physiology and other basic medical sciences. The primary focus of REF will be to assess research excellence of tenured university professors, as demonstrated by the quality of their research publications. This new system for assessing the quality of research in UK higher education will be undertaken by the country’s four major funding bodies and will be used from 2015–2016 to selectively allocate research funding. Basic medical sciences and clinical science departments within the medical schools are in direct competition for financial and human resources and will be judged by the same panel of experts and the same criteria. To be included as an active research scientist in an elite university’s submission to REF requires three recently published papers in journals with high impact factors. The number of citations that each submitted paper attracts will also count but in elite institutions only papers published in journals with an impact factor of 5 or greater will be submitted for assessment by REF. Papers graded by REF as “outstanding” will earn their institution £100,000 (~$154,000); those rated merely “excellent” will be awarded £25,000 (~$38,500); anything less will be given no funding. Three major British basic medical sciences academic journals, the Journal of Physiology, Biochemical Journal, and the British Journal of Pharmacology have impact factors for 2012 of 4.88, 4.1 and 4.4 respectively. Their US equivalents, American Journals of Physiology, Journal of Biological Chemistry, and Biochemistry have impact factors 3.3, 4.77, and 3.4, respectively. All of these are just below the cut to be included in REF assessments. In comparison, the top three gastroenterology journals have the following impacts: Gastroenterology, 11.6; Hepatology, 11.6; and Gut, 10.1. Similar high impacts are found in the major cardiovascular, renal, and respiratory clinical medical journals. Why is the average impact of a paper published in a clinical journal rated three times higher than similar papers published in basic academic science journals? Is it because they are three times better scientifically and thus merit their impact? Or is it because clinical science journals have a much larger readership? Many more clinicians work in cardiovascular medicine than lab researchers in basic cardiovascular science; similarly 10–20 times more clinical scientists work in gastroenterological or renal medicine than basic scientists in epithelial biology. So readership and numbers of active scientists in a specialty are a major factor in determining specialist journals’ impact factors—calculated as the number of citations of all the citable items published by journal in the preceding 2 years relative to the number of citable items published—and have little bearing on the scientific excellence of their content. Scientific publishers understandably need journals with a wide readership and impact factor to maximize income from sales and advertising. Regrettably, however, this business plan has been also largely adopted by the REF in determining the value of scientific output of individual scientists. This is not how we should be assessing the value of research—publications in lower impact journals do not necessarily reflect lower impact research. The ways in which citations are assessed by REF and by the triage committees within UK universities must be revised by normalizing for the relative size of the field. For example, if the average citation of papers in the field of physiology is 3.0 and in medicine is 6.0, then physiology papers should have a multiplier of 2 to maintain parity. Historically, and in contrast with this commercial approach adopted by REF, a primary function of academic scientific societies, such as the Physiological Society or the American Society of Biochemistry and Molecular Biology, is the furtherance of scientific research. They do this through meetings and by encouraging their members to publish in their society-run journals. Profit is not their raison d’être—it is promotion of their scientific activity. Allowing REF in UK universities, particularly those with medical schools, to use a crude impact factor rating for the comparison of the relative merits of clinical and basic medical research within the same school will result in a shift of funds from basic medical sciences to clinical sciences. This will also encourage scientists to angle their work towards a clinical audience purely for the sake of increasing their REF evaluations and bringing in money to the university. A good grounding in biological science is essential for both biological and clinical research. However, as the opportunities for basic research dry up, so will this source of well-trained scientists for both basic and clinical science. Richard Naftalin is Emeritus Professor of Physiology at King's College London. He was trained in medicine at Glasgow University and in biochemistry London University.