Opinion: Mind the Measures

These trends impact debates about federal funding at the national level, but there are also trickle down effects on universities, medical schools, and academic health systems across the US, where individual scientists and teams conduct research. Leaders at these institutions face increasing pressure to assess their investments in research and communicate the impact to their local stakeholders—state governors and legislators, boards of directors, community partners, patients, and their families.  In these academic settings, research success has been traditionally measured by respected and credible academic metrics, such as volume of grant funding and the number of publications and citations, founded on peer review.

But metrics that rely on quantifying monetary inputs and academic outputs alone cannot paint the full picture of the complex multi-year scientific process through which research translates into outcomes that matter to the public. While celebrating nationwide, population-level health improvements made possible by research investments, institutional leaders could benefit from evaluation frameworks that help them communicate those success to stakeholders.

A number of national research evaluation frameworks have been studied—for example, the United Kingdom Research Excellence Framework, Excellence for Research in Australia, and Evaluating Research in Context from the Netherlands. Within these frameworks are mechanisms capable of analyzing investments, allocation of funds, and advocacy and public education about the value of research.  

Aspects of these frameworks could provide institutional leaders with an enhanced suite of tools, beyond traditional academic metrics, to help assess their investments as well as to garner stakeholder support by framing measures that speak to the broader public.

We must, however, weigh these potential opportunities against some wariness in the research community about engaging in new methods of evaluation. One reason for caution is that evaluating research is not easy—the methodologies are complex and require a conceptual understanding of the theoretical underpinning, rationale, goals, and implementation of the evaluation tools. Research leaders have long worried that “evaluation methods du jour” will disregard the inevitable time lag for research to lead to breakthroughs in health, will become unfunded mandates that create laborious and meaningless reporting for researchers, or will upend the foundation of peer review at the heart of evaluating research in the US. New evaluation methods may also create a new set of “rankings” that could disregard the local context and value of research, and could ultimately result in ill-founded attempts to curtail the dwindling dollars so critical to research.

In order to address these concerns, any evaluation tool should be an adjunct to and guided by  experts who understand the context for conceptual models for how research translates to academic, health, social, and economic impacts, and who can build consensus on how best to measure those impacts in a rigorous, systematic, and institutionally-appropriate manner.

Nurturing and sustaining public support for the full spectrum of medical research from bench to bedside to community, and for ensuring a diverse, robust research pipeline is essential for our future. By taking the lead in demonstrating accountability as well as evaluating and communicating the value of medical research to broad audiences with a suite of academic and non-academic measures, the research community can bolster ongoing public support for funding.

Ann C. Bonham is Chief Scientific Officer at the Association of American Medical Colleges. Prior to joining the AAMC, Bonham served as executive associate dean for academic affairs and professor of pharmacology and internal medicine at the University of California at Davis School of Medicine.