Thomas J. Phelan states in his Oct. 2 Opinion1 that peer review has no alternatives for single research product quality evaluation, but publication and citation alternatives may exist for evaluating aggregated research quality. I have examined and developed myriad research evaluation methodologies for some time,2, 3 both for single and aggregate research units. For research evaluations that will impact real-world decisions, peer review is required at all levels of aggregation.2 Publications, patents, citations, and other output (and outcome) metrics, and even the more detailed retrospective studies, can supplement peer review usefully, but do not replace peer review.
The real problem in evaluating research productivity and impact at any level of aggregation (individual, research unit, agency or company, nation) is that shortcuts in the conduct and analysis of these evaluations are often substituted for the intensive detailed labor required for high-quality results. In many real-world cases, there is little motivation to ...
