Tough Crowd

Reviewers who are experts in a given field are more likely to deliver harsher critiques of papers that hit close to home, researchers report.

Joshua A. Krisch
Nov 4, 2016

FLICKR, NICMCPHEEThe best peer reviewers are the top experts in their fields—so goes the conventional wisdom. But the results of a study published last month (October 21) in PLOS ONE suggest that the harshest critiques often come from those reviewers who know their fields best, and that seniority and inherent bias can skew how some top experts evaluate manuscripts submitted to journals.

“These results highlight the need to explore how reviewers utilize their expertise,” Stephen Gallo of the American Institute of Biological Sciences in Reston, Virginia, and coauthors wrote in their study.

Peer reviewers are known to sometimes disagree on the merits of a given study. One recent analysis of biomedical funding applications found that reviewers with higher expertise detect more weaknesses than non-expert reviewers, suggesting that expertise might sway decision-making.

In the present study, Gallo and colleagues conducted a retrospective analysis of evaluations submitted by reviewers who had self-reported their levels of expertise and were aware of the demographics and training of the submitting authors. “We were then able to measure the effects of subject matter expertise on scoring in a scenario that may be governed by both social networking/status effects as well as bounded rationality,” the authors wrote in their paper. They found that the single largest factor influencing each review was not demographics or seniority, but expertise. And experts who reported feeling most familiar with the subject matter at hand doled out the harshest reviews.

“It is likely that the processes of bounded rationality dominate over social influences, which is an important finding given the many claims of bias and cronyism in peer review,” the authors wrote. But that doesn’t preclude other forms of bias from influencing reviews: “Social effects may play out in more subtle ways,” they added.