Redesigning Scientific Reputation

Peter Frishauf, founder of Medscape, has proposed that quantifiable “reputation systems” would reflect the depth of review a paper has undergone, and could serve as a reward system for those scientists who contribute to enhancing or judging a paper’s value. He draws his ideas from a 2003 paper by Jeff Ubois, published in Esther Dyson’s Release 1.0 newsletter, titled “Online Reputation Systems.” From this early work he expands on research by us and others who have proposed quantification schemes based on longevity and impact of text passages in articles and review comments. Thus, one’s reputation is not measured by credentials, but by one’s contribution both to expanding knowledge and to the community.

If an online reputation system is to replace the traditional process of review, several technical obstacles must be overcome. How to prevent the site from being swamped by second- or third-rate submissions, if not outright spam? How to ensure that all promising papers, including those by novices, receive their due share of insightful comments? And that the most qualified reviewers distribute their attention impartially, rather than flocking to authors they know already? How to ensure that comments actually add value, and how to prevent or eliminate reviews that are wide of the mark?

Our work in building large-scale reputation systems suggests that it may be possible to build such a system on two pillars: a system of incentives for authors of papers and reviews alike, and a content-driven way of measuring merit and attributing rewards. The reputation of people as authors would depend, as usual, on their publishing works of high measured value. And crucially, the reputation of people as reviewers would depend on their ability to be early reliable predictors of the future value of a work. Thus, two skills would be required of a successful reviewer: the ability to produce reviews that are later deemed by the community to be accurate, and the ability to do so early, anticipating the consensus. This is the main factor that would drive well-respected people to act as talent scouts, and to review freshly published papers, rather than piling up on works by famous authors. Reviews would be ranked by reputation, thus diminishing irrelevant comments, as Amazon has shown it is possible to do.

Using reputation scores to help search, to guide which reviews emerge, and to promote one’s academic standing would provide strong incentives to researchers to interact productively with their colleagues and to spotlight great science.

Bo Adler and Ian Pye are PhD students of Luca de Alfaro at UC Santa Cruz, who is currently on leave at Google Inc. They are working on the WikiTrust project to provide reputation analysis for Wikipedia and other open collaboration systems (e.g., Web forums, Google Maps, and DNS). They recently created a Wikipedia vandalism detection tool based on reputation analysis. Opinions expressed here are views of the authors and do not necessarily reflect those of their employers.