View comments(3) | Comment on this blog   

Researchers have proposed a new scheme for ranking the quality or impact of scientific journals that they say is more accurate than the Impact Factor, according to a paper published last week in PLoS ONE.

Rather than relying on an average of citations to rate a journal, the system uses a mathematical model to characterize the typical number of citations that papers in specific journals are likely to receive.

Since its inception in the 1960's by the Institute for Scientific Information (now Thomson Scientific), the Impact Factor has become the standard journal ranking system. It's calculated by adding up all the citations published in a particular journal during a two year period and dividing that number by the total number of citable papers during that time.

But Luis Amaral at Northwestern University and colleagues say that method has two major problems. First, it relies on the mean value, which "seems like a very natural number," Amaral explained, but only if the distribution of what one is measuring is normal, i.e. bell-shaped -- not the case for paper citations.

Amaral and his team looked though a database of 23 million papers published in more than 2,000 journals since 1955. The most cited paper in the pool got 200,000 citations, while half of all papers didn't get cited at all. "You have this really broad range, so the mean is a really bad measure to use," he said; if one paper in a given year receives 1000 citations, "no matter if none of the other papers have got many citations, that paper is going to change significantly the mean."

The second problem with impact factor, he said, is that papers in different fields get cited at different rates. In medicine and physics, papers tend to get cited very quickly after they're published, while those in economics or math can take a couple of years to accumulate citations. That's true for individual journals within a field as well, he added.

So they converted the number of citations of each paper to its logarithm, in order to "bring these numbers that are very broad closer to a narrower range." The logarithms of citations for individual papers did form a normal, bell-shaped distribution for within specific journals. Over time, the likelihood that a paper will get cited decreases, so its citations reach a stationary state, so the researchers then developed a mathematical model to predict how quickly papers in different journals acquired their "characteristic" number of citations. "That means there's a sort of pattern to a given journal," Amaral said, and that pattern is stable from year to year.

Under their system, journal rankings change somewhat: Trends in Biochemical Science, for example, ranks quite highly according to the Impact Factor (13.863), above journals such as EMBO, Plant Cell, and the Journal of Biological Chemistry, but much lower, and below those others, in the new ranking system. (See data comparing the group's ranking system to the Impact Factor for journals in biochemistry and molecular biology here and cell biology journals here. )

In a good ranking system, papers from higher ranked journals should have higher numbers of citations than papers from lower ranked journals most of the time -- a criterion that, according to their analysis, their system (called q) meets better than the Impact Factor, Amaral said.

Jim Pringle, vice president of product development at Thomson Scientific, wrote in an Emailed statement in response to the paper: "Thomson Scientific encourages experimentation and development of new citation-based methodologies and metrics. This particular article applies complex new mathematical techniques in a potentially interesting way. We look forward to the methodological discussion that will no doubt ensue." Pringle also directed readers to journals such as Scientometrics and the Journal of the American Society for Information Science and Technology (JASIST), and Thomson's recently launched Citation Impact Forum, for further insight into rankings.

What do you think about this method -- is it better or worse than the Impact Factor? Tell us in a comment; we promise not to take it personally. (The Impact Factor was developed by Eugene Garfield, who also founded The Scientist.)