Integrity Concerns Fail to Slow Citation of Problematic Papers

Neither retractions, expressions of concern, nor other editorial notices seem to keep authors from continuing to cite problematic papers, according to a look at what happened to more than 170 articles by one author.

“After the public notification of integrity concerns about an article, it would be expected that other authors would no longer cite the article because it is unreliable,” write the authors of a new preprint.¹ But that’s not what they found in a limited comparative study. Whether the study is generalizable has yet to be seen, says one other expert.

Four sleuths – the University of Aberdeen’s Hugo Studd and Alison Avenell and the University of Auckland’s Andrew Grey and Mark J. Bolland – charted citation data for 172 papers on clinical trials from Zatollah Asemi, a nutrition researcher at Kashan University of Medical Sciences in Iran, whose work has come under scrutiny.

Of these, 23 had been retracted, 38 had expressions of concern, 41 had some other type of editorial notice, and 70 had not been subject to any flag. The preprint authors sought to test how the type of editorial response affected citations, given “differences in their visibility or perceived seriousness.”

Bibliometric researchers sometimes refer to retracted or marked papers that continue to circulate as “zombie papers.”² Data suggest the vast majority of retracted papers continue to be cited as if they had not been retracted.³ Even when citations of retracted papers do dwindle, it’s been unclear whether that’s due to the paper’s age rather than its editorial status.

The authors of the new preprint have been involved with the Asemi case — and this specific dataset — for years. In 2019, they flagged the 172 papers to the journals that published them with allegations of unethical conduct, results that didn’t match the study design, data irregularities and myriad other integrity issues. For those papers on which editors have taken action, notices have been published an average of five years after publication. As Grey told us back in 2021 in regards to journals’ response times on Asemi’s papers: “Yep, pretty slow.”

The randomized controlled trials reported in the articles examined had collectively been cited more than 10,000 times in more than 6,000 articles. For both the flagged articles and those with no notices, “citations increased steadily, peaking 45-65 months post-publication,” with similar declines thereafter. The researchers found “little evidence that publication of an editorial response, whether it was retraction, publication of an EoC or issuing of an editorial notice, had any meaningful effect on the rate of citation.”

In a jointly signed response to our questions, the four authors told us they were unsurprised by their results, given the years of delay between publication time and notice. They were also unsurprised that the black-marked papers continued to be referenced in new work. “By then, these [trial] publications were probably imbedded within the literature and databases, resulting in continued citation regardless of editorial action,” they wrote.

The authors told us they believe the results of the paper are generalizable, even though it covered one author’s body of work. That’s because the trials spanned 65 journals and were published by 25 publishers.

Jodi Schneider, director of the Information Quality Lab at the University of Wisconsin-Madison, is not as convinced it’s generalizable. But, she added, “Their continued analysis of this case study is quite valuable for the research community — with sobering results.”

This article was authored by Avery Orrall, and it was first published at Retraction Watch.