arXiv Takes the First Step to Combat “AI Slop” in Submissions

The introduction of generative AI tools has expanded access to many writing resources in research, from helping translate between languages, improve fluency, and correct grammar. Alongside these benefits, though, some people have used AI beyond its role as an assistant to polish their finished product. In some instances, researchers have used AI to help them write their paper but left in copied text from the AI prompt, comments from the model, or hallucinated references, suggesting that they did not check the model’s output for accuracy. Some editors have referred to articles with these characteristics as “AI slop.”

Maxim Topaz, a researcher developing AI tools for clinicians at Columbia University, emphasized how the presence of articles with hallucinated citations in the public record has damaging consequences. “The scariest thing about that is that these papers with fabricated citations get cited down the line. They get cited in systematic reviews. They get cited in clinical guidelines. And then when you go to the doctor, that's how doctors make decisions on what to do with you,” he said. He and his team demonstrated recently that a large fraction of the literature in PubMed contains fabricated citations, possibly originating from AI.¹

This problem extends beyond published research, too. Seeing an increase in submissions with these problematic features, Thomas Dietterich, a computer scientist at Oregon State University and the chair of the computer science section of the preprint server arXiv, announced on social media that authors who upload articles with “incontrovertible evidence of AI slop” would be banned from the server for one year. After this time, they would be required to have their work accepted in peer reviewed outlets before they could upload their work to arXiv.

The announcement created a stir on social media, with some people applauding the decision, others raising concerns, and some arguing that the policy doesn’t go far enough.

Researchers Respond to arXiv’s Strict AI Penalty

Steinn Sigurðsson, an astrophysicist and the scientific director of arXiv, expressed the necessity for this action. Although scientific integrity issues are not new, he said, “This is happening quickly at scale as opposed to being a relatively rare, high-profile incident.” The overwhelming number of submissions, many of them low-quality, has forced some preprint servers to stop accepting articles.

By drawing attention to the problem and clarifying arXiv’s policies, Dietterich said that the team hopes to address this misconduct. “We want authors to be more responsible and more careful in their work, and we'd be delighted if, a year from now, we didn't have any papers like this,” he said.

When he made his initial post on social media, though, many researchers took the policy announcement to mean that the penalties would apply to all coauthors. Additionally, some people interpreted the requirement for peer review before being allowed to submit their preprint to arXiv to be a permanent penalty.

Lucas Sjulson, a physician scientist at Albert Einstein College of Medicine, was one researcher worried about how the far-reaching implications of such a policy could affect future collaborators and trainees. “I agree that there should be some sort of accountability and there should be penalties when people do irresponsible things, but I think that the notion of a permanent ban was what bothered me,” he said.

Others, though, felt the policy and its penalties were reasonable. Mathieu Rebeaud, a biochemist at the Swiss Federal Technology Institute of Lausanne, said that these improper uses of AI in research publications pollute the scientific literature. “The harm that you are doing to science by generating false data, false citation, false stuff like that is awful,” he said.

Dietterich clarified that the penalty only applies to the researcher who uploaded the article to the server. Additionally, he said that if the penalized author has a sufficient number of publications—often set at three—successfully accepted in peer reviewed journals, then that person can appeal to the arXiv team for the requirement to be lifted so that they can submit preprint articles without review again.

Sjulson said that this clarification addressed his major concerns with the announcement of the updated policy. “This situation highlights not only the importance of repositories having carefully designed policies, but also of communicating them clearly to the research community,” he said.

Dietterich’s social media post about AI use also garnered concerns from other researchers who assumed that this was a new policy and expressed confusion about not seeing it on the website; Dieterrich and Sigurðsson explained that this announcement was instead clarifying that this example of misconduct would fall under the server’s existing code of conduct policy, which is left broad to accommodate changes in technological abilities. “There's a bunch of stuff that you shouldn't do, and doing them in a new and clever way is not going to get you around the policy,” Sigurðsson said.

Considering his team’s findings of large numbers of hallucinated references in published literature, Topaz said, “I think the instinct is right,” of arXiv’s announcement. But he added that one issue could be enforcement, since preprint servers are predominantly operated by volunteers.

Dietterich said that the arXiv teams currently use a series of checks as part of their normal review process that can help identify improper AI use, including comparing the statistics of word frequencies and an article’s metadata. He added that if any article is flagged, two different people review the item in question before they assign a penalty.

As AI keeps advancing, though, Dietterich said, “I think we have greater challenges down the road. But right now, we can at least detect some of the most egregious cases and penalize them and try to improve things.”

Navigating AI Use in Science Publishing Remains a Challenge

Beyond preprint repositories, researchers and editors alike are learning how to manage irresponsible AI use. Rebeaud emphasized the need to stop fabricated content from entering the scientific record based on his own experience. He encountered a manuscript he was asked to review that included AI comments in the text and requested that the article be rejected on that premise. “If you are not reading the paper from your peers and you are not making this effort to understand what you are citing, it’s not possible to write something good. So, you must be rejected,” he said. He hopes that more publishers and preprint servers will follow arXiv’s lead.

Richard Sever, the cofounder of bioRxiv and chief scientific and strategy officer of openRxiv, said in an email that the team at the biological sciences preprint server also has seen an increase in these submissions. “We have not yet considered bans on submitters but feel it is important there are repercussions for individuals who do this,” he said. But he added that he is more concerned about the examples of AI fabrication that are less obvious to catch. Ultimately, Sever said that he expects there will be a combination of screening before uploading and detection afterward. “In some cases, the reputational damage from exposing this when articles are already public may be a significant disincentive for people to do it in the future,” he said.

Topaz said that the research community will need to discuss penalties of this nature and the extent of their reach, but he added that punishment cannot be the only solution. “[Bans] deter the careless; they don’t catch the careful,” he said. He said that publishers should begin using available tools to systematically curb improper AI use in writing.

A spokesperson for journals published by the Society for Neuroscience said that they have not seen an increase in submissions with fabricated AI references or AI comments and have thus not needed to update their guidelines. Meagan Phelan, the communications director for the Science family of journals, said that their teams have also not seen more articles with these problems, but that in the event of such an example, they would follow their current policies, which include the potential for rejecting or removing the content from the website. Phelan added that arXiv’s announcement related to penalties for inappropriate use of AI “speaks to the increasing importance of trusted filters for quality and integrity in a world of proliferating science content.”

Meanwhile, Sigurðsson said that he has talked to some journal editors who said that, although they are not announcing it, they are cracking down on researchers who submit articles with blatant AI-introduced errors.

The arXiv team hopes that their heightened enforcement of fabricated AI-generated content reduces the instances of submissions with these errors. Topaz added that it has started an important conversation that he hopes will continue.