Reviewer3: An AI-Based System to Streamline Peer Review

As a graduate student at the University of Southern California, biomedical engineer Natalie Khalil became frustrated by the peer review process. “I had always felt that peer review was a lottery system where it mattered more who was looking at your paper than the work that you did,” she said. She added that the idea that authors could recommend specific individuals as reviewers for their own study seemed “bizarre,” and that the whole process took too much time.

To address these issues, Khalil cofounded and built Reviewer3, an AI-based platform that can review claims and evidence in a manuscript. Unlike other AI-review platforms, Khalil and her cofounder, computer scientist TJ Reed, wanted to create a tool that scrutinized verifiable parts of a manuscript, like authenticating results, detecting AI-generated text, and confirming image accuracy, as opposed to judging a work’s novelty. The system uses multiple agents to evaluate the data and conclusions in a study—such as reproducing the findings, rerunning the statistics, or validating the code in an open sandbox environment—to identify which conclusions are supported and flags those that have logical gaps or “fatal flaws.” Reviewer3 also has access to five reference databases to crosscheck citations and includes built-in AI-detection tools.

Khalil said that Reviewer3 was designed to help authors strengthen their manuscripts and allow human reviewers to spend more time evaluating a work’s scientific importance and novelty. So far, the company has reviewed more than 30,000 manuscripts from over 10,000 researchers, most of them authors seeking to edit their own work before submission.

Felipe Muñoz-Guzmán, a systems biologist at the Millennium Institute for Integrative Biology, has previously used another AI-based reviewer tool, QED, to identify where he could strengthen his message or make it clearer. He compared this to sharing one’s draft with a colleague. “Because you are all the time totally into [your work], you never see the weakest part of your story,” he said.

Comparing QED to Reviewer3, he said that the two can both be helpful but noted that they focus on different aspects of the study. While he found QED to help find points that could be explained better, he said that Reviewer3 identified areas that could benefit from experimental support instead of narrative structure.

Osvaldo Contreras, a cell biologist at Victor Chang Cardiac Research Institute, also uses institutionally-provided AI reviewer tools as an author, which he said can help address typos and stylistic elements of manuscript drafts. Currently this does not include Reviewer3, so he has only explored the platform for assessing published works and other AI-generated reviews. He said that he would encourage his trainees to use a university-provided AI reviewer to first revise these tedious items out of their drafts so that his subsequent edits can focus on the scientific interpretation and technical details that rely on his expertise.

Muñoz-Guzmán added that catching and addressing issues with a manuscript’s clarity and supporting elements could also make peer review in scientific publishing more efficient. But he added that many tools still lack sufficient training in scientific domains. For instance, he has occasionally seen recommendations for experiments that don’t make sense in his work.

For reviewing manuscripts at the publishing stage, though, Muñoz-Guzmán and Contreras both emphasized the importance of human decision making remaining central to the process. Contreras agreed with Khalil, though, that peer review suffers from human bias and time constraints of many researchers. “That's why I also think AI could do, and is doing, an amazing job to at least standardize and help reviewers and editors to make those very critical decisions,” Contreras said. Muñoz-Guzmán noted that all parties should be transparent about their use of tools to edit or review manuscripts as well.

Contreras, on the other hand, said he tries not to use AI as a first approach in peer reviews. “I think you can become very sloppy when you start relying in AI in your daily basis,” he added. Instead, he reviews the study himself and then occasionally uses a reviewer tool to see if there are other errors he overlooked, such as typos or issues in clarity. “AI is a fantastic tool and a strategy to also help you from a copilot point of view to perhaps identify things that escape your mind,” he said.

Overall, though, the two agreed that AI-based tools could help address current challenges in the peer review process that is worsening as journals see increasing numbers of submissions, including those generated by AI. Khalil hopes that Reviewer3 can provide a helping hand to this struggling process. “Ultimately this crisis is unfolding before our eyes, where we just don't have enough human reviewers to scale with submission volumes,” she said. “We're going to need to use humans and AI tools effectively together, in a way that's complementary, to keep up.”