ABOVE: © ISTOCK.COM, FOTOGRAFIABASICA
Update (August 8): The University of Minnesota has now updated the trial record for its COVID-OUT study to show an actual primary completion date of February 14, 2022. Final enrollment numbers are not yet provided.
Confusion about terminology on the world’s largest clinical trials registry may be delaying the release of drug trial results and undermining rules designed to promote transparency, an investigation by The Scientist has found.
Key study dates and other information are entered into the ClinicalTrials.gov database by trial researchers or sponsors, and are used by US science and regulatory agencies to determine legal deadlines by which results must be reported. The rules are supposed to ensure timely public access to findings about a potential therapy’s harms and benefits, as well as provide the scientific community with an up-to-date picture of the status of clinical research.
But neither the agencies nor staff overseeing the database routinely monitor individual trial records for veracity, instead relying on the person in charge of a given record to correctly declare information such as when a study ends and how many people were enrolled.
Our examination of a subset of COVID-19 trial records shows that some of this information can be wildly inaccurate. Specifically, listed end dates for studies can be wrong by months or even years—meaning that reporting deadlines based on those dates are essentially arbitrary. The findings, which in some cases appear to reflect a lack of understanding among researchers about how to use the database, imply that many more studies than previously appreciated could be in violation of US reporting regulations.
Established in 2000 by the National Library of Medicine (NLM), part of the National Institutes of Health (NIH), ClinicalTrials.gov has become the go-to site for scientists, public health experts, journalists, trial participants, and the public to track the progress of clinical research on human therapies, vaccines, and other medical interventions. In the last couple of years, with the world racing to find effective treatments for SARS-CoV-2 infection, the registry has accumulated thousands of records related to COVID-19 alone.
Various regulations aim to promote transparency on the site. US legislation enacted in 2007, and beefed up by a 2016 rule, requires organizers of later-stage trials to post summary results within a year of completing the main part of the study unless they obtain an extension. For trials deemed applicable—a classification based on a study being conducted within the US or its territories and later than Phase 1, among other criteria—violating this so-called “final rule” can carry a fine of more than $10,000 per day and result in the cancellation of government grants.
Listed end dates for studies can be wrong by months or even years.
The rule applies whether a study was successful or not, and is designed to dissuade organizers from sitting on negative results or other information that could be used by researchers and physicians who want to try the same treatment in other patients. (The legislation has been enforced sparingly—the FDA issued its first noncompliance warning just last year despite thousands of cases of late reporting, according to a Science investigation.)
See “New Automated Tool Monitors Clinical Trial Reporting”
The final rule’s deadline is set one year after the primary completion date—the date on which a research team finishes data collection for the main, or “primary,” outcome measures of a study. “The primary completion date is the most important data element in ClinicalTrials.gov,” explains Anthony Keyes, program administrator for clinical research operations at Johns Hopkins University’s Institute for Clinical and Translational Research. “If that date is wrong, everything is wrong.”
Yet The Scientist’s investigation of COVID-19 research on ClinicalTrials.gov found instances where the guidelines on reporting this date and related information are not being followed, with listed primary completion dates sometimes overshooting the true date by a year or more. These inaccuracies are hard to detect, are not actively monitored by the NLM, and in some cases appear to reflect miscomprehensions on the part of trial organizers.
The database isn’t set up to detect “ignorance or intentionally deceptive” behavior, says Evan Mayo-Wilson, an epidemiologist at the University of North Carolina Gillings School of Global Public Health who studies transparency in clinical research. If some other date than the primary completion date is put into that field on ClinicalTrials.gov, “either the people entering the data don’t know what data they’re supposed to enter there, or people are entering something they know they’re not supposed to be entering. . . . There’s no way that the National Library of Medicine knows [if] you’re just making it up.”
Identifying possible cases of misreporting
The primary completion date is formally defined by the NIH as “the date that the final study participant was examined or received an intervention for the purpose of the final collection of data for the primary outcome.” This date is unaffected by how long it takes researchers to analyze data, write their manuscript, or complete any other tasks associated with the study. It is also distinct from the “study completion date,” which is when data on all primary and secondary outcomes, as well as any adverse effects, have been collected.
Until a trial finishes data collection for primary outcomes, its record lists an “estimated primary completion date” reflecting when organizers think the date will be reached. Once it is reached, researchers legally have 30 days to update the trial record to show the “actual primary completion date,” plus final patient enrollment numbers, NIH’s guidelines state.
In some cases, investigators or their institutions acknowledged that the listed primary completion date was inaccurate.
ClinicalTrials.gov currently contains more than 150,000 records of studies in the US. To investigate possible date errors, The Scientist created a defined sample of records where the listed completion dates seemed most likely to contain clear inaccuracies. To that end, we focused on ongoing, short-term studies of potential treatment interventions for active COVID-19 infection that were launched in the US before March 1, 2021, had stopped recruiting patients but had not been suspended, and had estimated primary completion dates at least one year after recruitment ended.
These trials had to start treating patients promptly—i.e., before people recovered or perished from the disease. Assuming a trial followed its stated protocol, the expected primary completion date can be roughly estimated by adding the time allocated for collection of primary outcome data to the date that the study stopped recruiting new patients. While such a focused approach can’t estimate the frequency of date errors in the database, it provides a way to explore whether such errors happen and why they might go unnoticed.
Our criteria yielded a dozen US trials with various sponsors—including biotech companies and major academic institutions—that ended enrollment months or years ago, yet are still listed as “active, not recruiting” with estimated primary completion dates in 2022 or 2023. The majority of these studies, launched before vaccines or treatments such as Paxlovid became widely available, have yet to release any results, and have thus offered little value in informing treatment decisions during the pandemic. Most have postponed their estimated primary completion dates at least once after stopping recruitment, in some cases by more than a year at a time.
Confusion about when or whether a trial has ended
One study for which there is an unexpectedly large gap between recruitment ending and the estimated primary completion date is a Phase 2 trial of hydroxychloroquine and azithromycin launched in April 2020 and sponsored by Rutgers University. (The NIH has long since recommended against the use of “hydroxychloroquine and/or azithromycin for the treatment of COVID-19.”)
This study, which was designed to follow patients for just six days for its primary outcome measures, stopped recruiting in February 2021 and had an estimated primary completion date of April that year, according to its trial record. However, in September last year, that date was pushed back to June 2022 (where it remains at the time of this story’s publication), giving it an official reporting deadline of June 2023. The trial record doesn’t yet include actual enrollment numbers.
Rutgers’s Sabiha Hussain, the trial’s principal investigator, did not respond to multiple requests for comment about the dates listed in her trial record. However, Rutgers spokesperson Michele Fisher explained in an email earlier this summer that “the info in the trial record is accurate and current. The trial is not recruiting at this time. Data is being analyzed.”
After The Scientist followed up, quoting the NIH definition of the primary completion date and asking when the last patient was seen for the purposes of collecting primary outcome data, Fisher replied “5/18/2020,” noting that “all data have been collected for the primary endpoint. All data are being analyzed—and there are no results available yet.”
Verification of information such as when primary data collection ended is typically only possible by confirming with trial organizers themselves.
Boston Medical Center, meanwhile, is sponsoring a Phase 2 trial launched in early 2020 to investigate the efficacy of the chemotherapy drug etoposide in COVID-19 patients. The trial record, which lists a follow-up time of an average of 45 days for the primary outcome, stopped recruitment in April 2021, according to its record. At that point, the estimated primary completion date was given as December 2021. In January of this year, it was bumped back to June 2022.
The study’s principal investigator, Mark Sloan, told The Scientist via email in May that, “We stopped recruiting when vaccines became available and it became clear that the trial wasn’t the future of COVID therapy. It isn’t officially closed because we are still working on the manuscript (in slow motion).” Sloan didn’t respond to follow-up questions about whether researchers were adhering to the NIH definition of the primary completion date. The record, which was updated last month to include its terminated status, now lists an actual primary completion date of July 2022.
Inaccuracies acknowledged and corrected (sometimes)
In some cases, investigators or their institutions acknowledged that the listed primary completion date was inaccurate. Johns Hopkins’s David Sullivan, who oversaw a Phase 2 trial of convalescent plasma as a treatment for COVID-19, told The Scientist over email in May that his study met its “primary endpoint completion date [on] Nov 1” last year. The final patient received an intervention on October 1, 2021, he explained; the last patient visit took place around January 14.
Asked why the trial record still showed an estimated primary completion date of December 2022, Sullivan replied: “Estimated.” He did not respond to further requests for clarification. However, after The Scientist contacted staff overseeing clinical trial management at Johns Hopkins, the record was updated to show the actual primary completion date as January 2022, when the last patient visit took place. A preprint describing the study’s results was posted on medRxiv last December.
Johns Hopkins’s Keyes explains that the inaccurate date was the result of a problem with updating the record through the registry’s online data-entry system, which only accepts submissions that have filled out all the required fields completely. He adds: “We believe that ClinicalTrials.gov should always be an accurate and up-to-date reflection of what is happening in the trial.”
David Boulware, an infectious diseases researcher and physician at the University of Minnesota, also acknowledged an apparent inaccuracy in the estimated primary completion date of a trial he is involved in. The Phase 3 COVID-OUT trial, which aimed to test fluvoxamine, among other drugs, as an early COVID-19 treatment, finished recruiting patients in February and had a 14-day follow-up for its primary outcome. The fluvoxamine arm of that study was stopped for futility earlier this year.
Yet the trial record, which was last updated when recruitment was completed, lists an estimated primary completion date of February 2023. Boulware told The Scientist in May this year that he thought the primary completion date should in fact be February 2022, and that he would pass the information on. (As of publication, the record has yet to be updated.)
In some other cases, The Scientist was unable to ascertain what had happened to a study. Memorial Sloan Kettering Cancer Center’s Phase 2 trial of tocilizumab, for example, stopped enrollment in November 2020 after recruiting just 9 patients, each of whom would be followed for 14 days, according to its record. The primary completion date was postponed twice: initially to May 2022, and then, earlier this year, to May 2023—a date that would give it an official one-year reporting deadline in early 2024. Neither the trial organizer nor the institution’s press office responded to requests for comment.
Who is responsible for clinical trial information?
Verification of information such as when primary data collection ended is typically only possible by confirming with trial organizers themselves, and our analysis could not estimate how frequently estimated and actual primary completion dates are misreported across the database. NLM is clear in a disclaimer on its website about its inability to check trial records beyond basic quality control for obvious errors.
But Mayo-Wilson says that it’s noteworthy that some of the examples uncovered by The Scientist concern studies based at large institutions that have experience running clinical research. “I am surprised that . . . major universities that should have a better handle on these things are making that mistake.”
Whether or not trial organizers face consequences in the form of fines or other penalties for diverging from US clinical reporting guidelines depends on the sort of trial being run, who is funding it, and whether or not agencies such as the FDA decide that there has been a compliance problem. Early-stage trials, including Phase 1 studies, are typically exempt from legal oversight and instead submit information voluntarily, for example.
In an email to The Scientist, FDA spokesperson Chanapa Tantibanchachai highlighted various reasons that a study might legitimately push back its estimated primary completion date, such as challenges in enrolling participants, difficulties in obtaining the investigational product being tested, or unanticipated safety issues. (While some of the trials examined did seem to have trouble recruiting participants, none of the researchers who responded to The Scientist cited these kinds of reasons to explain the dates in their records.) And even though an incorrect primary completion date can delay the formal deadline for summary results, it doesn’t necessarily follow that researchers who enter the wrong date will report their results late.
There’s really no way for [NLM] to monitor every trial for honesty.
—Evan Mayo-Wilson, UNC Gillings School of Global Public Health
Anna Fine, acting director of ClinicalTrials.gov, declined to discuss individual studies, but writes in an email that at least some of the responses received by The Scientist “suggest a misunderstanding of ‘Primary Completion Date.’”
She adds that the NIH acknowledged special challenges for trial organizers during the pandemic that might make it difficult for them to meet ClinicalTrial.gov deadlines—although the agency’s guidelines also state: “We expect clinical trial information to be updated or corrected as soon as any organization or staff-related delays are resolved and recommend that sponsors and investigators retain documentation that would allow for determination of the appropriateness of the delay.”
Mayo-Wilson says that some confusion around using NLM’s trial registry is inevitable. Many researchers do a maximum of one clinical trial during their careers, he notes. “It’s unrealistic to think that everybody who is using this system is going to understand the legal definitions of every term that’s in it.”
However, institutions could “invest more in monitoring and taking responsibility for the trials that they and their faculty are conducting,” he says, adding: “Some people tasked with managing ClinicalTrials.gov records might not understand their legal, scientific, and ethical responsibilities. Because it is difficult to train all individual investigators who are or might be involved in trials, large research organizations should have professional staff with relevant expertise to oversee their trial records.”
Such institutions, which can oversee hundreds or thousands of records, are increasingly moving in that direction, notes Keyes, adding that staff at Johns Hopkins are available to help faculty with clinical trial reporting from registration through to the results-posting stages. He and other staff particularly try to discourage a “kick the can down the road” approach whereby trial organizers delay updating records or reporting study findings due to a lack of time, reluctance to post results before peer review, comprehension problems, or other reasons.
In response to The Scientist’s findings, Fine says that the NLM will also work to tackle miscomprehension of trial dates. “We’ll consider ways to provide and disseminate clarification on the definition” of the primary completion date, she says.
ClinicalTrials.gov “remains one of the best, most-comprehensive records of what trials are happening in the world,” says Mayo-Wilson. But “whenever you’re working with a large database, you need to think about the quality of the data,” he adds. “There’s really no way for [NLM] to monitor every trial for honesty. I think we have to rely on people’s integrity.”