Update (June 3): The World Health Organization today announced that it would be resuming hydroxychloroquine testing in its Solidarity Trial, following widespread criticism of The Lancet paper and the dataset it was based on.
Scientists are questioning the validity of two influential studies of COVID-19 patients as concerns grow about the provenance of the dataset underpinning them. Despite assurances from Surgisphere Corporation, an Illinois-based company that owns the data, hundreds of researchers have now signed open letters to The Lancet and the New England Journal of Medicine (NEJM), voicing their concerns. Today, NEJM and The Lancet issued expressions of concern about the reliability of the papers’ conclusions.
“Recently, substantive concerns have been raised about the quality of the information in that database,” writes NEJM editor-in-chief Eric Rubin in the expression of concern. “We have asked the authors to provide evidence that the data are reliable. In the interim and for the benefit of our readers, we are publishing this Expression of Concern about the reliability of their conclusions.”
A number of hospitals in New York, New Jersey, and Illinois tell The Scientist they did not contribute to Surgisphere’s dataset.
A statement by The Lancet editors reads: “Although an independent audit of the provenance and validity of the data has been commissioned by the authors not affiliated with Surgisphere and is ongoing, with results expected very shortly, we are issuing an Expression of Concern to alert readers to the fact that serious scientific questions have been brought to our attention. We will update this notice as soon as we have further information.”
Both studies relied on Surgisphere Corporation’s database, which the papers state contains COVID-19 patient data from hundreds of hospitals around the world. A third paper from Surgisphere, which described the effects of the drug ivermectin in COVID-19 patients, was posted as a preprint in April.
The studies’ findings have influenced research and policy, with the World Health Organization recently suspending testing of malaria drug hydroxychloroquine over safety concerns reported in the Lancet study. Regulatory agencies in the UK and in France have also suspended recruitment into clinical trials of the drug in COVID-19 patients.
The search for sources of Surgisphere’s COVID-19 data
Scientists are expressing doubts that Surgisphere Corporation could have collected so much detailed patient data in such a short period of time. The open letter to the NEJM requests that, “at the very minimum, [Surgisphere should share] the aggregated patient data at the hospital level (for all covariates and outcomes).”
The Lancet study reported that Surgisphere’s registry contained data from more than 63,000 COVID-19 patients admitted to 559 hospitals in North America by April 14. By that date, around 580,000 total cases had been reported in the United States, with New York and New Jersey accounting for nearly half of them.
Surgisphere CEO and founder Sapan Desai has so far declined to release the names of any hospitals involved in providing the data, citing pre-arranged privacy agreements.
The Scientist has reached out to some of the largest health systems in the states hit hardest by the coronavirus pandemic to inquire whether they participated, but could not find any that did.
Instead, a number of hospitals confirmed that they did not contribute data, namely, New Jersey health systems RJWBarnabas Health, Cooper Health, and Atlantic Health System; NYC Health + Hospitals and NYU Langone in New York; and Illinois-based health systems Rush and Advocate Health Care.
Northwest Community Hospital, which employed Desai as a vascular surgeon until February 10 this year, did not contribute data, it confirms in an email to The Scientist. Nor did coauthor Mendeep Mehra’s institution, Brigham and Women’s Hospital.
In an interview last week, Desai told The Scientist that he would ask if any hospitals were willing to come forward voluntarily to confirm their participation, but has not provided any names yet.
Desai continues to stand by Surgisphere’s registry, saying in a statement to The Scientist that “it is essential that the scientific and lay community alike understand the value—and legitimacy—of databases such as Surgisphere’s.”
A growing list of discrepancies
The Lancet study’s full dataset included 96,032 patients admitted to 671 hospitals across six continents by April 14. Of those patients, 10,698 had died in hospital by April 21, the study reported.
Readers have argued that the data presented in these papers don’t jibe with reported numbers of COVID-19 patients. For example, researchers at Australian institutions told the The Guardian last week (May 28) that the number of Australian deaths in the Lancet paper, counted up until April 21, exceeded the number of COVID-19 deaths recorded by health authorities up until that date.
Desai told the Guardian that this was due to an error that caused one hospital in Asia to be included in the Australian dataset, but didn’t provide any further detail about country-level or hospital-level data. The Lancet subsequently published a correction from the authors with the amended Australian data, and with an added table that included continent-level raw data, instead of the adjusted data Desai said had been shown previously.
I am confident Surgisphere is able to address the concerns of NEJM, The Lancet and the broader scientific community.—Sapan Desai, Surgisphere
Researchers also expressed doubts about the African data in interviews with The Scientist and in the open letter to the Lancet, noting that the quality of electronic health records on the continent makes it unlikely that Surgisphere Corporation could have obtained the records of 4,402 hospitalized COVID-19 patients in Africa by April 14, when only 15,738 cases had been reported.
Mehra, the medical director of Brigham and Women’s Hospital Heart and Vascular Center, told The Scientist in a statement yesterday that “the results, findings and overall interpretations reported in the study remain unchanged.” However, he adds that the coauthors “have initiated an independent academic review of the data” in the Lancet study.
Desai says in a statement sent via Liz DeForest of the public relations firm Bliss Integrated, that he and his coauthors have today initiated a “voluntary third-party audit of that paper in collaboration with The Lancet.”
In the meantime, concerns about a second study, also coauthored by Mehra, have come to the fore. An open letter to the NEJM co-organized by James Watson, a senior scientist at the Mahidol Oxford Tropical Medicine Research Unit in Thailand who also organized the open letter to the Lancet, today noted several discrepancies between Surgisphere’s database and national COVID-19 data.
For example, the NEJM study reports data from 346 hospitalized COVID-19 patients in Turkey by March 15. However, the letter states, “countrywide, the first COVID-19 case was diagnosed at Istanbul Faculty of Medicine on the 9th of March. The second COVID-19 patient in that hospital was not seen until the 16th of March. The Turkish Ministry of Health reported a total of only 191 PCR positive cases by the 18th of March.”
In response to the NEJM expression of concern—issued just a few hours after the open letter was posted—Desai says he is “arranging a non-disclosure agreement that will provide the authors of the NEJM paper with the data access requested by NEJM. I am confident Surgisphere is able to address the concerns of NEJM, The Lancet and the broader scientific community.”
Reporting contributed by Kerry Grens.
Clarification (June 4): This story has been updated to include Atlantic Health System among the health systems that did not contribute data to Surgisphere’s database.
More from The Scientist’s investigation into Surgisphere
The company behind a now-discredited study on hydroxychloroquine also posted a report that has been cited by Latin American governments recommending ivermectin as a possible coronavirus treatment. Clinicians there say the effects have been extremely damaging.
The company had helped develop a tool to aid decision-making in distributing limited medical equipment among coronavirus patients, but two high-profile retractions call into question the validity of Surgisphere’s work in toto.
All authors other than company founder and CEO Sapan Desai were “unable to complete an independent audit of the data,” The Lancet states.
Scientists have raised questions about the dataset published in The Lancet last week that triggered the suspension of clinical trials around the world—and about Surgisphere Corporation, the company behind the study.
A paper published in The Lancet reported that hospitalized COVID-19 patients taking the drug had a higher risk of death, although some researchers have raised questions about the data.