Data Diving

Doshi’s group arrived at this conclusion because they’d run into a lack of available data. Some of the widespread belief that Tamiflu could blunt pneumonia and other dangerous health consequences of flu was based on a meta-analysis of several clinical trials whose results had never been published. Because the data could not stand up to independent scrutiny by the researchers, these trials were tossed out of the Cochrane review; other published trials were disqualified because of possible bias or lack of information.

Just as the 2009 BMJ paper was to be published, Roche, the maker of Tamiflu, opted to do something unorthodox—the company agreed to hand over full clinical study reports of 10 trials, eight of which had not been published, so that independent researchers could do a proper analysis. Within a few weeks after the publication of its review, the Cochrane team was downloading thousands of pages of study files.

One publication of a Tamiflu trial was seven pages long. The corresponding clinical study report was 8,545 pages.

Clinical study reports are massive compilations of trial documents used by regulators to make approval decisions. Doshi says he had never heard of, let alone worked with, a clinical study report. “This is how in the dark most researchers are on the forms of data there are. Most people think if you want to know what happened in a trial, you look in the New England Journal of Medicine or JAMA.”

And in fact, that is how many meta-analyses or systematic reviews of drugs are done. As publications amass, independent analysts gather up the results and publish their own findings. At times they might include unpublished results offered by the trial investigators, from the US Food and Drug Administration’s website, or from conference abstracts or other “grey literature,” but for the most part, they rely simply on publications in peer-reviewed journals. Such reviews are valuable to clinicians and health agencies for recommending treatment. But as several recent studies illustrate, they can be grossly limited and misleading.

Doshi and his colleagues began poring over the reams of information from Roche, and realized that not only had their own previous reviews of Tamiflu relied on an extremely condensed fraction of the information, but that what was missing was actually important. For instance, they found that there was no standard definition of pneumonia, says Tom Jefferson of the Cochrane Collaboration and lead author of the 2009 review. And among people who had been infected with influenza, it appeared that the placebo and treatment groups were not on equal footing. “We realized that all of these [analyses] led to misleading results because the treatment groups [were] not comparable for that subpopulation,” Doshi says.

In January of this year, the group published its latest review of Tamiflu, which included the unpublished evidence obtained from Roche in 2009.[2. T. Jefferson et al., “Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children,” Cochrane Database of Systematic Reviews, Issue 1, 2012.] The authors concluded that Tamiflu falls short of claims—not just that it ameliorates flu complications, but also that the drug reduces the transmission of influenza. In an e-mail sent to The Scientist, Roche says the Cochrane review was not limited to people who had laboratory-confirmed flu, but encompassed people with influenza-like symptoms, thereby possibly underestimating Tamiflu’s efficacy. “Independent and eminent scientists reviewed data from the Tamiflu trials, the inception and design of the studies which produced the data, and the assumptions made,” the company states. “Roche stands behind the robustness and integrity of our data supporting the efficacy and safety of Tamiflu.”

Jefferson is not convinced, and the experience has made him rethink his approach to systematic review, the Cochrane method of evaluating drugs. For 20 years, he has relied on medical journals for evidence, but now he’s aware of an entire world of data that never sees the light of publication. “I have an evidence crisis,” he says. “I’m not sure what to make of what I see in journals.” He offers an example: one publication of a Tamiflu trial was seven pages long. The corresponding clinical study report was 8,545 pages.

“It just blows the mind,” says Doshi. “A trial’s an extraordinarily complex process, and what we see in the published literature is an extreme synthesis of what goes on.” The big question is: What does that mean for the validity of independent reviews?

PUSHART

Unpublished data—Is it all bad news?

Clinical study reports like those provided by Roche are the most comprehensive descriptions of trials’ methodology and results, says Doshi. They include details that might not make it into a published paper, such as the composition of the placebo used, the original protocol and any deviations from it, and descriptions of all the measures that were collected.

But even clinical study reports include some level of synthesis. At the finest level of resolution are the raw, unabridged, patient-level data. Getting access to either set of results, outside of being trial sponsors or drug regulators, is a rarity. Robert Gibbons, the director of the Center for Health Statistics at the University of Chicago, had never seen a reanalysis of raw data by an independent team until a few years ago, when he himself was staring at the full results from Eli Lilly’s clinical trials of the blockbuster antidepressant Prozac.

FDA, time, Gibbons had questioned the belief that antidepressants are linked to an increased risk of suicide. Previous meta-analyses by independent reviewers on suicidal thoughts and behaviors among people taking the drugs had for the most part relied on summary data, Gibbons says. At a meeting at the Institute of Medicine a few years ago, Gibbons spoke with a senior investigator at Eli Lilly and brought up the idea of doing a full workup of the original data.

If there is some lid put on some aspects of those trials, that is frustrating one important goal of research, which is sharing information.
—SIDNEY WOLFE, PUBLIC CITIZEN

Much to his surprise, shortly after the meeting Gibbons was in possession of the numbers. “We haven’t seen anybody get these kinds of data,” he says. He decided to push his luck. Gibbons had served as an expert witness for Wyeth, and he approached attorneys for the pharmaceutical company to ask if they would also share data from trials of the company’s antidepressant Effexor. “They got back to me, and they were agreeable to provide all their adult data,” he recalls.

Gibbons and his colleagues went to work reanalyzing the data. “Everything was exquisitely well documented,” he says. The raw data allowed them to take into account each person’s depression severity and to determine individual outcomes rather than averages. Their results, published earlier this year, ended up bucking much of the published literature on antidepressants.[3. R.D. Gibbons et al., “Suicidal thoughts and behavior with antidepressant treatment: Reanalysis of the randomized placebo-controlled studies of fluoxetine and venlafaxine,” Archives of General Psychiatry, online February 6, 2012.]^,[4. R.D. Gibbons et al., “Benefits from antidepressants: Synthesis of 6-week patient-level outcomes from double-blind placebo-controlled randomized trials of fluoxetine and venlafaxine,” Archives of General Psychiatry, online March 5, 2012.] For one, they found no link between Prozac and suicide risk among children and young adults. And secondly, they found that Prozac appeared to be more effective in youth, and antidepressants far less efficacious in the elderly, than previously thought. “I think these kinds of analyses and the discrepancies in the findings are good reason to be concerned about our reliance on traditional meta-analyses,” Gibbons says.

Although some of his results reflect negatively on the drugs, others are clearly very positive. There’s been an understanding for some time that publication bias is a real occurrence, and that it often favors the drug. Trials that show no efficacy are less likely to get into print than trials that demonstrate a positive effect.[5. K. Lee et al., “Publication of clinical trials supporting successful new drug applications: A literature analysis,” PLoS Medicine, September 2008.] So when Lisa Bero at the University of California, San Francisco, decided to redo 42 published meta-analyses of drugs and include unpublished, but available, data, she suspected the drugs would fare poorly. “But that’s not what we found,” she says.

She and her colleagues analyzed nine drugs using unpublished data from the FDA. For any approved drug, the agency makes available a summary of data used to vet the medication. When Bero’s group added these data to the meta-analyses, they found that all but three turned out to have a different result. Nineteen of the redone analyses showed a drug to be more efficacious, while 19 found a drug to be less efficacious.[6. B. Hart et al., “Effect of reporting bias on meta-analyses of drug trials: Reanalysis of meta-analyses,” BMJ, 344:d7202, 2012.] The one harm analysis that was reanalyzed showed more harm from the drug than had been reported. “We showed data that make a difference are not being published,” Bero says.

While the FDA’s summaries of trial data are available to any researcher, they’re not necessarily easy to work with, and often researchers don’t include them in meta-analyses. “I think the FDA reports are an extremely valuable data source, but they’re not the full application [for drug approval], and they have redacted parts,” Bero says. She’s found that potentially important elements, such as patient characteristics or conflict-of-interest information, have been blacked out. The quality of the PDFs can also be poor, with crooked pages or light print; and sometimes there is no index for a document hundreds of pages long.

Such data files are quite different from the quality of the documents Gibbons was able to work with. While he urges independent researchers to try to access raw data, he notes that “getting all the data is not a trivial problem.”

Why aren’t the data shared?

Although summaries of clinical trials are available from the FDA, unabridged clinical study reports or the raw data are hard to come by. Keri McGrath Happe, the communications manager at Lilly Bio-Medicines, wrote in an e-mail to The Scientist that the company has a committee that reviews requests to obtain unpublished clinical trial results. “I can tell you that it is not common” to have a request filled for raw data, she says. “Granting access to raw data isn’t as easy as opening file cabinets and handing over documents. A team has to go through each piece of data to find what specific data [are] needed to fulfill the request.”

f being an administrative burden, handing over clinical reports or raw data is considered hazardous to the integrity of a drug’s worth. “The simple truth is that drug discovery is enormously expensive,” says Jeff Francer, the assistant general counsel of the Pharmaceutical Research and Manufacturers of America (PhRMA). “In order for companies to engage in the immensely capital-intensive work to develop a medicine, there has to be some protection of the intellectual property. And the intellectual property is the trial data.”

The FDA tends to concur. The agency receives much more information about a drug than it ever releases. According to Patricia El-Hinnawy, an FDA public affairs officer, “as a matter of law and regulation, patient-level clinical trial data has been historically regarded as confidential commercial and/or trade secret information.”

The other route to obtaining unpublished results is through a Freedom of Information Act (FOIA) request, but just as with putting in a request to a company, there is no guarantee that the information will be released. Plus, “FOIA requests take a long time,” says Michelle Mello, a professor of law and public health at the Harvard School of Public Health. “In a world where we’re concerned about being able to rapidly assess certain safety signals, this is not a route to producing timely information.”

Gibbons says his studies on antidepressants make a strong case for greater data sharing. The other argument, says Sidney Wolfe, director of the health research group at the advocacy organization Public Citizen, is that “it’s a moral and ethical thing too. People who are participating in clinical trials, aside from whatever possible benefit will happen to them . . . are doing it for the benefit of humanity. And if there is some lid put on some aspects of those trials, that is frustrating one important goal of research, which is sharing information.”

The question of whether results from human experiments are private information or a public good has been debated for some time. In 2010, the European Medicines Agency (EMA), the European Union’s equivalent of the FDA, finally made a decision. “We had resolved that clinical data is not commercial confidential,” says Hans-Georg Eichler, the EMA’s senior medical officer. “It doesn’t belong to a company, it belongs to the patient who was in the trial.”

PUSHART

Efforts to increase data sharing

The EMA’s new policy is that if someone requests data from clinical trials of an approved medication, the agency will provide it. Doshi’s group took advantage of this to obtain about 25,000 pages of information on Tamiflu, which they used for their 2012 Cochrane update.²

Eichler says there have only been a handful of requests to date, too few to know how the policy is working out. Fulfilling such requests can be cumbersome, he says. It takes time to carefully review the data and make sure patients cannot be identified. Eichler adds that in the future he’d like to see a system where all clinical trial results are entered into a system accessible by other researchers.

Under the FDA Amendments Act of 2007, the agency requires trial sponsors to post the summary results of registered trials on clinicaltrials.gov within one year of completing the trial. But few comply. A recent survey of the website found that of 738 trials that should have fallen within the mandate, just 163 had reported their results.[7. A.P. Prayle et al., “Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: Cross sectional study,” BMJ, 344:d7373, 2012.] In a statement sent to The Scientist, Congressman Henry Waxman (D-CA) says, “I was alarmed by the recent studies showing that compliance with this law has been sorely lacking and that industry is not reporting the required study results.”

While companies are certainly part of the problem in this case, they were actually more likely to report results than were researchers whose clinical trials had no industry backing, but were funded by foundation or government money. “I think it’s so important to acknowledge that is a huge problem throughout” the clinical research enterprise, says Kenneth Getz, a professor at Tufts Center for the Study of Drug Development. And industry has made some moves to be more proactive about sharing data.

Last year, the medical device company Medtronic agreed to share all of its original data regarding Infuse, a bone growth product that had been facing considerable skepticism about its efficacy. Yale professor Harlan Krumholz approached the company with a challenge: if Medtronic thinks the Infuse data can stand up to external scrutiny, then let an external group have a look. The company agreed, and a Yale University group serves as the middleman between the company and the independent reviewers.

Joseph Ross, a Yale Medical School professor who’s involved in the project, says two review teams have been selected, and they should have results by the summer. Medtronic is paying $2.5 million for the external reviews, a price Ross says is small compared to what gets invested in—and ultimately earned from—a successful drug. He says it’s the first experiment of its kind. “In my most optimistic moments I think it has to be the way of the future. I don’t think the public realized that this data isn’t available for everybody to understand,” says Ross. “In my most pessimistic moments, this only happens one other time—when a company gets in hot water.”

Journals are also lighting a fire under trial sponsors to provide their results to independent reviewers more quickly and completely. In 2005, the International Committee of Medical Journal Editors initiated a requirement that trials had to be registered, say on clinicaltrials.gov, in order to be published. “That sent shock waves,” says Elizabeth Loder, an editor at the British Medical Journal.

Since then, Loder’s own publication has been digging into the effects of unpublished data. She says the BMJ asks independent reviewers and meta-analysts to what extent they tried to obtain unpublished results for their studies. And this January, the journal published a special issue of reports dedicated to “missing” clinical trial data.[8. R. Lehman and E. Loder, “Missing clinical trial data,” BMJ, 344:d8158, 2012.] “I suppose you could say that publishing the original [2009] report on Tamiflu, we were newly sensitized to the dangers,” says Loder. “I think we wanted to keep everybody focused on that problem.”

For a special issue next year, Loder says BMJ is going to look at what exactly is the harm of having used incomplete data sets for so many meta-analyses and systematic reviews over the years. “Even though going forward new requirements for posting study results will probably improve the situation, we remain concerned about previously done studies that are unpublished and unavailable, and how that might affect the existing evidence base.”

While Getz agrees that more data could improve meta-analyses, he cautions against “data dumping”—completely opening the floodgates to unpublished results. “I think just the idea of making more information available misses the point. You reach a level of data overload that makes it very hard to draw meaningful and reasonable conclusions, unless you’re an expert with a lot of time.”

But Cochrane Collaboration’s Jefferson says bring it on. While the clinical study reports he received numbered in the thousands of pages, they were still incomplete. Roche says it provided as much as the researchers needed to answer their study questions. But accepting that response would require a trust that is clearly eroded. “We hold in the region of 30,000 pages. That’s not a lot,” Jefferson says. “We don’t know what the total is. We’re looking at the tip of the iceberg and we don’t know what’s below the waterline.”