Implementing Change

Implementing Change Illustrations by Katrina StanleyCan better science save global health initiatives? By Bob Grant 1 It turns out that villagers living near the wells were drinking from them, but they were also using non-well water in sufficient quantities to keep the cholera rate high. Because monitoring and evaluation were not incorporated into the design of the program, says Glass, the intervention seemed to be missing its mark. Though he was not involved with the project, Glass says that systematically monitoring people's preferences for, or aversions to, well water throughout the implementation of the program could have increased the success of the program, and possibly saved lives. "Scientifically, we did the right thing," Glass says, "but we didn't understand people's behavior." Since that early failure, cholera rates in Bangladesh have indeed remained lower thanks in part to access to clean drinking water. (Unfortunately, no one was testing for arsenic when the wells were drilled, and the country now faces the unintended consequence of arsenic poisoning.) But on a global scale, public health programs have lacked the systematic evaluation of outcomes that seem necessary for success. Researchers observing public health interventions from the outside have, however, uncovered similarly disappointing results in the implementation of other programs in the developing world. For example, a program centered on the distribution of insecticide-laced bed nets to slow the spread of malaria in sub-Saharan Africa seems to have faltered initially because young children were not sleeping under the nets in great numbers. 2 Researchers didn't know that because they were conducting follow-up surveys only once every three to five years. Had they conducted annual studies, they might have been able to change the outcome more rapidly. Why have even well-intentioned public health programs given the evaluation of outcomes short shrift? The reasons are complex and touch on every aspect of health program planning, from the economics of fund allocation to ingrained modes of thinking about large-scale public health interventions. Public health program administrators "really don't like to talk about technical issues - whether the programs they're designing are working or not," says David Sack, a professor at Johns Hopkins Bloomberg School of Public Health. Sack says that administrators tend to place emphasis on the immediate amelioration of public health concerns and view tracking the impact of interventions as a secondary consideration. "If you call something 'research,' then it may seem irrelevant or an ivory tower type of thing," says Sack, who is also the director of the Center for Health and Population Research in Bangladesh. Then there's the aspect of devoting already scarce funding to evaluation in global health programs. Johanna Daily, an infectious disease doctor and professor at Harvard Medical School, says that thorough evaluations may cost less than purchasing drugs, hiring health workers, and other implementation costs, they still cost something. "When people are criticized for not measuring, you have to understand that they are making that decision," she says. "There's a cost to measurement that has to be balanced with rational decisions." Daily says that public health funding is so hard to come by in the developing world, that money is spent on treatments and interventions that are known or suspected to work, while funding for evaluation of the impact is often ignored. "There's so much work to be done and there's so little money," she says. Given the paucity of resources, says Daily, who also works on malaria projects with a Boston-based aid organization, Partners in Health, this disconnection is understandable. "If I had $100,000 I guess I would buy $100,000 worth of vaccines." Ruth Levine, a senior fellow at the Center for Global Development (CGD), says that program managers must change how they think about large public health programs. "Their fundamental reward system is not built around learning," Levine says, "and it's certainly not built around discovering that you've made a mistake." Levine, a health economist who has worked at the World Bank, says she has witnessed several programs whittle back money set aside for evaluation and redirect it towards implementation as interventions progressed. "On any given day, the incentive choices all drive toward getting the money out the door or getting the next incentive: getting the next drug or training the next teacher," she says. "There are lots of disincentives to doing rigorous evaluation." Levine, who cochaired CGD's Evaluation Gap Working Group, says that the public health community is awaking to the need for structured evaluation. To bring about wide changes, however, policy makers should devote and protect program resources with firewalls that prevent their "cannibalization." Making seemingly obvious changes to public health programs will enhance implementation science's effect, she says. Increasing the reach and utility of implementation science in public health programs can be achieved by increasing access to existing epidemiologic and demographic information, building the capacity of local researchers in developing nations to conduct new impact evaluations, and devoting more resources to impact evaluations. This investment in implementation science, says Levine, would lead to more rigorous random control evaluations, which would aid public health administrators and policy makers immensely. Emmanuela Gakidou, a Harvard researcher, was part of an international team that in 2006 exhaustively quantified the effects of sweeping Mexican healthcare reforms. 3 She agrees that health programs should be evaluated with every bit of the rigor that bench scientists use to approach basic biological questions. "We should hold health policy experiments to the same standards that we hold clinical trials," she says. This is especially crucial in the developing world, where unique cultural, infrastructural, and demographic challenges can doom well-intentioned public health interventions to failure, whereas a firm grasp of the outcomes of interventions can spell success. "Policy makers need to send a signal that says, 'We care about impact,'" Levine says. "How will we know if we're really seeing the impact we expect?" Glass asks of the public health community. "We spend a lot of effort to develop new tools, vaccines, and drugs, and then we wash our hands of them." Some programs have relied on rigorous evaluations for success. Onchocerciasis - known as river blindness because of its most extreme symptom - is the second leading infectious cause of blindness in the world. According to World Health Organization (WHO) statistics, the disease exists in more than 30 countries, with 99% of cases confined to tropical Africa. Onchocerciasis commonly strikes populations living near fast flowing African rivers where blackflies breed and transmit the parasitic filarial worm, Onchocerca volvulus, through their bites. The disease manifests itself gradually with itching, skin lesions, and eventually blindness as exposure to blackfly bites increases. A boy with river blindness in the Nuba mountains, Sudan. In 1974, WHO with the support of the World Bank, the United Nations Development Program, and several other donors, launched the Onchocerciasis Control Program (OCP) in West African countries hardest hit by the disease. At the OCP's outset, transmission of onchocerciasis could be controlled in only one way - aerial insecticide spraying of blackfly breeding areas where swarms of aquatic larvae metamorphosed into disease-harboring adults. Widespread larviciding seemed to be working well, but in the late 1970s, a new disease control tool emerged. Ivermectin, an antiparasitic drug, was tested in clinical trials throughout Africa and found to be effective in treating onchocerciasis. By the late 1980's, the OCP program had adopted ivermectin as another prong in its attack on onchocerciasis, and it began distributing the drug in the most onchocerciasis-ridden areas of West Africa, thanks to donations by pharmaceutical giant Merck. The OCP didn't just distribute the drug and hope that the desired effect would materialize. Instead, it enlisted the assistance of Dik Habbema, a public health researcher at Rotterdam's Erasmus Medical Center, to carefully track the effect of the drug on populations in the path of onchocerciasis. Habbema says that he originally collaborated with the OCP earlier, studying the longevity of the adult Onchocerca volvulus worm in the human body4 and the likelihood of the disease's recrudescence in core OCP areas where larviciding was reducing blackfly populations. 5 Left: A pharmacist sorts antiretroviral drugs at GHESKIO's Institute of Infectious Diseases and Reproductive Health. Inset: Jean Pape, cofounder and director of GHESKIO. Now Habbema and his colleagues were called in to model the transmission and control of the disease, given a variety of potential strategies. They conducted community-based drug trials in which infected and uninfected individuals were treated with ivermectin. They showed that administering the drug resulted in a 96%-99% drop in parasite loads in individuals with onchocerciasis, and that these loads rebounded more slowly in the population than had been shown in clinical trials. 6 With constantly refreshed data, support from the OCP's financial backers, and open communication between his research team and OCP administrators, Habbema was able to influence OCP policy decisions through evaluation of the program's impact concurrently with its implementation. "The continuation [of my collaboration with the OCP] was not planned at all," Habbema recalls, "but has been a consequence of mutual trust and satisfaction with how things were going." It's not hard to imagine why the OCP has been satisfied with Habbema's contribution to the program. In continuing the systematic evaluation of the program's administration, Habbema has affected substantive changes in ivermectin distribution strategies, which have helped to prevent 600,000 new cases of onchocerciasis, according to WHO estimates. The OCP ended in 2002, but Habbema's research continues to inform an African-based onchocerciasis control program, the African Program for Onchocerciasis Control (APOC), which started in 1995 to battle the disease in less severely afflicted countries outside the OCP's control area. "We use both data from the new program and scientific results from the old program," Habbema says. "We have combined them as much as possible for the APOC program." Pape's office sits at the end of a hallway. As I enter GHESKIO - the clinic, research, and training facility that he cofounded in 1982 - I step over the concrete threshold and walk through water. Tides regularly invade this hallway, leaving staffers to bail water from the floor as if trying to save a sinking boat. "The fact that it has worked here means it will work anywhere," Pape says. "I think it will be best exported to sub-Saharan Africa. One of the goals is to share experience." By the mid-1980s, though he had effected real change in the way infantile diarrhea plagued his country, neither Pape nor his countrymen realized the magnitude of the health crisis that was poised to befall Haiti next. They began seeing patients with a baffling constellation of symptoms, including rare opportunistic infections and tumors that were thought to affect only elderly men. With funding from US universities and international aid organizations, Pape cofounded The Haitian Study Group on Kaposi's Sarcoma and Opportunistic Infections (GHESKIO, which Haitians pronounce jess-kee-o). Early studies at GHESKIO sought to describe a new and frightening disease on the already precarious Haitian healthcare landscape. Pape and his colleagues at GHESKIO were the first to examine the clinical manifestations of HIV/AIDS in Haiti. 7 In 1985, he and his colleagues compared different doses and administration routes of a common drug for pneumonia in their efforts to curb skin reactions in a handful of GHESKIO's patients with AIDS.8 More implementation studies followed. One compared the efficacy of two drugs used to treat esophageal yeast infections commonly seen in GHESKIO's patients with AIDS, 9 while another evaluated different medication regimes for AIDS patients with severe diarrhea caused by a microbial infection. 10 Pape and his colleagues have been fine-tuning the ways that they administer care, treatment, and counseling to GHESKIO's patients. A constant flow of research from the center, utilizing the almost 150,000 patients GHESKIO sees per year, helps Pape and his colleagues to adapt GHESKIO policies to better serve the beleaguered Haitian populace. For example, recent GHESKIO research has explored the use of a simplified p24 antigen-capture ELISA as an alternative to using PCR (scarce in Haiti) to provide the facility's 30 or so rural extension clinics with a low-cost, low-tech method to test infants for HIV.11 The HIV epidemic in Haiti continues, though it appears to have abated. National prevalence peaked at 5% in 1994.12 According to current WHO estimates, the prevalence of HIV in Haiti is now 3.8%. That progress has come in the midst of chaos. GHESKIO has withstood a procession of transitional governments, two revolutions, and widespread disorder in the poorest nation in the Western Hemisphere in a guarded compounded girded by high walls topped with razor wire. Pape and his colleagues have been fine-tuning the ways that they administer care, treatment, and counseling to GHESKIO's patients. GHESKIO sits in the middle of one of Port-au-Prince's worst neighborhoods. Inhabitants of the city call the slum-fringed area "Kosovo." Just four years ago, the echoes of gunfire crackled off crippled buildings in Kosovo as acrid smoke wafted through the cacophony of another revolution. Kosovo is quieter now, but the chaos of revolution has been replaced with the white noise of a country teetering between collapse and rejuvenation. Smoke still swirls through the air, but now the fires cook Haitian street fare instead of razing buildings. A river of people flows through the wide expanse of Harry Truman Boulevard and fills the winding side alleys, but these days instead of rioting, people jam the streets to hawk goods such as phone cards and rubber steering-wheel covers. Wiry dogs stalk the crumbled sidewalks, and United Nations peacekeepers still patrol the area. The center's leader, dressed in his white lab coat, roams the crumbling hallways conversing with his staff and asking for input and advice. "This is what encourages me," Pape says of his staff, "the fact that we have a really excellent team. It's not me who created this model, it's all of us." Pape and his collaborators have kept this project alive through the tumult of Haiti's recent history by utilizing outside help from global health organizations. They've built an internal network to systematically monitor the success of health interventions and establish a robust link between evaluation and adaptive management to fit the unique challenges of delivering public health in Haiti. Hatians wait to be tested for HIV at GHESKIO's Institute of Infectious Diseases and Reproductive Health.left: Malcolm Linton/Getty Images "Jean Pape does that better than anyone else," says NIH's Glass. Pape has no plans of resting on his laurels, and there are a few, including a Legion d'Honneur award from the French government. Instead, he is thinking broadly about exporting the model for combining research, care, and training at GHESKIO beyond the chaotic mélange that surrounds his clinic. "The fact that it has worked here means it will work anywhere," Pape says. "I think it will be best exported to sub-Saharan Africa. One of the goals is to share experience." Though his path to success among the hardscrabble realities of living, doctoring, and studying in Haiti was a winding one, Pape, leaning back into his chair behind a desk filled with manuscripts to review and a computer screen choked with e-mails to answer, renders it down to a simple equation. "It's not easy," he says, "but you can obtain huge results if you concentrate, stay out of politics, and focus on your work." References 1. R. Levine et al., "Failure of sanitary wells to protect against cholera and other diarrhoeas in Bangladesh," Lancet, 2:86-9, 1976. 2. R. Monasch et al., "Child coverage with mosquito nets and malaria treatment from population-based surveys in African countries: a baseline for monitoring progress in roll-back malaria," Am J Tropical Med Hygiene, 71:232-8, 2004. 3. E. Gakidou et al., "Assessing the effect of the 2001-06 Mexican health reform: an interim report card," Lancet, 368:1920-35, 2006. 4. A. Plaisier et al., "The reproductive lifespan of Onchocerca volvulus in West African savanna," Acta Tropica, 48:271-84, 1991. 5. A. Plaisier et al., "The risk and dynamics of onchocerciasis recrudescence after cessation of vector control," Bull World Health Org, 69:169-78, 1991. 6. J. Remme et al., "Large-scale ivermectin distribution and its epidemiological consequences," Acta Leidensia, 59:177-91, 1990. 7. M. Boncy et al., "Acquired immunodeficiency in Haitians," N Engl J Med, 308:1419-20, 1983. 8. J. DeHovitz et al., "Cutaneous reactions to trimethoprim-sulfamethoxazole in Haitians," Ann Internal Med, 103:479-80, 1985. 9. M. Deschamps et al., "Treatment of candida esophagitis in AIDS patients," Am J Gastroenterol, 83:20-1, 1988. 10. J. Pape et al., "Treatment and prophylaxis of Isospora belli infection in patients with the acquired immunodeficiency syndrome," N Engl J Med, 320:1044-7, 1989. 11. E. George et al., "Potential of a simplified p24 assay for early diagnosis of infant human immunodeficiency virus type 1 infection in Haiti," J Clin Microbiol, 45:3416-8, 2007. 12. E. Gaillard et al., "Understanding the reasons for decline of HIV prevalence in Haiti," Sexually Transmit Infect, 82:i14-i20, 2006.

Implementing Change

Implementing Change

Can better science save global health initiatives?

Interested in reading more?

Become a Member of