The nationwide experiment will initially include around 100,000 volunteers.
Why are there so many contradictory nutrition studies, and how can they be improved?
June 1, 2014|
© STUDIOM1/ISTOCKPHOTO.COMThe media commonly reports on what appear to be shocking contradictions and reversals in studies of diet and health. One day oat bran works for lowering cholesterol, the next day it doesn’t. One day the perils of butter drive up sales of margarines, until the risks of trans fats in margarine swing sales back in the other direction. For decades “low-fat” was the public-health mantra for weight loss, until a series of studies seemed to indicate “low-carb” was at least as effective. In truth, the findings from these studies are rarely as diametrically opposed to one another as the media portrays in a bid to capture the attention of their audience.
The recipe for creating controversy is built into the complexities linking food and nutrition to health and disease. The main ingredients include the target population, the type of diet or food or nutrient of interest, the dose, the outcome, and the duration. The general public wants to know whether “oat bran” is “healthy” to eat. For the nutrition scientist, that practical question is too broad to be answered effectively. If a study on the topic is to be effectively designed, proposed, funded, conducted, and published, it would first go through an almost torturous process of trying to make the question “answerable,” and this would involve narrowing the question to a set of specifics.
Ultimately and ideally, the general question about the healthfulness of oat bran would take into account all possible combinations of target population, type of diet/food/nutrient, dose, duration, and outcome. But no single study could answer all those questions simultaneously. In 1990, two research studies, published nine months apart, reported apparently contradictory conclusions (N Engl J Med, 322:147-52, 1990; Am J Clin Nutr, 52:495-99, 1990). The later study reported that oat bran lowered blood cholesterol; the earlier one reported that it did not. The study reporting a benefit was conducted in an older population with elevated blood cholesterol and room for improvement. The study reporting no benefit was conducted in a younger, healthier population with normal blood cholesterol levels and limited room for improvement.
Contradictory? No. The two target populations were different. When combined, do the two studies address all the possible combinations mentioned above? No. Do the studies address health? Yes, but only a small slice of the oat-bran-effects-on-health pie—far from the whole enchilada. Controversial? Sure, if you want them to be, because they reach opposite conclusions about whether eating oat bran lowers blood cholesterol. But truly controversial? No, these just represent two different pieces that both fit into a larger puzzle.
The findings from nutrition studies are rarely as diametrically opposed to one another as the media portrays. The recipe for creating controversy is built into the complexities linking food and nutrition to health and disease.
For most nutrition topics, there are several types of study designs that can be used to generate useful evidence for presence or absence of benefit. Each type of study design has multiple components with options that range from more to less ideal. Comprehensive coverage of this area is beyond the scope of this discussion, but several of the more important sources of variability are described below.
Type of study. Observational studies involve simply observing the participants, their dietary habits, and their health status. When dietary components are linked to health outcomes in observational studies, the conclusions are always framed as “associations,” as opposed to “cause and effect.” Intervention studies are used to establish cause and effect. Knowing about cause and effect is more ideal than knowing about associations, but observational study is often the only realistic option.
Participant adherence. A central component of intervention studies is that participants must change their diet in order to determine if this causes a subsequent change in a health outcome. Ideally, a participant would keep everything else in his life constant and change just one nutrition component of interest. When the component is a single nutrient, like a vitamin or mineral, this can be achieved fairly easily using a dietary supplement and a placebo. However, if the dietary component of interest is food (e.g., garlic, soy, cruciferous vegetables, nuts), or more complicated yet, a dietary pattern (e.g., Paleo, vegetarian, Mediterranean), adherence becomes critical. A common problem in this area of research is low or only partial adherence to the particular food or diet pattern being tested.
Assessment of adherence. The typical approach to assessing adherence has multiple sources of error. Study participants, when asked to report what they ate during a particular time period, often provide answers that include some combination of overestimation, underestimation, forgetfulness, and lies. Another common source of error arises from the frequent occurrence that the collected nutrition data, while generally accurate for a particular day, represents an atypical food day for the participant. A third source of error occurs when the reported dietary intake of the study participants is converted to an estimation of grams, milligrams, and micrograms of nutrients by a database based on the same type of food reported by the participant, but not the same actual food.
Study outcome. The choice of the outcome being measured—blood cholesterol, dental caries, life or death, for instance—is critical to the conduct of an ideal study. Some health outcomes can be measured more easily and accurately than others. Whichever study outcome is selected, there will be a corresponding ideal number of enrollees and an ideal amount of time to track them. Unfortunately, many studies are conducted with outcomes that are difficult to measure at all or to measure accurately, with sample sizes that are too small, and durations that are too short. These issues often arise due to funding limitations.
“Ideal” studies are rare, since pursuing the ideal leads to choosing among competing priorities. When we prioritize generalizability, we lose rigor and control. When we prioritize the most accurate measurement, we lose affordability. When we seek ideal participants, we often lose generalizable individuals. Being aware of these trade-offs, we can seek to design the most ideal study possible.
While the most important and complex nutrition questions will likely remain dauntingly problematic to resolve for many years to come, technological advances could address some of the challenges of designing better studies, particularly in the areas of diet adherence, assessment of adherence, and cost effectiveness.
Recruiting participants who understand and adhere to the study guidelines may be the greatest challenge of all. Typically, study staff need to explain the dietary guidelines at the outset, and then continue to field follow-up questions or provide further clarification. Much of this is repetitive, since many participants have similar questions or have similar problems with adherence. Internet-based technology can help in several ways that can decrease both the workload and the number of staff required to run a study: Make the answers to frequently asked questions available quickly and easily through a Web-based platform accessible by computer or smartphone; make study staff more accessible through e-mail or Web-based appointments and interactions; and use technology as a part of the intervention. An example might be exploring how diet reminders could be initiated through smartphone programming.
Assessment of diet adherence is another challenge that may be tackled with technology. Using a smartphone to take photos of eating episodes could improve the accuracy of reported food consumption by having study participants review their photos while reporting on their dietary intake to study staff. There have even been reports of attempts to use photographs of food to automatically estimate caloric and nutrient content of the items in the photo, using algorithms for calculations.
The latest technology can also benefit nutrition research by making such studies more cost-effective. A limiting factor in nutrition studies is often sample size. Recruiting, enrolling, intervening, and following up with large numbers of study participants typically require large staff sizes, which can be prohibitively expensive. Online screening tools have already helped investigators decrease the staff time necessary to recruit and enroll participants (e.g., Survey Monkey, Qualtrics). Web portals allow participants to schedule themselves for various study requirements (e.g., clinic visits, labs) and to fill out questionnaires online, rather than using paper copies that require subsequent data entry. Funds from these staff savings can be redirected to recruiting larger sample sizes, which then have the capacity to generate results that are more conclusive.
There are likely many other ways that technological advances could help to improve the design and process of nutrition studies, and it is clear that such advances have already had an impact. These approaches will likely continue to improve dietary adherence and accuracy in the assessment of adherence by study participants. These advances, combined with larger sample sizes made possible by greater cost-effectiveness, will allow for more-definitive and less controversial findings. These are confusing but also exciting times in the field of nutrition research.g
Christopher Gardner is Director of Nutrition Studies at the Stanford Prevention Research Center and a professor of medicine at Stanford University. He is in the process of developing a new interdisciplinary Stanford Food Systems Initiative. Michael Stanton is a postdoctoral fellow at the War Related Illness and Injury Study Center (WRIISC), VA Palo Alto Health Care System, and at Stanford University School of Medicine. A clinical psychologist specializing in behavioral medicine, he studies the development of compulsive eating and complementary medicine–based interventions for obesity and diabetes.
June 5, 2014
Population studies aren't accurate.
Nutrition studies should be concerned about nutrients rather than foods that contain multiple nutrients.
Animal studies have shown that saturated fat causes CVD and diabetes, but the palm oil and coconut oil producers can find all sorts of ways to confuse the issue.
June 5, 2014
While Dr. Gardner is a respected epidemiologist (I've not heard of Dr. Stanton), his focus on study design misses the central point - humans are the most adaptable land animal on the planet. Since we can differ genetically and epigenetically (within a generation) and have a malleable gastrointestinal microbiome there is no dietary standard with which we can compare healthy to non-healthy foods. Even in a population of CVD patients, one or two will be dietarily resistant to trans fat elevations in LDL cholesterol. Humans make crappy subjects!
Secondly, because mammals do silence their genomes over their lifetime, and can and DO change their GI bacterial population based on diet, when addressing whether a dietary intervention has a biologic effect (e.g., oat bran), we may never know who/what the responsible party is for a statistical change: the person's DNA, some other life behavior (e.g., supplement X), subtle silencing of important genes, alterations of gut flora (due to oat bran fiber?), etc.
Third, and I do give credit to Dr. Gardner for mentioning this, most dietary intervention studies ultimately report VERY MINOR DIFFERENCES in outcome. Yes, they may be scientifically significant using statistics, but in most cases they would have a subtle impact on long term health. For example, even if oat bran caused a 5% reduction in serum LDL - following a rigorously adhered to meal plan, in patients with established CVD what effect would that have on mortality? Could it even be measured?
As a former nutrition scientist (and now a nutrition professor) I would argue that most studies are done fairly well - with as much control of confounding factors as science and money can afford. It's just that once you start using a wacka-do diversity of subjects (even if they all look the same on the outside but are completely different on the inside) you end up with a large variation of outcome.
It's not the study design.
It's the subjects!
June 5, 2014
good observation about the 'image' problem that nutritional research outcomes are having with the public; cynicism just about summarises how results and advisories are largely viewed by the public.
more troubling is the unpleasant arguments that can be generated at dinner table amongst family members ( including children) when each is flagging different conclusions on same diets, fuelled by seeemingly contradictory results from published dietary/ nutritional studies.
the 'way forward' suggestions by these authors are worthy of consideration but inspite of the best intentions and strategies, nutritional studies will remain mired in a ' catch 22' dilemma, be these observational or interventional;
To have epidemiological impact or generalisability, the studies must be rich in numbers with large sample sizes and controls; the larger the number and thus the statistical power, the more difficult it becomes to guarantee procedural rigor, and thus accuracy and predictive power, both on the part of the researchers and that of the enrollees.
under non- laboratory conditions, conformity and compliance are often so difficult to attain, the goals are compromised in various ways and the final surrogates published are often anything beyond statistical compromises.
when we then realise the additional temptations of some trialists to doctor figures to arrive at preconceived positions, should they for any reasons have some, the potential toxicities in nutritional studies can be well imagined.......
afterall if we torture data hard enough, it will confess to anything, including the absurd.. and how sadly often, the absurd gets published.
clinicians among us are daily faced with the practical problem of how to support our nutritional advices to patients with coherent and consistent research outcomes, undermined as we often are, by inconsistencies, worse still, irreconcilabilities ( apparent or real), in available data.
to have to wade through the nuances of a study as this article seems to suggest, before extricating it of its own interpretational strait jackets, can be too laborious in the context of daily clinical work and patients's understanding.
nevertheless, such studies remain necessary or a 'must do' if sense is to made of our foods and a reliable compass is to be fashioned.
June 5, 2014
I was at a school of public health for a long time and noticed that the publication of a contradictory epi study often was associated with a new doctoral candidate, that is, the same dataset gets looked at by a different person using different methods. Nothing wrong with reexamining an analysis...but perhaps we should not be sending out a press release every time we get a paper published. If you need to send out a press release, perhaps the study ain't worth much anyway. Good studies are recognized by our peers. Bad or trivial studies that are highlighted in the lay press just come back to bite us and diminish public perception of scientists as a whole.