When Should We Rely on Observational Studies?

How legitimate is the common corporate criticism of the scientific nutrition literature that the credibility of observational studies is questionable?

While randomized controlled trials are highly reliable in assessing interventions like drugs, they’re harder to do with diet. Dietary diseases can take decades to develop. People can’t eat placebo food, and it’s difficult to get them to stick to assigned diets, especially for the years it would take to observe effects on hard endpoints like cancer or heart disease. That’s why we have to use observational studies of large numbers of people and their diets over time to see which foods appear to be linked to which diseases. And interestingly, if you compare data obtained from observational population studies versus randomized trials, on average, there is little evidence for significant differences between the findings. The effects were not only in the same direction, but also of the same general magnitude in about 90% of the treatments they looked at. But what about the hormone replacement therapy disparity I talked about in the last blog, where it appeared that women taking Premarin had fewer heart attacks, yet randomized controlled trials showed the opposite? Looking at the data, it turns out the discrepancies were rooted in timing—e.g., when Premarin was started—and the studies actually showed the same results after all.

But even if observational trials did provide lower quality evidence, maybe we don’t need the same level of certainty when we’re telling someone to eat more broccoli or drink less soda, compared to when we’re deciding whether to prescribe a drug. After all, prescription drugs are the third leading cause of death in Europe and the United States. The first is heart disease, then cancer, then doctors. About 100,000 Americans are wiped out every year from the side effects of prescription drugs—when taken as directed! So, given the massive risks, there had better be rock-solid evidence that the benefits outweigh the risks. You’re playing with fire, so you’d better believe I want randomized, double-blind, placebo-controlled trials for drugs. But when you’re just telling people to cut down on donuts, you don’t need the same level of proof.

In the end, the sugar industry–funded paper concluding that the dietary guidelines telling people to cut down aren’t trustworthy because they’re based on “low-quality evidence” is an example of “the inappropriate use of the drug trial paradigm in nutrition research.” Some might say, but what were the authors supposed to do? If GRADE is the way you judge guidelines, then you can’t blame them. But there are other tools, like NutriGrade, a scoring system specifically designed to assess and judge the level of evidence in nutrition research. One of the things I like about NutriGrade is that it specifically takes funding bias into account, so industry-funded trials are downgraded. (It’s no wonder the industry-funded authors chose the inappropriate drug method instead.)

HEALM is another one. It stands for Hierarchies of Evidence Applied to Lifestyle Medicine, and it was specifically designed because existing tools such as GRADE are not feasible options when it comes to questions that can’t be fully addressed through randomized controlled trials. Each research method has its unique contribution, as you can see below and at 3:17 in my video: Observational Studies Show Similar Results to Randomized Controlled Trials.In a lab, you can explore the exact mechanisms; randomized controlled trials can prove cause and effect, and huge population studies can study hundreds of thousands of people at a time for decades.

Take the trans fat story, for example. There were randomized controlled trials showing trans fats increased risk factors for heart disease and population studies showing that the more trans fats people ate, the more heart disease they had. Together, these studies built a strong case for the harmful effects of trans fat consumption on heart disease, and, as a consequence, it was largely removed from the U.S. food supply, preventing as many as 200,000 heart attacks every year. Now, it’s true that we never had randomized controlled trials looking at hard endpoints, like heart attacks and death, because that would take years of randomizing people to eat something like canisters of Crisco every day. You can’t let the perfect be the enemy of the good when there are tens of thousands of lives at stake.

Public health officials have to work with the best available balance of evidence there is. Consider how tolerable upper limits for lead exposure or PCBs are decided. It’s not as if kids were randomized to drink different amounts of lead and were then followed so we could see who grew up to have tolerable brain damage. Those kinds of experiments can’t be conducted; so, evidence must be pulled from as many sources as possible to make the best approximation.

Even when there aren’t any randomized controlled trials on a particular topic or if they’re impossible to conduct, observational studies give us a lot of evidence on the nutritional causes of a lot of cancers, like red meat consumption increasing colorectal cancer risk. So, if dietary guidelines for the prevention of cancer were to be assessed with the drug-designed GRADE approach, the same conclusion as the sugar paper would be reached—the evidence is low quality. So, it’s no surprise a meat industry–funded institution hired the same guy who helped conceive and design the sugar industry–funded study. And unsurprisingly, he became the lead author of a paper saying we can ignore the dietary guidelines to reduce red and processed meat consumption because they used GRADE methods to rate the evidence, and though current dietary guidelines recommend limiting meat consumption, their results predictably demonstrated that the evidence was “of low quality.”

Before I dive deep into the meat papers, there’s one last irony about the sugar paper. The authors used the inconsistency of the exact recommendations across sugar guidelines over a 20-year period to raise concerns about the quality of the guidelines. Obviously, we would expect guidelines to evolve, but the most recent guidelines “show remarkable consistency,” with one exception: the 2002 Institute of Medicine guideline that said a quarter of our diet could be straight sugar without running into deficiencies. But that outlier was partly funded by an institute funded by Coke, PepsiCo, and cookie and candy companies—an institute that is now saying, See? Since recommendations are all over the place (thanks in part to us), they can’t be trusted.

Doctor’s Note

This is the third in an eight-part series on how industries impact dietary and health guidelines. The first two videos introduced what happened when the 2015 Dietary Guidelines committee recommended reducing sugar consumption: How Big Sugar Undermines Dietary Guidelines and How Big Sugar Manipulated the Science for Dietary Guidelines.

Next, we will look at the recent articles in the Annals of Internal Medicine on meat consumption; see related posts below.