How to Critically Analyze Nutritional Science

There is such a wide amount of wrong and misleading information about nutrition, that I wanted to discuss how one can sort the good from the bad. Here are some things to watch out for:

1. Consider the source.

YouTube, a blog, or a random website on a Google search are not reliable sources of information. Sure, you can find accurate information there, but there's also a massive glut of inaccurate information. Even accurate information is often taken out of context or left uncited.

A good source of info is PubMed, the National Institute of Health's scientific and medical journal indexing site. It has abstracts for a huge number of journals, and it has free full text for any research funded in part or whole by the US government via the NIH (although the journals may keep the article exclusive for 12 months from date of publication).

For general health and nutrition, Mayo is pretty good. They have a ton of web content and it's usually of high quality.

2. Consider the claims.

An extraordinary claim requires an extraordinary burden of proof. Some common themes to look out for - blaming complex problems on a single, simple cause, or conversely having one "miracle food" that will cure everything that ails you and let you walk on water. If it sounds too good (or too bad) to be true, it probably is.

The bottom line, there's not just one single cause of obesity (except for tautologies like "people eating more than they burn", which doesn't address the deeper question of why).

3. Look at study design.

So now you're reading an abstract or full text article from PubMed. How do you interpret their findings? A few key things to look out for:

3a) Studies on nonhuman animals are not always relevant to humans. Rats are widely used in experimentation, not because they give particularly good results, but because they are particularly easy to experiment on. Any nonhuman studies at best suggest opportunities for human research, they are not in and of themselves conclusive.

3b) Correlation is not causation. Too often the mainstream media totally overlooks this, with worrying headlines such as "coffee causes cancer" or the like. The problem with correlation studies (which, rather than do an actual experiment, look at other sources of data and draw correlations) is that you can never prove you've accounted for every confounding variable.

For example, say researchers did look at a big set of data and found Pokemon Go players are less likely to have cancer compared to non-players. You might be tempted to conclude that Pokemon Go prevents cancer, but there are many other explanations. Pokemon Go players tend to be young, for one, and cancer is much more prevalent among the elderly. Pokemon Go players may also as a group get more exercise than non-players, which may be responsible for a protective effect. Or maybe people in poor health are less likely to be able to play Pokemon Go.

Particularly in diet and nutrition, few choices are made in a vacuum. For example, someone who eats a lot of whole vegetables is more likely to also bike to work and get more exercise than someone who eats a lot of twinkies. Health and fitness are often whole lifestyles, and the health impact of the lifestyle is likely caused by a wide number of different factors, not any single one.

3c) Even experimental studies often have a high number of false positives. Often a p-value of 0.05 is used in science (might also be called a 95% CI). In rough terms, this means that 1 time in 20 we call something positive when it was actually a negative. There's a worse problem, though, and it occurs when we investigate a lot of questions, of which most are truly negative. For example, say you wanted to look at 100 possible carcinogens, and we'll pretend we have perfect knowledge and 1 is really carcinogenic. Let's even say that we get that one right - we correctly label it as a carcinogen. Using our p=0.05, however, we likely also call 5 non-carcinogenic substances as carcinogens.

So of the 6 carcinogens we "identified", we were actually wrong about 83% of the them - only one of the six was really carcinogenic! This is often called the "false positive paradox", where even a good test can give more false positives than true positives if the true results would be overwhelmingly negative. In the absolute worst case, if zero of the 100 were actually carcinogenic, then you only have 5 false positives - and in fact 100% of your "95% confident" significant findings were wrong!

Better p-values can help (although it can't help against certain types of errors) but what you also want to look for is a consensus of evidence, not just a single study. Single studies are often wrong - due to poor experimental design or sometimes just due to chance.

4. Consider the biology.

If trying to draw a causal link between A and B, is there a mechanism by which A affects B, and is this scientifically valid? This one can be tricky, because pseudoscience is rampant and invents "mechanisms" all the time. For example, one pseudoscientific claim I've seen goes, in part, that acidic blood pH causes bone mineral loss. Now, on the surface, it's true that hydroxyapatite, the main mineral in bone, does dissolve in acids, but if you dig deeper, you find that to dissolve it, you need to drop the pH below about 5.5, and the pH of your blood is tightly regulated between 7.35 and 7.45. By the time your body pH fell below 5.5 and your bones started to suffer, you'd have died a thousand times over. Debunking these kinds of claims requires some knowledge of biology, chemistry, or physics, and doing some independent research.