How to Read a Health Study Before You Believe the Headline

Almost every questionable claim about memory and brain health traces back to a real piece of research. The distortion happens between the study and the headline, and it follows recognisable patterns.

Question one: what kind of study was it?

This single question resolves most of it.

Laboratory and animal studies establish mechanism and generate hypotheses. They do not establish that anything works in humans. A great many supplement claims rest on cell culture work described as though it applied to people.

Observational studies follow people and look for associations. They can show that two things occur together. They cannot establish that one caused the other, because the groups being compared differ in countless ways.

Randomised controlled trials assign participants to conditions by chance, which balances known and unknown differences. These are far better positioned to establish causation.

Systematic reviews and meta-analyses combine studies. Quality depends on the quality of what was combined.

Question two: compared with what?

An intervention compared against nothing tells you little, because expectation, attention and simply doing something produce effects. The informative comparison is against an equally plausible alternative.

This is why much brain training research is weak: games compared against a waiting list rather than against an equally engaging activity.

Question three: what was actually measured?

Improvement on a laboratory task is not improvement in daily life. A supplement showing an effect on one test in one subgroup has not been shown to help memory.

Watch for surrogate outcomes as well — a biomarker changing is not the same as an outcome changing.

Question four: how many people, and for how long?

Small studies produce unstable results. Short studies cannot address outcomes that develop over decades. A twelve-week trial cannot tell you about dementia risk.

Question five: how large was the effect?

Statistical significance means a result is unlikely to be chance. It says nothing about size. A statistically significant improvement can be far too small to notice.

Relative risk is the other trap. A fifty percent reduction sounds substantial; if the baseline risk was two in a thousand, the absolute change is one in a thousand. Reporting that omits absolute figures is doing work.

Question six: who paid, and was it registered?

Industry funding does not invalidate research, but it is worth knowing. More informative is whether the trial was registered in advance with its outcomes specified — which limits the ability to measure many things and report whichever worked.

Question seven: has it been replicated?

A single study is a starting point. Many high-profile findings have failed to replicate. Reviews and repeated findings across independent groups carry far more weight than one striking result.

Warning phrases

• “Studies show” without naming any
• “Clinically proven”, which has no fixed meaning in supplement marketing
• “Linked to”, which usually indicates observational data
• “Breakthrough”, “miracle”, or “what doctors won't tell you”
• Testimonials in place of data
• A mechanism explained in detail with no human outcome data
• Claims to prevent, treat or cure a disease attached to a supplement

Applying it here

Apply these questions to this site as well. Where we describe observational findings, we should be saying so. Where evidence is weak, we should be saying that too. If an article here reads as more confident than the research behind it, that is a failure on our part — and we would like to hear about it.