Signal vs. Noise in Health Headlines

A health headline is a compression of a compression. A study becomes an abstract, the abstract becomes a press release, the press release becomes a news story, and the story becomes a headline, a post and a product pitch. Each step drops detail, and the details dropped first are usually the ones that limit the finding: who was studied, what was measured, and how big the difference was. NCCIH, which runs a public "Know the Science" initiative on exactly this problem, notes that some health news stories are missing important information and some are confusing, conflicting or misleading [1] [2].

This explainer covers five patterns that account for most of the gap. None requires reading the original paper to spot, though the paper is always the final word.

Pattern 1: skipping rungs on the evidence ladder

Research on a question usually climbs a ladder. Each rung answers a different question, and a headline that quotes a low rung as if it were the top one is the most common kind of noise.

1. Cells in a dish

   Shows that a compound can do something to cells under laboratory conditions, often at concentrations a body would never see. Useful for generating ideas; says nothing yet about people.

2. Animal studies

   Show an effect in a living system, but doses, biology and outcomes often differ from humans. NCCIH's checklist for health news starts with this question: was the product or approach tested on people [3]?

3. Observational studies in people

   Follow people who already do or eat something and compare them with people who do not. They can reveal associations, but other differences between the groups can explain the result. In the GRADE system used to grade evidence in systematic reviews, non-randomized studies start at low certainty because of confounding and selection bias [4].

4. Randomized controlled trials

   Assign people by chance to the intervention or a comparison, which balances the other differences. NCCIH describes well-planned clinical trials as giving the clearest information about whether an approach is effective and safe in humans [5].

5. Systematic reviews and meta-analyses

   Gather all the trials that meet pre-set criteria and assess them together [6]. When several agree, the signal is at its clearest. Our explainer on what a meta-analysis is covers how these can still mislead.

A useful habit is to ask, for any headline, "which rung is this?" If the answer is cells or animals, the honest summary is "interesting early research," whatever the headline says.

Pattern 2: a marker standing in for an outcome

Many studies measure something in the blood or on a scan rather than something a person experiences. FDA defines a surrogate endpoint as a measure used as a substitute for a direct measure of how a patient feels, functions or survives, which does not measure the benefit of interest itself but is expected to predict it. FDA also distinguishes candidate surrogates, still being evaluated, from validated ones backed by strong evidence that a change in the marker predicts a specific benefit [7].

Supplement research leans heavily on markers: an antioxidant capacity score, an inflammatory protein, a level of a compound in the blood. A change in one of these can be real and still say little about how anyone feels or performs. When a headline describes a feeling or a function, check whether the study measured that feeling or function, or a marker thought to be related to it.

Pattern 3: relative numbers without absolute ones

Consider an illustration with invented numbers. In a trial, an unwanted outcome happens to 5 of every 1,000 people in the comparison group and 4 of every 1,000 in the group taking something. That result can be described accurately in two ways:

• Relative: a 20% lower rate (one is 20% smaller than five).
• Absolute: one fewer person per 1,000, or a difference of 0.1 percentage points.

Both are true. The relative number sounds large; the absolute number tells you how much the result would matter to any one reader. Headlines strongly prefer the first. NCCIH's checklist asks the practical version of this question: were the results big enough to be meaningful to you [3]? The FTC makes the same distinction for advertisers, noting that some statistically significant results are too small to have real consequences for consumers [8].

Pattern 4: inflation between the paper and the press release

Exaggeration often starts before a journalist sees the study. A 2014 analysis in The BMJ examined 462 press releases on health-related research issued by 20 leading UK universities in 2011, along with the underlying papers and 668 related news stories. It found that 40% of the press releases contained advice more direct than the paper supported, 33% made causal claims from correlational research, and 36% inferred human effects from animal research. When a press release exaggerated, news stories were much more likely to exaggerate too: for example, 86% of stories repeated an animal-to-human leap when the release contained one, against 10% when it did not [9]. It was an observational study, so it shows an association, but the pattern is a reason to read past both the headline and the press release to the paper's own conclusion.

Pattern 5: the single-study headline

One study, especially a small or first study, is a data point. NCCIH's checklist asks whether enough people were studied, whether they resemble you in age, sex or other ways that matter, whether the study lasted long enough to show longer-term benefits or risks, and whether this is the first time the result has been reported; one study rarely settles anything [3]. A headline that presents one short trial in a few dozen people as settled news about an everyday ingredient is the classic case.

A one-minute check for any health story

MedlinePlus suggests asking who runs a source and who pays for it, looking for dates on content, and checking research details such as how many people were studied, for how long and who funded it [10]. NCCIH's guide to online resources adds a question worth asking of any page selling something: are you reading news or advertising [11]? Combined with the patterns above, that gives a quick routine:

• Which rung of the ladder is the study on: cells, animals, observational, randomized, pooled?
• Did it measure something people feel or do, or a marker?
• Is there an absolute number, or only a percentage change?
• How many people, for how long, and are they like you?
• Is this the first study to report the result?
• Who funded it, and does the source sell a product linked to it?

For supplement ingredients specifically, our routine for reading a trending-ingredient claim takes these questions a step further, to the dose and the reading.

Sources

1. NCCIH. Know the Science: hub of tools for understanding health research. National Institutes of Health. nccih.nih.gov
2. NCCIH. The Facts About Health News Stories: Complementary Health Approaches in the News. National Institutes of Health. nccih.nih.gov
3. NCCIH. Checklist for Understanding Health News Stories. National Institutes of Health. nccih.nih.gov
4. Cochrane. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), Chapter 14: grading the certainty of the evidence (GRADE starting points for randomized and non-randomized studies). cochrane.org
5. NCCIH. How To Make Sense of a Scientific Journal Article: Methods, Types of Research. National Institutes of Health. nccih.nih.gov
6. Cochrane. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), Chapter 1: Starting a review (definition of a systematic review). cochrane.org
7. U.S. Food and Drug Administration. Surrogate Endpoint Resources for Drug and Biologic Development (definition and levels of validation). fda.gov
8. Federal Trade Commission. Health Products Compliance Guidance (statistical versus clinically meaningful results). ftc.gov
9. Retrospective observational study of exaggeration in 462 university press releases and 668 related news stories. The BMJ, 2014. PMID 25498121. pubmed.ncbi.nlm.nih.gov
10. MedlinePlus, National Library of Medicine. Evaluating Health Information, updated February 26, 2024. medlineplus.gov
11. NCCIH. Know the Science: Finding and Evaluating Online Resources (including "Are You Reading News or Advertising?"). National Institutes of Health. nccih.nih.gov

This explainer is educational and is not medical advice. Ask a qualified clinician or pharmacist before starting any supplement, especially if you have an existing condition or take medication.