A meta-analysis is a statistical method for combining the results of several studies that asked the same question. It sits near the top of most evidence ladders because one well-run pooled analysis can say more than any single small trial. It is also one of the most misused phrases in supplement marketing, because pooling only works when the studies going in are sound and comparable. This explainer covers what the method does, where it can go wrong, and what to look for before you trust the conclusion.
Three kinds of review, often confused
The words "review," "systematic review" and "meta-analysis" are used loosely, but they describe different things.
A narrative review is an expert's summary of a field. It can be excellent, but the author chooses which studies to discuss, so it can also reflect the author's view. There is usually no stated method for finding or selecting the studies.
A systematic review follows a written plan. The Cochrane Handbook, the standard methods manual for these reviews, defines one as an attempt to collate all the empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. The question, the rules for including studies and the search strategy are set in advance, and the included studies are assessed for bias [1].
A meta-analysis is the arithmetic step. The Handbook defines it as the statistical combination of results from two or more separate studies [2]. Most good meta-analyses sit inside a systematic review, but not all do, and not every systematic review includes one: when the studies are too different to combine, reviewers may describe them side by side instead.
NCCIH's guide to journal articles makes the same point more simply: when researchers look at many studies on the same subject and the results agree, that helps show the findings are reliable [3].
"Meta-analysis" describes a method, not a verdict. A pooled result from ten small, short, biased trials is still ten small, short, biased trials, just averaged.
How pooling works: the forest plot in plain words
Most meta-analyses show their results in a forest plot. The Handbook describes it as a display of effect estimates and confidence intervals for both the individual studies and the meta-analysis [2]. Once you know the parts, it takes seconds to read.
- One row per study. Each study appears as a small square placed at its result, with a horizontal line showing its confidence interval, the range of results compatible with the data.
- Square size is weight. Bigger squares count for more. In the common inverse-variance method, each study's weight is the inverse of the variance of its estimate, so larger, more precise studies pull the average harder [2].
- The vertical line of no effect. Results on one side favor the intervention, results on the other favor the comparison. A study whose line crosses it did not show a clear difference on its own.
- The diamond at the bottom. This is the pooled result. Its center is the combined estimate and its width is the combined confidence interval [2].
An illustration, with invented numbers: six small trials each find a slight improvement in some measured outcome, but every one of their lines crosses the line of no effect. Pooled, the diamond may sit clear of that line, because combining them narrows the uncertainty. That is the method working as intended. The same diamond can also mislead, which is the subject of the next section.
Two modeling choices appear in methods sections. A fixed-effect analysis assumes the true effect is the same in every study and that differences are due to chance. A random-effects analysis assumes the true effects vary between studies, usually following a normal distribution [2]. When studies differ in population, dose or duration, as supplement trials often do, the random-effects assumption is usually the more realistic one, and its wider interval is the honest price of that realism.
Where a meta-analysis can mislead
Pooling cannot fix problems inside the studies, and it can introduce a few of its own. Four are worth knowing by name.
Heterogeneity
If the studies disagree more than chance would explain, a single average may describe none of them. The Handbook's rough guide to the I² statistic reads: 0 to 40% might not be important, 30 to 60% may represent moderate heterogeneity, 50 to 90% substantial, and 75 to 100% considerable [2]. The ranges overlap on purpose; they are a guide, not a cutoff. A high I² is a signal to ask why the studies differ (dose? population? product?) before trusting the diamond.
Small-study effects and missing results
Studies with exciting results are more likely to be published and cited. The Handbook defines non-reporting bias as arising when decisions about how, when or where to report results are influenced by their P value, size or direction [4]. A funnel plot, a scatter of each study's effect against its size, can show a tendency for smaller studies to report different effects from larger ones. The Handbook advises that formal tests for funnel-plot asymmetry be used only when a meta-analysis includes at least 10 studies, and cautions that asymmetry is not proof of missing studies, because it has other possible causes [4].
Garbage in, garbage out
If most included trials were short, unblinded, or funded and run in ways that raise the risk of bias, the pooled result inherits those weaknesses with a false air of precision. Good reviews assess each study's risk of bias and often report what happens when the weakest studies are removed.
Apples and oranges
Pooling a powder, a juice and a capsule, or a week-long trial with a year-long one, can produce an average that matches no product on a shelf. Before you apply a pooled result to a specific label, check that the included studies used a comparable form and dose. Our explainer on study dose versus label dose covers how.
Reading the abstract and its certainty language
NCCIH describes an abstract as a brief description of the paper's key points, usually covering objectives, methods, results and conclusions, and suggests checking the publication date because standards and techniques change over time [5]. For a pooled review, pull out these details before you read the conclusion:
- How many studies and how many participants in total, and how many were in the largest single study.
- Who the participants were (age, health status, training level) and how long the studies lasted.
- What exactly was measured, and whether it is something people feel or do, or a lab marker.
- The size of the pooled effect and its confidence interval, not just whether it was "significant."
- Any mention of heterogeneity, risk of bias or publication bias.
- The certainty-of-evidence grade, if one is given.
That last item is often the most useful sentence in the abstract. Many reviews grade their evidence using the GRADE approach, which Cochrane has formally adopted for its own reviews [6]. GRADE has four levels: high, moderate, low and very low certainty. Evidence from randomized trials starts as high and can be rated down for five reasons: risk of bias, inconsistency, indirectness (the studies do not quite match the question), imprecision, and publication bias [6]. A sentence such as "low-certainty evidence suggests a small benefit" is the authors telling you, plainly, that the next good study could change the answer.
What the research shows
What the hype says
- Product pages often cite "a meta-analysis" as if the phrase settled the question.
- The certainty grade and the number of participants are rarely mentioned.
- Results for one form or dose are often carried over to a different product.
Hype is described in our words, not quoted, and is not a claim by this site.
This is also how we use pooled reviews on this site. Our method gives the highest readings only when several pooled reviews of human trials agree on a specific outcome at a dose a label can match, and each ingredient entry names the reviews behind its reading so you can apply the checks above yourself.
Even a high-certainty finding describes an average across the people studied, not what will happen to one person. Existing conditions, medicines and pregnancy can change what is appropriate, so talk with a clinician or pharmacist before acting on any review.
Sources
- Cochrane. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), Chapter 1: Starting a review (definition of a systematic review). cochrane.org
- Cochrane. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), Chapter 10: Analysing data and undertaking meta-analyses (definition, forest plots, inverse-variance weighting, I² guide, fixed- and random-effects models). cochrane.org
- NCCIH. How To Make Sense of a Scientific Journal Article: Methods, Types of Research. National Institutes of Health. nccih.nih.gov
- Cochrane. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis (non-reporting bias, funnel plots, the 10-study guidance). cochrane.org
- NCCIH. How To Make Sense of a Scientific Journal Article: Abstract and Main Sections. National Institutes of Health. nccih.nih.gov
- Cochrane. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), Chapter 14: Completing summary of findings tables and grading the certainty of the evidence (GRADE levels and domains). cochrane.org
This explainer is educational and is not medical advice. Ask a qualified clinician or pharmacist before starting any supplement, especially if you have an existing condition or take medication.