Multiple comparisons become important when a study tests many hypotheses, outcomes or group differences at once. The core aim of Science mastery is not to make students memorise a list of correction methods. It is to help them understand a simple statistical danger: the more chances you give random noise to look significant, the more likely one false positive becomes.
For students and parents searching for multiple comparisons, multiple testing problem, Bonferroni correction, false positive rate, p-value correction or multiple hypothesis testing, the most useful principle is this: one 5% false-positive risk repeated many times is not the same as one 5% risk overall.
Scientific analysis must account for how many opportunities it creates to find a seemingly impressive result.
The 60-Second Multiple-Testing Problem
Suppose 20 independent tests are performed at α = 0.05.
Even if every null hypothesis is true, the chance of seeing at least one nominally significant result is much greater than 5%.
The approximate probability is:
1 − 0.9520 ≈ 64%
under the simplifying assumption that the tests are independent.
This is why repeated testing needs special care.
Wait, What? If You Test Enough Things, Something May Look Significant by Chance?
Exactly.
Imagine measuring 100 unrelated outcomes after a treatment.
At a 0.05 threshold, some outcomes may cross the significance line just by random variation.
If researchers report only those “successful” results, the evidence becomes misleading.
Family-Wise Error Rate
The family-wise error rate is the probability of making at least one Type I error within a family of tests.
Multiple-comparison corrections can control this overall risk.
This is especially important when the scientific question treats the tests as one family.
Bonferroni Correction
A simple correction divides the desired family-wise α by the number of comparisons.
Example:
Desired overall α = 0.05
Number of tests = 5
Adjusted threshold:
0.05 ÷ 5 = 0.01.
Each test must then meet p < 0.01 to control the family-wise rate conservatively.
Wait, What? Correction Can Reduce Power?
Yes.
Stricter thresholds reduce false positives.
But they also make real effects harder to detect.
This can increase Type II errors.
Multiple-testing control therefore involves the same broader balance between false positives and false negatives.
See Type I and Type II Errors.
False Discovery Rate
In large-scale studies, researchers may control the false discovery rate rather than the probability of any false positive.
This approach asks:
Among the results called significant, what proportion are expected to be false discoveries?
Methods such as Benjamini–Hochberg are commonly used in high-dimensional research.
The details belong to more advanced statistics, but the principle matters: different correction goals answer different error-control questions.
A Worked Example: Testing Many Fertilisers
A study compares 10 fertilisers against one control.
Each comparison is tested independently at p < 0.05.
Even if none truly works, the chance that at least one looks significant is greater than 5%.
A multiple-comparison method can reduce the risk of declaring a random winner.
A Worked Example: Many Outcomes
A health study measures:
- memory;
- attention;
- reaction time;
- mood;
- sleep;
- heart rate;
- blood pressure;
- stress.
If only one outcome reaches p = 0.04, the reader should ask:
- Was it pre-specified?
- How many tests were performed?
- Was a correction used?
The context changes interpretation.
Pre-Specifying Primary Outcomes
One way to reduce multiple-testing problems is to define the main outcome before looking at the data.
This separates:
- confirmatory tests;
- exploratory analyses.
Exploration is valuable, but exploratory findings should be labelled honestly and ideally replicated.
See Preregistration.
Multiple Comparisons and p-Hacking
If researchers repeatedly try:
- different outcomes;
- different subgroups;
- different exclusions;
- different statistical models;
until something becomes significant, the false-positive risk can inflate dramatically.
This practice is often called p-hacking.
Transparent analysis plans reduce this risk.
Multiple Comparisons and Replication
A surprising result discovered among many exploratory tests should usually be treated as a hypothesis for future confirmation.
Independent replication helps determine whether the finding is real or a chance discovery.
See Repeatability and Reproducibility.
Multiple Comparisons and Effect Size
Do not focus only on adjusted p-values.
Also examine:
- effect size;
- confidence interval;
- scientific plausibility;
- replication.
A tiny corrected p-value does not automatically make the effect important.
Primary Science Foundations
Primary learners can understand the simple version:
If you keep checking many random patterns, eventually one may look special by accident.
This builds healthy scepticism about cherry-picked results.
Secondary Science Multiple Comparisons
Secondary students should increasingly recognise:
- inflated false-positive risk;
- Bonferroni-style correction;
- false discovery rate;
- pre-specified outcomes;
- p-hacking;
- replication.
How to Practise
For any study with many outcomes, ask:
- How many tests were performed?
- Which outcomes were planned in advance?
- Was the threshold adjusted?
- Which findings were exploratory?
- Have surprising findings replicated?
Common Mistakes
- treating each p-value as if it were the only test;
- ignoring how many outcomes were examined;
- using Bonferroni mechanically for every problem;
- forgetting that stricter correction can reduce power;
- reporting only successful comparisons.
Frequently Asked Questions
What is the multiple comparisons problem?
It is the increased chance of false positives when many statistical tests are performed.
What is Bonferroni correction?
It is a conservative method that divides the desired overall significance level by the number of comparisons.
Why not always use a very strict correction?
Because stronger correction reduces false positives but can also reduce power and increase false negatives.
What is p-hacking?
It is trying many analyses or selections until a significant result appears, then presenting that result without full transparency.
Useful eduKateSG Routes
- Type I and Type II Errors
- Statistical Significance
- Preregistration
- Repeatability and Reproducibility
The Core Aim
Multiple comparisons create multiple chances for random noise to look important.
Count the tests. Predefine the main outcomes. Correct where appropriate. Replicate surprising findings.
That is the core aim: stop statistical opportunity from masquerading as scientific discovery.
Properly taught kids shine a bright light into the future.