Multiple testing works by recognising that every additional statistical test creates another opportunity for a false-positive claim, then defining which claims belong to the same inferential family and using procedures—such as family-wise error-rate control or false-discovery-rate control—to calibrate the collection of decisions rather than pretending each test exists alone.
One test at α = 0.05 has a familiar long-run error property under its null model.
Twenty tests at α = 0.05 do not magically preserve that same property for the collection.
If the tests were independent and every null were true, the chance of at least one false rejection would be:
1 − (1 − 0.05)20 ≈ 64%
Dependence changes the exact number, but not the underlying problem.
More opportunities to search create more opportunities to find something that looks unusual by chance.
The governing question: what set of claims are we making together, what kind of false-positive burden matters for those claims, and how should the decision thresholds change so the whole inferential system—not merely each isolated test—has a defensible error rate?
Quick Read
RESEARCH QUESTION → FAMILY OF CLAIMS → NUMBER + DEPENDENCE OF TESTS → ERROR NOTION → FWER / FDR / OTHER TARGET → PROCEDURE → ADJUSTED THRESHOLDS OR P-VALUES → POWER → DISCOVERIES → UNCERTAINTY → TRANSPARENT REPORTING
Modern multiple-testing literature distinguishes several error targets. Family-wise error rate controls the probability of making at least one false rejection in a family. False discovery rate controls the expected proportion of false discoveries among the rejected hypotheses. Those are different promises and serve different scientific goals.
1. Multiplicity Begins With Opportunity
Imagine flipping one fair coin ten times.
Ten heads would look surprising.
Now imagine checking ten thousand different coins and reporting only the strangest one.
Some coin will probably look remarkable even if every coin is fair.
Multiple testing is the statistical version of remembering how many opportunities existed to find something surprising.
2. A Single-Test p-Value Does Not Know How Many Other Tests You Ran
A p-value is calculated for one test under one null model.
It does not automatically encode that the analyst also tested nineteen other outcomes, forty subgroups and eight model variants.
The broader search process must be handled at a higher level.
This is why transparent reporting and preregistration matter alongside formal corrections.
3. The Family of Hypotheses Is the First Hard Problem
Which tests belong together?
All outcomes in one trial?
Only the primary outcomes?
All pairwise comparisons after one ANOVA?
All genes in a genome-wide screen?
All subgroup analyses that support one regulatory claim?
Recent methodological guidance continues to emphasise that deciding whether to adjust and over which family cannot be solved by arithmetic alone.
4. A Family Should Follow the Scientific Claim Structure
If a study will claim success if any one of five outcomes reaches significance, those five outcomes collectively create five opportunities to declare success.
They form a natural family.
If five analyses answer genuinely separate pre-specified questions with separate decisions, one universal family may be less appropriate.
The error-control unit should follow the decision unit.
5. Family-Wise Error Rate Protects Against Any False Rejection in the Family
The family-wise error rate, or FWER, is the probability of making at least one Type I error among a defined family of hypotheses.
Controlling FWER at 5% means constructing the procedure so that the probability of one or more false rejections in the family is no greater than 5% under the relevant conditions.
This is a stringent promise.
6. Bonferroni Is the Simplest FWER Control
If m hypotheses form one family and the desired family-wise alpha is α, Bonferroni tests each hypothesis at α/m.
For twenty tests and α = 0.05, each test uses 0.0025.
Bonferroni is easy to understand and works under very general dependence conditions.
It can be conservative because it protects the family using a worst-case bound rather than exploiting the exact dependence structure.
7. Bonferroni Does Not Mean Divide Every p-Value in the Universe Forever
The correction applies to a defined inferential family.
If the family is poorly defined, the correction is poorly defined.
Correcting every analysis a researcher has ever performed would be absurd.
Correcting none of twenty interchangeable routes to one headline claim is also indefensible.
The scientific claim defines the boundary.
8. Holm’s Procedure Improves on Simple Bonferroni
Holm’s step-down procedure orders p-values from smallest to largest.
The smallest is compared with α/m, the next with α/(m−1), and so on.
It controls FWER while being at least as powerful as ordinary Bonferroni.
When strong family-wise control is needed, Holm is often a better default than mechanically applying Bonferroni to every test independently.
9. Hochberg Uses a Step-Up Logic Under Stronger Dependence Conditions
Hochberg’s procedure examines ordered p-values from the other direction and can provide greater power under suitable assumptions.
The price of additional power is stronger conditions on dependence than Holm requires.
Multiplicity methods differ not only in formula but in the dependence structures under which their error guarantees hold.
10. Tukey’s Method Is Built for All Pairwise Mean Comparisons
After comparing several group means, researchers may want every pairwise comparison.
Tukey’s honestly significant difference procedure exploits the structure of that specific family.
Specialised procedures can be more efficient than generic Bonferroni because they use the geometry of the comparison problem.
11. Dunnett’s Method Is Built for Many Treatments Versus One Control
If five treatments are each compared with one common control, the comparisons are correlated through that shared control.
Dunnett-type procedures exploit that structure.
Again, multiple-testing control is stronger when it understands the actual comparison architecture.
12. Gatekeeping Procedures Protect Hierarchies of Claims
Clinical trials may have a primary endpoint and several secondary endpoints.
A gatekeeping strategy can require the primary claim to succeed before formal testing proceeds to secondary families.
This allocates the error budget according to scientific priority.
Not every hypothesis needs equal status.
13. Alpha Spending Treats Type I Error as a Resource Across Sequential Looks
Multiplicity arises across time as well as across outcomes.
If a trial checks significance every week and stops the first time p < 0.05, the overall false-positive probability rises.
Group-sequential methods and alpha-spending functions allocate the Type I error across interim analyses.
Repeated looking is repeated testing.
14. Optional Stopping Is a Multiplicity Problem Through Time
Collect twenty observations.
Check p.
If not significant, collect ten more.
Repeat until significance appears.
The nominal single-look p-value no longer describes the actual procedure.
A valid sequential design must account for the stopping rule.
15. False Discovery Rate Solves a Different Problem
Suppose a genomics experiment tests 100,000 markers.
Demanding almost zero probability of even one false positive may be too stringent for a screening problem.
Researchers may instead tolerate some false discoveries if the proportion among discoveries is controlled.
That is the motivation for false discovery rate.
16. FDR Is the Expected Proportion of False Discoveries Among Discoveries
Informally, if a procedure controls FDR at 5%, then across repeated use the expected fraction of rejected hypotheses that are actually false positives is controlled around that target under the procedure’s conditions.
This is not the same as saying every individual rejected hypothesis has a 5% chance of being false.
FDR is a property of the collection of discoveries.
17. Benjamini–Hochberg Is the Classic FDR Procedure
The Benjamini–Hochberg procedure sorts p-values from smallest to largest and compares them with increasing thresholds proportional to their rank.
Find the largest rank satisfying the criterion.
Reject that hypothesis and those with smaller p-values.
The procedure allows more discoveries than stringent FWER control when many signals may exist.
18. FWER and FDR Protect Different Receivers
If one false positive could approve an ineffective drug, FWER-like control may be appropriate.
If a genomic screen generates candidates for later validation, accepting a small fraction of false leads may be efficient.
Error control should follow consequence.
There is no universally superior multiplicity target detached from the decision system.
19. q-Values Re-Express FDR-Oriented Evidence
In some FDR frameworks, a q-value can be interpreted as the minimum FDR level at which a particular test would be called significant under the method.
It is not a conventional p-value.
It is attached to an error-control framework across many tests.
20. Adjusted p-Values Encode the Multiplicity Procedure
Instead of changing the significance threshold, software often reports adjusted p-values.
An adjusted p-value is the smallest family-level alpha at which that hypothesis would be rejected under the specified procedure.
Adjusted p-values are procedure-specific.
“Adjusted” without naming the method is incomplete reporting.
21. Dependence Among Tests Matters
Educational outcomes such as reading accuracy, fluency and comprehension are correlated.
Gene-expression tests are correlated through biological pathways.
Brain-imaging voxels are spatially correlated.
Some procedures remain valid under arbitrary dependence.
Others gain power by assuming independence or particular positive dependence structures.
22. Correlation Can Make Bonferroni Overly Conservative
If twenty outcomes are nearly duplicates, they do not represent twenty fully independent opportunities for chance findings.
Bonferroni still divides alpha by twenty.
That robustness is useful.
It can also sacrifice power.
Resampling or model-based methods can sometimes exploit the actual dependence structure more efficiently.
23. Permutation and Resampling Methods Can Preserve Dependence
Westfall–Young and related procedures use resampling to approximate the joint null distribution of many test statistics.
Because variables are resampled together, their correlation structure can be preserved.
These approaches can provide stronger power than generic corrections when their assumptions are appropriate.
24. Multiplicity Creates a Power Trade-Off
Stricter control of false positives makes true discoveries harder.
That is not a flaw in the correction.
It reflects the fact that demanding stronger protection requires more evidence per claim.
This connects directly to How Statistical Power Works.
25. More Primary Outcomes Can Require a Larger Study
If a trial has one primary outcome, sample-size planning can focus on one main test.
If success is defined by any of five primary outcomes after multiplicity control, each outcome may need a stricter threshold.
Maintaining power can require more participants.
Multiplicity belongs in design, not only in final analysis.
26. One Primary Outcome Is Not Automatically Better
Some phenomena genuinely have multiple essential dimensions.
A single outcome may oversimplify the decision.
The goal is not to minimise test count at all costs.
It is to define the claim architecture honestly and power the study accordingly.
27. Composite Outcomes Can Reduce Multiplicity but Create Interpretation Problems
Clinical trials sometimes combine several events into one composite endpoint.
This can reduce the number of tests and increase event counts.
But components may differ greatly in importance and treatment effect.
A composite can become statistically efficient while clinically opaque.
28. Omnibus Tests Can Ask Whether Any Group Difference Exists Before Pairwise Claims
An ANOVA F-test can ask whether all group means are equal.
If the omnibus test rejects, follow-up procedures can examine pairwise differences with appropriate multiplicity control.
Omnibus testing can reduce the number of initial claims but does not eliminate multiplicity in post hoc comparisons.
29. Planned Contrasts Can Be More Efficient Than Testing Every Pair
Theory may predict one comparison: treatment A versus the average of B and C.
Testing every possible pair wastes error budget on questions the theory did not ask.
Pre-specified contrasts align the inferential family with the scientific hypothesis.
30. Post Hoc Subgroup Analyses Are a Multiplicity Minefield
Age.
Sex.
Baseline severity.
Country.
School type.
Prior attainment.
With enough subgroups, something will look exceptional by chance.
Subgroup findings need pre-specification, interaction testing, multiplicity awareness and independent replication.
31. “Significant Here, Not Significant There” Is Not Evidence of a Significant Difference
A treatment is significant in men and not significant in women.
That does not prove treatment effects differ by sex.
The direct interaction between treatment and sex must be estimated.
Multiplicity can create differences in significance status even when subgroup effects are similar.
32. Multiple Model Specifications Create Hidden Tests
With covariates.
Without covariates.
Log-transform outcome.
Keep raw scale.
Exclude two outliers.
Include them.
If analysts explore many pipelines and report only the model that crosses the threshold, the effective number of opportunities can be much larger than the number of p-values printed in the paper.
33. Researcher Degrees of Freedom Are Multiplicity Without a Neat Test Count
Formal correction is easier when there are twenty named hypotheses.
It is harder when the analyst made dozens of flexible choices interactively.
This is why multiplicity control cannot be reduced to one software command.
Preregistration, blinded analysis, multiverse analysis and full reporting help expose hidden search paths.
34. Preregistration Reduces the Effective Garden of Forking Paths
Preregistration can identify primary outcomes, primary models and planned subgroup analyses before results are visible.
That does not eliminate exploratory analysis.
It preserves which paths were confirmatory and which were discovered after looking.
Multiplicity becomes easier to interpret when the search space is visible.
35. Exploratory Research Can Use FDR Rather Than Pretend Every Candidate Is Confirmatory
A discovery screen may intentionally test thousands of features.
FDR control acknowledges that the goal is to generate a useful candidate set with a controlled expected false-discovery proportion.
Later independent studies can test the strongest candidates confirmatorily.
Exploration and confirmation can use different error contracts.
36. Replication Is Stronger Than Ever-Tighter Correction for Discovery Claims
No multiplicity correction proves a discovered association is real.
Independent replication changes the information state by testing the finding in fresh data.
A well-designed discovery pipeline often combines FDR control with independent validation rather than demanding one dataset do everything.
37. Selective Reporting Can Defeat Correct Multiplicity Methods
A study tests fifty outcomes, applies no correction, and reports the three significant ones.
Readers see only three tests.
The multiplicity problem has been hidden rather than solved.
Proper inference requires reporting the full family or enough information to reconstruct it.
38. Adjusting Only Reported Tests Is Too Late if Unreported Tests Existed
Suppose ten analyses were run and three were published.
Applying Bonferroni for three tests does not repair the selection among ten.
The relevant family is defined by the opportunity structure, not the final table.
39. Multiple Confidence Intervals Need Simultaneous Coverage When Interpreted as a Family
Ordinary 95% confidence intervals have individual coverage properties.
If twenty are viewed together, the probability every interval covers its parameter is lower than 95% under independence.
Simultaneous intervals correspond to family-wise control in estimation rather than testing language.
40. Multiplicity Does Not Disappear if You Replace p-Values With Confidence Intervals
Reporting effect sizes and intervals is better than threshold-only reporting.
But if hundreds of intervals are estimated and only extreme ones are highlighted, selection still operates.
The statistical object changed.
The search problem did not.
41. Bayesian Models Can Partially Pool Multiple Estimates
Hierarchical Bayesian models treat many related effects as draws from a common distribution.
Noisy extreme estimates are shrunk toward the group distribution.
This can reduce winner’s-curse behaviour and improve estimation.
It is a different inferential framework from classical FWER or FDR control, not simply an alternative spelling of Bonferroni.
42. Machine Learning Has Multiplicity Through Hyperparameter Search
Train one model and evaluate once on a truly untouched test set.
Now train 2,000 variants and repeatedly inspect the test score.
The highest score partly reflects search over many opportunities.
Repeated benchmark tuning is multiplicity expressed through optimisation rather than p-values.
43. Leaderboards Can Overfit Through Collective Multiple Testing
Even if one team only submits a few models, thousands of teams can collectively optimise against the same public benchmark.
The benchmark becomes contaminated by repeated selection.
A private holdout, rolling benchmark or fresh test set restores some independence.
Multiplicity can exist across a community, not only inside one paper.
44. Education Research Faces Multiplicity Through Many Outcomes and Subgroups
Math score.
English score.
Attendance.
Motivation.
Confidence.
Primary students.
Secondary students.
Boys.
Girls.
High-attainment students.
With enough intersections, a positive subgroup is easy to discover.
Predefined primary outcomes and interaction tests protect interpretation.
45. Medicine Faces Multiplicity Because One False Claim Can Change Care
Drug trials may have multiple doses, endpoints, interim analyses and subgroup claims.
Regulatory decisions therefore often require explicit multiplicity strategies.
The consequence of a false positive can be treatment of millions of patients.
Error control is not mathematical fussiness.
It is decision safety.
46. Genomics Faces Multiplicity Because the Search Space Is Enormous
Genome-wide studies may test hundreds of thousands or millions of markers.
An unadjusted 0.05 threshold would generate huge numbers of false positives under widespread nulls.
Genomics therefore helped drive development and widespread use of large-scale FDR procedures and stringent genome-wide thresholds.
47. The Hostile Test: Twenty Outcomes, One Significant Result
A trial measures twenty equally important outcomes.
Nineteen have p-values above 0.05.
One has p = 0.04.
The abstract says the treatment works.
Without a pre-specified primary outcome or multiplicity strategy, the result is exactly the kind of pattern chance can generate across many opportunities.
48. The Second Hostile Test: Bonferroni Across Unrelated Questions
A researcher conducts five independent studies over five years.
Someone argues each study must use α = 0.01 because five tests exist somewhere in the researcher’s career.
That is not a scientifically coherent family.
Multiplicity correction should follow shared claim logic, not arbitrary counting.
49. The Third Hostile Test: Ten Subgroups, Only the Winner Reported
An intervention has no clear overall effect.
Ten subgroup analyses are run.
One subgroup produces p = 0.03.
The paper claims the intervention “works especially well” for that subgroup.
Without multiplicity control, interaction evidence and replication, the subgroup may simply be the maximum of ten noisy estimates.
50. The Fourth Hostile Test: Correction Applied After Choosing the Family
Fifty analyses are performed.
Four promising ones are selected.
Bonferroni is then applied only to those four.
The selection happened before the correction.
Error control must account for the opportunity set that generated the shortlist.
51. The Fifth Hostile Test: FDR Interpreted as Probability Each Discovery Is False
A genomics paper controls FDR at 5%.
A reader concludes each significant gene has exactly a 5% chance of being false.
That is not what the frequentist FDR guarantee says.
FDR is an expected error proportion across the discovery set under repeated application of the procedure.
52. Primary School: Multiplicity Begins as “If I Keep Looking, I Will Eventually Find Something Weird”
A child rolls one die six times.
Then they roll one hundred dice and choose the strangest sequence.
The chosen sequence looks surprising because it was selected from many chances.
The foundational habit is:
Remember how many chances you gave yourself to be surprised.
53. Secondary School: Multiple Testing Becomes the Difference Between One Question and Many Questions
Students can compare:
- one planned comparison;
- twenty comparisons with one chosen afterward.
The observed p = 0.04 means something different inside those two search structures.
Context includes the number of opportunities to obtain the result.
54. JC and University: Multiplicity Becomes Error-Control Architecture
At higher levels, learners should reconstruct:
- the hypothesis family;
- the scientific claim hierarchy;
- dependence among tests;
- FWER;
- Bonferroni;
- Holm;
- Hochberg;
- specialised comparison procedures;
- FDR;
- Benjamini–Hochberg;
- adjusted p-values;
- sequential looks;
- subgroups;
- model flexibility;
- power trade-offs;
- replication strategy.
The calculation becomes secondary to the error promise the study is making.
55. Where Multiple Testing Fits in the eduKateSG “How Works” Landscape
- How Preregistration Works — identifies confirmatory families before outcomes are visible.
- How Statistical Power Works — quantifies the detection cost of stricter thresholds.
- How Confidence Intervals Work — individual versus simultaneous coverage.
- How Effect Sizes Work — prevents threshold crossing from replacing magnitude.
- How Research Bias Works — selective reporting and analytical flexibility.
- How Probability Works — the mathematical basis of repeated false-positive opportunities.
- How Meta-Analysis Works — multiple outcomes, subgroups and missing-evidence synthesis.
Multiple testing owns one precise canonical job: control the error properties of a collection of statistical claims when the analysis creates more than one opportunity for discovery.
56. What This Article Does Not Claim
- Every set of tests ever run by a researcher does not form one universal family.
- Bonferroni is not the only multiplicity procedure.
- Bonferroni can be conservative, especially with correlated tests.
- FDR and FWER are not interchangeable error promises.
- An FDR of 5% does not mean each discovery individually has a 5% probability of being false.
- Multiplicity can arise through repeated interim looks, subgroup searches and model flexibility, not only many printed p-values.
- Replacing p-values with confidence intervals does not automatically remove selection problems.
- Correcting only the tests that survive selective reporting does not repair the original search process.
- Preregistration reduces hidden multiplicity but does not eliminate the need for appropriate formal control.
- Replication remains necessary for many discovery claims even after multiplicity adjustment.
57. A Compact Multiple-Testing Audit
- What claims are being made?
- Which hypotheses form one family?
- Why is that the correct family?
- How many tests are in it?
- Are the tests independent or correlated?
- Is the goal strict control of any false positive or control of the proportion of false discoveries?
- Is FWER or FDR the appropriate target?
- Which procedure is used?
- Why was that procedure chosen?
- Does its guarantee hold under the dependence structure?
- Are p-values adjusted or thresholds adjusted?
- Were primary and secondary outcomes pre-specified?
- Are there hierarchical or gatekeeping claims?
- Were interim analyses performed?
- Was alpha spending or sequential correction used?
- How many subgroup analyses were examined?
- Were interactions tested directly?
- How many model specifications were explored?
- Were unsuccessful analyses reported?
- How much power was lost through correction?
- Was sample size designed with multiplicity in mind?
- Are discoveries exploratory or confirmatory?
- Is independent replication planned?
58. Frequently Asked Questions
What is multiple testing?
Multiple testing occurs when more than one statistical hypothesis is evaluated within a shared inferential problem, creating multiple opportunities for false-positive claims and therefore changing the error properties of the collection.
What is family-wise error rate?
Family-wise error rate is the probability of making at least one Type I error among a defined family of hypotheses. Procedures such as Bonferroni and Holm can control it.
What is false discovery rate?
False discovery rate is the expected proportion of false rejections among all rejected hypotheses under repeated application of the procedure. It is often useful in large discovery problems where controlling any single false positive would be too restrictive.
What is Bonferroni correction?
Bonferroni divides the desired family-wise alpha by the number of hypotheses in the family, or equivalently adjusts p-values upward. It is simple and robust but can be conservative.
What is Benjamini–Hochberg?
Benjamini–Hochberg is a step-up procedure that orders p-values and compares them with rank-based thresholds to control false discovery rate under specified dependence conditions.
59. Authoritative Research Corridor
- When to Adjust for Multiple Testing: A Unifying Guiding Principle — 2026
- The Problem With Unadjusted Multiple and Sequential Statistical Testing
- Multiple Testing and Its Applications — FWER, FDR and False Discovery Proportion
- The False Discovery Rate: A Key Concept in Large-Scale Genetic Studies
- The Perils of Multiple Testing — Bonferroni, Holm, Hochberg and FDR
- ASA President’s Task Force Statement — Multiplicity, Uncertainty and Replicability
Final Thought: The Result Remembers How Hard You Looked
One surprising result after one planned test is one kind of evidence.
The same surprising result chosen from ten thousand attempts is another.
The number printed beside the result may be identical.
The route that produced it is not.
Multiplicity is therefore not mainly a punishment for doing lots of science.
It is a memory system.
It remembers how many opportunities existed to fool yourself.
It remembers how many doors were tried before one opened.
It remembers whether the winning door was chosen before or after the room was visible.
A trustworthy discovery is not judged only by how surprising it looks. It is judged by how surprising it remains after we count all the chances we gave ourselves to find it.