Statistical significance works by defining a statistical test and a decision threshold before interpreting the result, calculating how incompatible the observed data are with the test’s null reference model, and declaring the result “statistically significant” when the resulting p-value falls below the chosen significance level α. That label is therefore a property of a testing procedure and threshold—not a direct measure of truth, effect size, practical importance, causal validity or probability of replication.
There may be no phrase in quantitative research that sounds more authoritative than:
The result was statistically significant.
The sentence feels like a door closing.
Significant.
Therefore real.
Not significant.
Therefore nothing happened.
That interpretation is far stronger than the mathematics earns.
The American Statistical Association has repeatedly emphasised that p-values and significance tests can be useful when properly applied, but statistical significance is an assessment relative to sampling variation and a chosen threshold. It is not a measure of practical significance, effect magnitude or scientific truth.
The governing question: under this test, this null model, this error threshold and this analysis plan, did the observed result cross the pre-specified decision boundary—and what does that boundary still fail to tell us?
Quick Read
QUESTION → NULL HYPOTHESIS → TEST STATISTIC → REFERENCE DISTRIBUTION → P-VALUE → PRE-SPECIFIED α → COMPARE p WITH α → SIGNIFICANT / NOT SIGNIFICANT DECISION → EFFECT SIZE → INTERVAL → MULTIPLICITY / DESIGN / BIAS CHECK → SCIENTIFIC INTERPRETATION
1. Statistical Significance Is a Decision Rule
At its simplest, null-hypothesis significance testing defines a significance level α before the test is interpreted.
If p ≤ α, the result is called statistically significant and the null hypothesis is rejected under that procedure.
If p > α, the procedure does not reject the null.
The words “significant” and “not significant” are therefore labels created by comparing a continuous quantity with a threshold.
2. Alpha Is the Designed Type I Error Level
Under the conditions defining a valid test, α controls the probability of rejecting the null when the null hypothesis is true.
With α = 0.05, the procedure is designed so that false rejection occurs no more than about 5% of the time under the relevant null conditions.
This is a long-run operating property of the test.
It is not a 5% probability that today’s significant finding is false.
3. A Significance Threshold Must Be Chosen Before Seeing the Result
If researchers decide that α = 0.05 is the test rule only after seeing p = 0.047, the operating rule was not actually fixed in advance.
If p had been 0.06 and the threshold were then loosened to 0.10, the nominal error guarantee would not represent the data-dependent decision process.
Thresholds belong to the design of inference, not retrospective decoration.
4. The P-Value and the Significance Label Are Different Objects
A p-value is a continuous result from a test.
“Significant” is a binary label created after comparing that result with α.
p = 0.049 and p = 0.051 are numerically almost identical.
The threshold gives them different labels.
The evidence did not suddenly change at the third decimal place.
5. Statistical Significance Is Not the Probability the Hypothesis Is True
A significant result at α = 0.05 does not mean the null hypothesis has less than a 5% probability of being true.
The frequentist test does not assign posterior probability to the hypothesis.
For that kind of probability statement, a Bayesian model would need prior probabilities and likelihoods.
See How Bayesian Inference Works.
6. Statistical Significance Is Not the Probability the Result Happened by Chance
A sentence such as “there is less than a 5% chance this result occurred by chance” is usually wrong.
The p-value measures extremeness under a reference model.
It does not divide the observed effect into “real” and “chance” percentages.
See How P-Values Work.
7. Statistical Significance Is Not Effect Size
Suppose a website changes the colour of one button and measures fifty million visits.
Conversion rises by 0.003 percentage points.
The enormous sample can make that tiny difference statistically significant.
The effect may still be commercially irrelevant.
Magnitude belongs to How Effect Sizes Work.
8. Practical Significance Asks Whether the Effect Matters
A statistically significant two-mark improvement may matter if two marks moves students above a critical mastery threshold.
It may not matter if the scale spans 1,000 points and normal measurement error is ten marks.
Practical significance depends on consequence, cost, risk and the receiver’s decision.
Statistical significance does not contain that information automatically.
9. Sample Size Strongly Influences Significance
For a fixed effect magnitude and variance, larger samples generally reduce standard errors.
The test statistic grows relative to its noise.
The p-value becomes smaller.
This is why a significance label cannot be interpreted without sample size and precision.
10. Small Studies Can Miss Large Effects
A treatment improves scores by twelve marks in a tiny trial.
The confidence interval is wide and p = 0.09.
Calling the treatment ineffective confuses absence of statistical significance with evidence of absence.
The study may simply lack resolution.
11. Statistical Power Determines How Often Meaningful Effects Cross the Threshold
Power is the probability that the planned testing procedure will reject the null under a specified alternative effect and design.
A low-powered study may miss effects that matter.
See How Statistical Power Works.
12. Low Power Can Also Exaggerate the Effects That Become Significant
When a small study has a noisy effect estimate, only unusually large estimates may cross the significance threshold.
If journals preferentially publish the significant ones, the visible literature can contain inflated effects.
This is one route to winner’s curse and magnitude exaggeration.
13. A Significant Result Can Be a False Positive
That possibility is built into the test.
If α = 0.05 and the null is true, a correctly calibrated procedure still rejects in about 5% of repeated cases under the relevant conditions.
Significance is not proof.
It is a controlled decision with a designed error rate.
14. The False Positive Risk Among Published Significant Results Can Exceed Alpha
Alpha is P(reject | null true) under the procedure.
Researchers often want P(null true | significant result).
Those are different conditional probabilities.
The latter depends on how many tested hypotheses are actually null, power, selection and other features of the research environment.
15. Multiple Testing Inflates Opportunities for Significance
Run one test at α = 0.05.
Now run 100 unrelated null tests and highlight whichever cross 0.05.
The expected number of false positives is approximately five under simple independence.
The single-test significance label cannot forget the other 99 opportunities.
See How Multiple Testing Works.
16. Family-Wise Error Control Changes the Threshold
If a family contains many tests and the goal is to control the chance of any false positive in the family, Bonferroni or Holm procedures can use stricter effective thresholds.
A p-value that is significant unadjusted may no longer be significant after multiplicity control.
Significance is therefore relative to an error-control architecture.
17. False Discovery Rate Creates Another Significance Contract
In genomics or other high-dimensional discovery problems, researchers may control the expected proportion of false discoveries rather than the probability of any false discovery.
Benjamini–Hochberg adjusted significance means something different from ordinary single-test significance.
The label only makes sense when the error target is named.
18. Optional Stopping Creates Repeated Opportunities Across Time
Collect data.
Check significance.
If not significant, collect more.
Repeat until p < 0.05.
The nominal fixed-sample Type I error no longer describes the actual stopping rule.
Sequential methods must allocate error across repeated looks.
19. Interim Analyses Need Alpha Spending or Other Sequential Control
Clinical trials may examine evidence several times before the planned maximum sample is reached.
Group-sequential boundaries and alpha-spending functions preserve a designed family-wise Type I error across the sequence.
The threshold may be much stricter early in the trial.
20. One-Sided Significance Is Not a Shortcut for Smaller P-Values
A one-sided test places the rejection region in one direction.
It is appropriate only when effects in the opposite direction would not lead to the same inferential claim and the direction was specified before seeing results.
Choosing one-sided significance after observing a favourable direction invalidates the nominal error rate.
21. Confidence Intervals Expose the Magnitudes Hidden by the Label
For many standard tests, a two-sided 95% confidence interval excludes the null exactly when the corresponding two-sided test is significant at 0.05.
The interval gives much more information.
It shows whether compatible effects are tiny, moderate or huge.
See How Confidence Intervals Work.
22. A Significant Confidence Interval Can Still Lie Entirely Inside a Trivial Region
An effect estimate is +0.12 with a 95% interval from +0.08 to +0.16.
The interval excludes zero.
Suppose the smallest useful effect is +2.0.
The entire significant interval is practically negligible.
23. A Non-Significant Confidence Interval Can Contain Large Effects
An effect estimate is +5 with a 95% interval from −4 to +14.
The result is not statistically significant.
The interval contains moderate harm and substantial benefit.
The evidence is inconclusive, not evidence of no effect.
24. Equivalence Testing Reverses the Usual Null
If the question is whether two treatments are sufficiently similar, failing to reject a zero-difference null is inadequate.
Equivalence testing defines a region of effects too large to count as equivalent and asks whether the data are precise enough to reject those meaningful differences.
A significant equivalence result can support the claim that any remaining difference is smaller than the chosen equivalence margin.
25. Non-Inferiority Uses an Asymmetric Threshold
A cheaper or safer treatment may only need to show it is not worse than standard care by more than a pre-specified margin.
Non-inferiority testing therefore uses a one-sided clinically meaningful boundary.
The significance question is tied to that margin, not simply zero.
26. Significance Depends on the Null Hypothesis You Chose
Testing “effect = 0” asks one question.
Testing “effect ≤ −5” for non-inferiority asks another.
Testing “effect lies outside ±2” for equivalence asks another.
Statistical significance has no meaning without naming the hypothesis being tested.
27. Statistical Significance Is Conditional on the Statistical Model
A t-test assumes a particular sampling and variance structure.
A regression coefficient test assumes the regression model and standard-error method.
If observations are clustered but treated as independent, standard errors can be too small and significance too easy to obtain.
Model failure can manufacture significance.
28. Statistical Significance Is Conditional on Measurement Quality
A study can find a highly significant difference in an invalid outcome.
An educational app may significantly increase daily logins while durable learning remains unchanged.
The test can be statistically correct and scientifically misdirected.
Significance cannot repair construct validity.
29. Statistical Significance Is Conditional on Study Design
A significant association in observational data does not prove causation.
Confounding can produce a precise non-zero coefficient.
Causal interpretation needs randomisation or defensible identification assumptions.
See How Causal Inference Works.
30. Randomisation Gives Significance a Stronger Causal Foundation
In a well-conducted randomised trial, the assignment mechanism supports causal comparison between groups.
The statistical test can then be interpreted relative to that randomisation and outcome model.
Post-randomisation missingness, non-adherence and biased outcome measurement can still damage causal validity.
31. Preregistration Preserves the Meaning of Planned Significance Tests
If a study pre-specifies its primary outcome, test direction, analysis model and multiplicity strategy, the significance result has a clearer operating interpretation.
If hundreds of analyses were explored and one was presented as planned, the nominal threshold no longer describes the true opportunity set.
See How Preregistration Works.
32. HARKing Turns Discovery Into Apparent Confirmation
A hypothesis is discovered after looking at the data.
The paper rewrites the story as though the hypothesis was predicted beforehand.
The p-value is numerically unchanged.
The evidential context is not.
33. P-Hacking Is Significance Optimisation
Try different exclusions.
Try several covariates.
Try multiple outcome definitions.
Stop when one analysis becomes significant.
The final threshold crossing sits at the end of an unreported search process.
Significance was optimised rather than tested.
34. Researcher Degrees of Freedom Create Hidden Multiplicity
The number of printed p-values may be five.
The number of analytical paths considered may be five hundred.
Hidden flexibility makes nominal significance less trustworthy because the opportunity set is invisible.
35. Selective Reporting Distorts the Literature
If significant outcomes are published and non-significant outcomes disappear, readers see an evidence sample selected by threshold crossing.
The published proportion of significant results becomes far larger than the true evidence landscape warrants.
Full reporting is part of valid significance interpretation.
36. Publication Bias Can Convert Nominal Alpha Into a Literature-Level Illusion
Suppose twenty small studies test an ineffective treatment.
One becomes significant by chance and is published.
The nineteen null studies remain in drawers.
A literature reader sees one apparently positive study rather than twenty mostly negative attempts.
37. Replication Tests Whether Significance Travels to Fresh Data
A statistically significant result should not be treated as final.
Replication re-estimates the effect under new sampling variation and often a new implementation context.
If the original effect was inflated by winner’s curse, the replication may be smaller even when the underlying effect is real.
38. Failure to Replicate Is Not Defined Only by Crossing 0.05 Again
An original study reports effect 0.40 with p = 0.03.
A larger replication estimates 0.35 with p = 0.08 because its design differs or variance is larger.
Calling that a failed replication solely because the second p-value is above 0.05 ignores the similarity of effect estimates.
Replication should compare magnitudes and intervals, not merely significance labels.
39. Meta-Analysis Should Synthesize Effect Estimates, Not Vote on Significance
Counting how many studies were significant throws away magnitude and precision.
Ten small non-significant studies can collectively support a precise effect.
Ten significant studies can hide severe heterogeneity.
40. Significance Thresholds Can Be Useful When Decisions Must Be Binary
A regulator may need approve / do not approve.
A quality-control system may need stop / continue.
A trial may need continue / stop for futility.
The ASA’s task-force statement explicitly notes that thresholds can be useful when actions are required.
The threshold should be chosen with the consequences of wrong decisions in mind.
41. Different Decisions Can Justify Different Alpha Levels
There is no universal natural law that says α must equal 0.05.
Genome-wide association studies use much stricter thresholds because millions of tests create immense multiplicity.
Exploratory screening may use FDR control.
Decision design should precede convention.
42. Lowering Alpha Reduces False Positives but Increases False Negatives
Move α from 0.05 to 0.005.
False rejection becomes harder.
True effects also become harder to detect unless sample size increases.
Every threshold embeds a trade-off between error types and resource costs.
43. Statistical Significance Is a Coordinate, Not the Whole Map
A strong interpretation should usually report:
- effect estimate;
- confidence or credible interval;
- p-value or posterior probability where relevant;
- sample size;
- design;
- measurement quality;
- multiplicity;
- practical threshold;
- robustness and sensitivity.
Significance is one coordinate inside that evidence space.
44. Education Research Is Especially Vulnerable to Significance Without Educational Meaning
A huge dataset can establish that a teaching intervention changes test scores by 0.3 marks.
If the assessment has normal day-to-day variation of several marks, the finding can be statistically strong and educationally weak.
Learning claims require construct validity, transfer and meaningful effect size.
45. Medicine Needs Absolute Benefit and Harm Beside Significance
A drug may significantly reduce relative risk by 20%.
If baseline risk is one in 100,000, the absolute benefit is tiny.
If the drug creates serious adverse effects, the decision can still be negative.
Significance cannot perform benefit–harm analysis.
46. Engineering Often Needs Tolerance Thresholds More Than Zero-Effect Thresholds
An engineer may care whether a manufacturing shift exceeds 0.05 mm, not whether it differs exactly from zero.
A statistically significant 0.001 mm shift can be irrelevant to safety.
Operational tolerances define practical significance.
47. AI Benchmark Significance Needs Independence and Search History
A model beats another by 0.2 percentage points with p = 0.01.
If thousands of model variants were tuned against the same benchmark, the final comparison sits inside repeated adaptive search.
Fresh holdout data may matter more than nominal significance on an overused test set.
48. The Hostile Test: p = 0.049 Versus p = 0.051
Two studies estimate nearly identical effects with nearly identical precision.
One is labelled significant.
The other is labelled non-significant.
The correct interpretation is that the evidence is almost the same.
The threshold creates a reporting discontinuity, not an evidence discontinuity.
49. The Second Hostile Test: Tiny Effect, Giant Sample
Five million students are analysed.
The intervention effect is 0.05 marks.
p < 10−20.
The result is statistically significant and educationally negligible.
50. The Third Hostile Test: Large Effect, Tiny Study
A small trial estimates a twelve-mark benefit with p = 0.08.
The conclusion says “no effect”.
The interval includes substantial benefit and some harm.
The correct description is insufficient precision.
51. The Fourth Hostile Test: One Significant Result From Fifty Outcomes
Fifty outcomes are measured.
One has p = 0.03.
The abstract says the intervention works.
The significance label ignored forty-nine other opportunities for chance findings.
52. The Fifth Hostile Test: Significant Association, Invalid Cause
Tutoring attendance is associated with higher scores at p < 0.001.
Motivation and prior attainment are not measured.
The regression coefficient may be precisely non-zero.
The causal claim remains unidentified.
53. Primary School: Significance Begins as “A Rule for Deciding When Surprise Is Big Enough”
A child flips a fair coin and gets ten heads in a row.
That would be unusual under the fair-coin rule.
The earliest lesson is that science sometimes needs a pre-agreed boundary for saying, “this is surprising enough that we should question the rule.”
A threshold helps us make a decision. It does not turn a decision into certainty.
54. Secondary School: Separate Three Questions
- Is the result unusual under the null model?
- How large is the observed effect?
- Does that magnitude matter in the real problem?
These questions prevent significance from swallowing magnitude and meaning.
55. JC and University: Statistical Significance Becomes an Operating Characteristic
At higher levels, learners should reconstruct:
- null hypothesis;
- alternative hypothesis;
- test statistic;
- reference distribution;
- p-value;
- alpha;
- Type I error;
- power;
- one- versus two-sided design;
- multiplicity;
- stopping rules;
- effect size;
- confidence interval;
- practical threshold.
The significance label becomes one designed output of a full inference system.
56. Where Statistical Significance Fits in the eduKateSG “How Works” Landscape
- How P-Values Work — the continuous tail probability compared with alpha.
- How Statistical Inference Works — the larger estimation and testing system.
- How Effect Sizes Work — magnitude that significance does not measure.
- How Confidence Intervals Work — precision and compatible magnitudes.
- How Statistical Power Works — ability to cross the threshold under meaningful alternatives.
- How Multiple Testing Works — error control when many significance opportunities exist.
- How Preregistration Works — preserves which tests and thresholds were planned.
Statistical Significance owns one precise canonical role: describe whether a specified statistical test crossed a pre-defined error-control threshold, while keeping that threshold decision separate from effect magnitude, scientific importance, causal validity and truth.
57. What This Article Does Not Claim
- Statistical significance is not the probability a hypothesis is true.
- Statistical significance is not the probability a result happened by chance.
- Statistical significance is not an effect-size measure.
- p > α does not prove no effect.
- p < α does not prove practical importance.
- Alpha is not the false-positive probability of one already-observed significant result.
- 0.05 is a convention, not a law of nature.
- Multiplicity, optional stopping and researcher flexibility can invalidate naive threshold interpretation.
- A significant association is not automatically causal.
- Replication should compare estimates and uncertainty, not merely whether significance repeats.
58. A Compact Statistical-Significance Audit
- What null hypothesis was tested?
- What alternative hypothesis was relevant?
- What test statistic was used?
- What assumptions generate its reference distribution?
- Was the test one- or two-sided?
- Was direction specified before observing results?
- What alpha level was chosen?
- Was alpha chosen before seeing the data?
- What exact p-value was obtained?
- How far is it from the threshold?
- What effect size was estimated?
- What confidence interval surrounds the effect?
- What smallest effect would matter practically?
- Does the interval lie inside or outside that meaningful region?
- What was the study’s power for a meaningful effect?
- How many hypotheses were tested?
- Was multiplicity controlled?
- Were interim analyses performed?
- Was optional stopping possible?
- Was the analysis preregistered?
- How many analytical paths were explored?
- Could selection or publication bias influence what is visible?
- Does the design justify causal interpretation?
- Does the outcome validly measure the scientific construct?
- Would replication or meta-analysis change the evidence state?
59. Frequently Asked Questions
What does statistically significant mean?
It means the p-value from a specified statistical test fell below the pre-defined significance threshold alpha, so the procedure rejects the null hypothesis under its decision rule.
Does statistically significant mean important?
No. Statistical significance concerns a threshold relative to sampling variability. Practical or scientific importance depends on effect magnitude, uncertainty, cost, risk and context.
Does non-significant mean no effect?
No. A non-significant result may reflect a small effect, an absent effect or insufficient precision. Confidence intervals and equivalence methods help distinguish these possibilities.
Why is 0.05 used?
It is a historical convention for Type I error control. Different scientific and decision contexts may justify stricter or different error thresholds.
60. Authoritative Research Corridor
- American Statistical Association — Statement on Statistical Significance and P-Values
- ASA President’s Task Force — Statistical Significance and Replicability
- eduKateSG — How P-Values Work
- eduKateSG — How Confidence Intervals Work
- eduKateSG — How Multiple Testing Works
Final Thought: Significance Is a Gate, Not the Destination
A gate can be useful.
It can make a procedure consistent.
It can control one class of error.
It can force a decision when action cannot remain indefinite.
But the world beyond the gate is still full of questions.
How large is the effect?
How precisely is it known?
Was the design valid?
Was the analysis planned?
How many other gates were tried?
Does the result matter?
Will it survive new data?
Statistical significance is a disciplined way to cross one inferential boundary. Science becomes trustworthy when we remember that crossing the boundary is where interpretation begins, not where thinking ends.