Choose your reading route
eduKateSG · Why Mathematics? · Statistical Significance and Practical Importance: Effect Size, Uncertainty and Decisions
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 8–14
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 15–21
Check assumptions
See where design and calibration matter.Route 4 · Chapters 22–28
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 29–34
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when a result is “statistically significant” but may not matter in real life? Significance tests compare data with a null model, while effect sizes and confidence intervals describe magnitude and uncertainty. Good decisions need all three, plus costs, benefits and context.
Did You Know? A tiny effect can achieve a very small p-value in a huge sample, while an important effect can miss a conventional threshold in a small, noisy study.
Full chapter index
Chapters 1–7 · Understand the question
Chapters 8–14 · Reproduce the example
Chapters 15–21 · Check assumptions
Chapters 22–28 · Explore applications
Chapters 29–34 · Study and verify
A p-value addresses compatibility with a statistical model. It does not by itself answer whether an effect is large, valuable, causal or worth acting on.
Under a specified null model, the p-value is the probability of a test statistic at least as extreme as observed, not the probability that the null hypothesis is true.
CHAPTER 3 OF 34 · Understand the question
3. An authoritative professional statement
The American Statistical Association warns that scientific conclusions and decisions should not be based only on whether a p-value crosses a threshold. Read the ASA statement.
Study A estimates an improvement of 0.2 marks with SE=0.05. Study B estimates 4 marks with SE=2.5.
Against zero, z_A=0.2/0.05=4. Study B gives z_B=4/2.5=1.6.
| Study | Estimate | SE | z |
|---|---|---|---|
| A | 0.2 | 0.05 | 4 |
| B | 4 | 2.5 | 1.6 |
A is precisely different from zero but only 0.2 marks. B is much larger but uncertain. Whether either matters depends on educational and practical thresholds.
Approximate 95% intervals are A: 0.102 to 0.298 and B: -0.9 to 8.9 marks. Intervals show magnitude ranges hidden by binary labels.
| Study | Lower | Upper |
|---|---|---|
| A | 0.102 | 0.298 |
| B | -0.9 | 8.9 |
Report raw differences in meaningful units and, when useful, standardized effects. Standardisation aids comparison but can hide the original educational scale.
Define a practical threshold before seeing results. An interval entirely above it supports a stronger usefulness claim than merely excluding zero.
A p-value above 0.05 does not prove no effect. It may reflect low power, noisy measurement or a wide range of compatible effects.
Testing many outcomes raises the chance of small p-values under null models. Pre-specification, adjustment and transparent reporting limit selective claims.
Flexible exclusions, outcomes and models can change p-values. Sensitivity analyses and complete reporting show robustness.
A small p-value for an observational association does not establish causal identification. Design and assumptions matter.
Normality, independence, variance structure, randomisation and survey design affect the reference distribution and SE.
For a fixed effect, larger n usually lowers SE and increases the test statistic. Significance can become almost inevitable for trivial nonzero differences.
When the goal is to show differences are acceptably small, equivalence-style hypotheses are more appropriate than failing to reject zero.
Posterior distributions can address probability statements under specified priors and likelihoods, but still require decisions about meaningful magnitude and loss.
Combine estimated benefits, harms, costs, uncertainty and values. Statistical evidence supplies inputs; mathematics does not choose social priorities alone.
Check pre-specified hypotheses, effect units, interval, practical threshold, power, multiple testing, design, assumptions and sensitivity.
Reproduce z=4 and 1.6, calculate both intervals, vary sample size, apply a practical threshold and simulate multiple testing.
No single p-value, effect size or interval resolves measurement bias, publication bias, confounding or decision values.
Store estimate, SE, test statistic, p-value, confidence interval, effect unit, practical threshold, multiplicity adjustment and decision consequence.
Reproduce test statistics, compute intervals, simulate power and multiplicity, and run sensitivity specifications without outcome-driven deletion.
Researchers, policymakers, engineers and education leaders interpret statistical evidence for decisions.
Simulate the same tiny effect at several sample sizes, then compare p-values, intervals and practical-threshold decisions.
p<0.05 is not proof, p>0.05 is not no effect, statistical significance is not practical importance, and association is not causation.
CHAPTER 27 OF 34 · Explore applications
27. Connection to recent mathematics articles
The confidence-interval guide shows compatible magnitudes. The sample-size guide explains power and precision inputs.
Students should translate effects back into marks, minutes or probabilities. Parents can ask whether a school intervention changed outcomes by a meaningful amount, not only whether p<0.05.
- Calculate two z statistics.
- Calculate both intervals.
- Define a practical threshold.
- Increase n.
- Diagnose multiple testing.
### Is a p-value the probability the null is true? No.
### Does significance mean large effect? No.
### Does nonsignificance prove no effect? No.
### Should p-values be hidden? No; they should be interpreted with magnitude, uncertainty and design.
Report hypothesis and model, estimate and units, SE, interval, p-value, practical threshold, sample and design, multiplicity, assumptions, sensitivity and decision context.
Evidence has at least two dimensions: how large the effect is and how precisely it is known. A binary threshold collapses both.
Confirm p-value meaning, effect units and interval, pre-specified importance threshold, power, multiplicity, causal design, assumption checks, sensitivity and no threshold-only decision.
Continue through the Mathematics Learning Hub. Recalculate one uncertainty measure and explain which decision it can and cannot support.
Extended practice laboratory
Practice case 1: Define null
Setup: Start from the article’s significance-importance mechanism for case 1. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 2: Set estimate A
Setup: Start from the article’s significance-importance mechanism for case 2. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 3: Set SE A
Setup: Start from the article’s significance-importance mechanism for case 3. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 4: Calculate z A
Setup: Start from the article’s significance-importance mechanism for case 4. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 5: Set estimate B
Setup: Start from the article’s significance-importance mechanism for case 5. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 6: Set SE B
Setup: Start from the article’s significance-importance mechanism for case 6. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 7: Calculate z B
Setup: Start from the article’s significance-importance mechanism for case 7. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 8: Calculate interval A
Setup: Start from the article’s significance-importance mechanism for case 8. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 9: Calculate interval B
Setup: Start from the article’s significance-importance mechanism for case 9. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 10: Define practical threshold
Setup: Start from the article’s significance-importance mechanism for case 10. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 11: Increase n
Setup: Start from the article’s significance-importance mechanism for case 11. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 12: Decrease n
Setup: Start from the article’s significance-importance mechanism for case 12. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 13: Add multiple tests
Setup: Start from the article’s significance-importance mechanism for case 13. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 14: Adjust multiplicity
Setup: Start from the article’s significance-importance mechanism for case 14. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 15: Add confounding
Setup: Start from the article’s significance-importance mechanism for case 15. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 16: Add randomisation
Setup: Start from the article’s significance-importance mechanism for case 16. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 17: Test equivalence
Setup: Start from the article’s significance-importance mechanism for case 17. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 18: Add Bayesian view
Setup: Start from the article’s significance-importance mechanism for case 18. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 19: Add cost
Setup: Start from the article’s significance-importance mechanism for case 19. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 20: Add harm
Setup: Start from the article’s significance-importance mechanism for case 20. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 21: Audit p-value
Setup: Start from the article’s significance-importance mechanism for case 21. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 22: Audit effect size
Setup: Start from the article’s significance-importance mechanism for case 22. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 23: Audit interval
Setup: Start from the article’s significance-importance mechanism for case 23. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 24: Name limits
Setup: Start from the article’s significance-importance mechanism for case 24. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 25: Write conclusion
Setup: Start from the article’s significance-importance mechanism for case 25. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and sampling design, verify the uncertainty identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.