Choose your reading route
eduKateSG · Why Mathematics? · Statistical Power: Effect Size, Sample Size and Type II Error
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 8–14
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 15–21
Check assumptions
See where design and calibration matter.Route 4 · Chapters 22–28
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 29–42
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when a study needs a fair chance to detect an effect that really exists? Statistical power is the long-run probability that a specified testing procedure rejects its null hypothesis under a specified alternative. It connects effect size, variability, sample size, significance level and study design before data are collected.
Did You Know? Power is not a property of a study in isolation. It changes with the particular alternative effect, analysis method, missing-data pattern and assumptions being considered.
Full chapter index
Chapters 1–7 · Understand the question
Chapters 8–14 · Reproduce the example
Chapters 15–21 · Check assumptions
Chapters 22–28 · Explore applications
Chapters 29–42 · Study and verify
- Limits
- A spreadsheet workflow
- A coding workflow
- Applications and careers
- A student investigation
- Common misconceptions
- Connection to recent mathematics articles
- Guidance for students and parents
- Practice questions
- Frequently asked questions
- A reporting checklist
- The detectable-effect view
- Final power audit
- Useful next reading
A researcher may care about a five-mark improvement, not every nonzero improvement. Power planning asks how often the intended analysis would detect that chosen effect under a complete probability model.
Type II error probability beta is the chance of failing to reject the null under the specified alternative. Power equals 1-beta. If beta is 0.20, power is 0.80.
The NIST/SEMATECH e-Handbook explains power as the probability of rejecting a false null hypothesis and links it to sample size, variability and the distance from the null. Read the NIST power guidance.
Suppose two independent groups will be compared on a continuous outcome. The meaningful mean difference is 5 marks, the common standard deviation is 10 marks, alpha is 0.05 two-sided and target power is 0.80.
The standardised difference is delta/sigma=5/10=0.5. Standardisation helps calculation, but the five-mark raw difference must remain visible because it carries educational meaning.
| Type II error beta | Power 1-beta |
|---|---|
| 0.2 | 0.8 |
For equal groups under a large-sample z approximation, n per group is approximately 2(z_(1-alpha/2)+z_(1-beta))^2 sigma^2/delta^2.
Using z_0.975=1.96 and z_0.80=0.84 gives 2(2.80)^2(10^2)/(5^2)=62.72, rounded up to 63 analysable participants per group.
Rounding down would miss the planning target. The result is an analysable sample, so anticipated exclusions and missing outcomes need a separate recruitment calculation.
| Difference | SD | Alpha | Power | Approx n per group |
|---|---|---|---|---|
| 5 | 10 | 0.05 | 0.80 | 63 |
A smaller target effect requires more information to distinguish it from noise. Halving delta while holding everything else fixed multiplies this simple sample-size approximation by four.
Doubling sigma while holding the raw difference fixed also multiplies n by four. Better measurement can improve power by reducing irrelevant noise, provided it does not change the target construct.
Larger samples reduce standard errors roughly with the square root of n. Doubling n does not double the signal-to-noise ratio; it multiplies it by about sqrt(2).
Raising alpha makes rejection easier and power higher, but also raises the Type I error rate under the null. The choice should be justified by consequences rather than selected to rescue a result.
A one-sided test concentrates alpha in one direction and can have more power for a prespecified directional alternative. It is inappropriate if a harmful effect in the other direction would matter.
Equal allocation is efficient when group costs and variances are similar. Different costs, variances or ethical constraints can justify unequal allocation, but the formula and total sample change.
If 20% attrition is expected, recruiting 63/0.80=78.75 means at least 79 per group under the simple assumption that attrition merely reduces sample size. Informative missingness can also bias estimates.
Students within the same class may be correlated. A design effect such as 1+(m-1)rho inflates variance, but cluster number and analysis degrees of freedom also matter.
A strong prespecified baseline predictor can reduce residual variance and improve power. Planning must use a defensible R-squared and the exact intended analysis rather than an optimistic guess.
Protecting a family of claims often lowers the per-comparison alpha or changes the critical value, reducing power for each claim unless sample size or design is adjusted.
Repeated interim looks need a planned stopping rule. Alpha-spending or group-sequential boundaries protect error rates while allowing early stopping for benefit, harm or futility.
When formulas do not match mixed models, missingness, nonlinear outcomes or adaptive rules, simulate data under the full design, run the exact analysis and count valid rejections.
A power curve plots rejection probability across a range of plausible effect sizes. It is more informative than quoting one power value without naming the alternative.
Vary the assumed standard deviation, intraclass correlation, attrition and effect. A robust plan shows how sample needs change instead of presenting one fragile number.
A design can have adequate power for a threshold yet still produce an interval too wide for practical decisions. Interval-width planning directly targets estimation precision.
After observing an effect and its standard error, power computed from the observed effect usually restates information already contained in the p-value. Report the estimate and confidence interval instead.
Low power means important effects may be missed and statistically significant estimates can be unstable. It does not mean every nonsignificant result is false or every significant result is exaggerated.
For a tutoring comparison, define the outcome, five-mark threshold, assessment reliability, class clustering, allocation, analysis and missing-data rules before choosing n.
Check the null and alternative, test direction, effect units, standard deviation, alpha, power, allocation, design effect, attrition, analysis model and rounding.
Reproduce 63 per group, halve the target effect, double the standard deviation, add 20% attrition and compare formula results with simulation.
Power calculations cannot repair biased sampling, invalid measurement, confounding, data-dependent analysis or an effect threshold chosen after results are seen.
Put assumptions in labelled cells, compute quantiles, sample size, attrition inflation and a sensitivity grid. Lock the direction and units beside the formula.
Write a function that simulates the planned design, fits the intended model, records valid rejections and reports Monte Carlo uncertainty for the estimated power.
Researchers, engineers, data scientists and programme evaluators use power to plan informative studies within ethical and financial constraints.
Choose a five-mark target and compare power curves for standard deviations of 8, 10 and 14, then explain which assumption drives the largest change.
Eighty percent power does not mean an 80% chance the hypothesis is true, low power does not prove no effect, and post-hoc power is not a substitute for an interval.
The sample-size planning guide covers survey precision. The statistical significance guide separates detection from importance.
Students should keep the meaningful raw effect beside any standardised effect. Parents can ask whether a school programme comparison allowed for classes, attrition and the size of improvement that would matter.
- Verify power=1-beta.
- Reproduce 63 per group.
- Inflate for 20% attrition.
- Add class clustering.
- Draw a power curve.
### Is power the chance the null is false? No.
### Does more data always solve bias? No.
### Can simulation estimate power? Yes.
### Should power be calculated after the result? Usually the estimate and interval are more informative.
Report the target effect and units, null and alternative, alpha and sidedness, target power, variability, allocation, design, attrition, analysis, software or simulation, rounding and sensitivity assumptions.
Instead of asking only for required n, fix a feasible sample and calculate the minimum effect the design can detect with stated power. Label it a design property, not the smallest important effect.
Confirm the alternative and raw units, exact analysis, error rates, design dependence, missingness, sensitivity range and reproducible calculation before recruitment begins.
Continue through the Mathematics Learning Hub. Recalculate one result, state its assumptions and explain which decision it can and cannot support.
Extended practice laboratory
Practice case 1: Define target effect
Setup: Start from the article’s power-analysis mechanism for case 1. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 2: Set alpha
Setup: Start from the article’s power-analysis mechanism for case 2. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 3: Set beta
Setup: Start from the article’s power-analysis mechanism for case 3. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 4: Calculate power
Setup: Start from the article’s power-analysis mechanism for case 4. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 5: Set standard deviation
Setup: Start from the article’s power-analysis mechanism for case 5. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 6: Set sample size
Setup: Start from the article’s power-analysis mechanism for case 6. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 7: Increase sample size
Setup: Start from the article’s power-analysis mechanism for case 7. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 8: Reduce noise
Setup: Start from the article’s power-analysis mechanism for case 8. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 9: Change effect size
Setup: Start from the article’s power-analysis mechanism for case 9. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 10: Choose one-sided test
Setup: Start from the article’s power-analysis mechanism for case 10. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 11: Choose two-sided test
Setup: Start from the article’s power-analysis mechanism for case 11. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 12: Add attrition
Setup: Start from the article’s power-analysis mechanism for case 12. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 13: Add clustering
Setup: Start from the article’s power-analysis mechanism for case 13. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 14: Add unequal allocation
Setup: Start from the article’s power-analysis mechanism for case 14. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 15: Add multiplicity
Setup: Start from the article’s power-analysis mechanism for case 15. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 16: Add covariate adjustment
Setup: Start from the article’s power-analysis mechanism for case 16. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 17: Simulate alternatives
Setup: Start from the article’s power-analysis mechanism for case 17. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 18: Inspect power curve
Setup: Start from the article’s power-analysis mechanism for case 18. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 19: Compare interval width
Setup: Start from the article’s power-analysis mechanism for case 19. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 20: Set practical threshold
Setup: Start from the article’s power-analysis mechanism for case 20. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 21: Audit design
Setup: Start from the article’s power-analysis mechanism for case 21. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 22: Audit assumptions
Setup: Start from the article’s power-analysis mechanism for case 22. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 23: Audit missingness
Setup: Start from the article’s power-analysis mechanism for case 23. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 24: Name limits
Setup: Start from the article’s power-analysis mechanism for case 24. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 25: Write conclusion
Setup: Start from the article’s power-analysis mechanism for case 25. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the target population and design, verify the statistical identity, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.