EDUKATE SINGAPORE · GOLD STANDARD PRACTICAL CASE STUDY 217
A small p-value cannot give the probability that a teaching hypothesis is true
A fictional school finds p = 0.04 and claims its new algebra teaching method is 96% likely to work. This confuses a statistical result conditional on a model with the probability a hypothesis is true. The correct school response is to keep the measured result, correct the meaning and examine whether the educational conclusion is justified.
All participants and data are invented. The statistical reasoning is informed by the American Statistical Association; no actual school or pupil outcome is being evaluated.
The decision in brief
- Reported hypothetical p-value: 0.04, relative to a specified statistical test.
- Incorrect headline: 96% probability that the method is true or better.
- Appropriate conditional reading: results at least as incompatible with the null model arise about 4% of the time when the model and test assumptions hold.
- The result does not determine effect size, causality, long-term educational value or probability of a hypothesis.
- Next step: disclose the design, effect, uncertainty and relevant alternatives before making a recommendation.
Chapter navigation
- 1. The report that turned a statistic into a promise
- 2. What p = 0.04 actually tells the researcher
- 3. The ASA’s six principles reject the 96% reading
- 4. Correct arithmetic can still answer the wrong question
- 5. Statistical detectability is not the size of the teaching benefit
- 6. The study design decides what can be inferred
- 7. Twenty independent tests and the risk of a striking result
- 8. The 0.05 line is not a truth boundary
- 9. How the school can correct its public claim
- 10. Exercise: interpret the number before judging the method
- 11. Decision record: keep the result, reject the overclaim
1. The report that turned a statistic into a promise
A fictional school compares two ways of teaching secondary algebra. Its analyst reports p = 0.04 from a defined statistical test and a presentation calls the new teaching method ‘96% likely to be better’. Although 1 − 0.04 = 0.96 is correct arithmetic, the conclusion does not follow from the statistic. The p-value answers a different, conditional question.
School leaders may be pleased with the new teaching routine and the work involved in collecting evidence. A correction need not dismiss the project. It should distinguish the reported test result from the broader claim the slide makes to parents, teachers and students.
All participants, methods and findings in this case are fictional. The article examines how a valid calculation can be transformed into a misleading public statement without anyone intentionally falsifying an observation. The school must find the step where that happened before using the figure in a decision.
2. What p = 0.04 actually tells the researcher
A p-value summarises the compatibility of observed data with a specified statistical model. Broadly, p = 0.04 means that, if the stipulated null model and testing assumptions applied, a result at least as extreme as the observed test statistic would arise in about four percent of repetitions of that procedure.
This is not the probability that the null hypothesis is true, nor the probability that the school team’s favoured explanation is true. The form of the condition matters. The question about possible data under an assumed model cannot be automatically inverted into a probability for a hypothesis given actual data.
The school can make the wording more accurate simply by naming the null model and the outcome tested. Saying ‘assuming our specified model’ reduces the temptation to treat a computed number as an unconditional proof about teaching. The interpretation still depends on whether the model and research design are appropriate.
Similarly, ‘only a four percent chance the improvement happened by luck’ is not a faithful translation. A p-value does not allocate four percent of an observed effect to chance or determine which explanation caused the difference.
3. The ASA’s six principles reject the 96% reading
The American Statistical Association’s statement announcement explains the misunderstandings around statistical significance. Its published statement on p-values specifically rejects interpreting p-values as probabilities that the studied hypotheses are true.
The ASA also warns against making scientific or policy decisions based only on crossing a threshold such as 0.05. A small p-value does not describe effect size, practical importance or the quality of an experiment. Those are separate parts of an informed conclusion.
These publications do not assess the imagined school. They establish a methodological standard for communicating a result honestly. Readers should understand what data were collected, what model was tested and what the result cannot prove.
A well-designed school pilot may offer promising evidence. Its value is not improved by exaggerating a 0.04 into a 96% truth guarantee; the exaggeration instead makes the evidence harder to trust.
4. Correct arithmetic can still answer the wrong question
The calculation 100% − 4% = 96% is numerically sound, but it subtracts quantities with the wrong meanings. A familiar-looking percentage does not become a posterior probability merely because a presenter wants a headline.
If researchers want the probability of a hypothesis after seeing evidence, they need an explicitly suitable inferential framework with additional assumptions. Conventional p-values do not directly supply that result, and a school must not smuggle a new framework into an ordinary significance-test report.
Students often learn that algebra begins by defining quantities before manipulating them. Statistics demands the same discipline. First ask what a figure represents; only then decide which operations are meaningful. A beautifully formatted percentage still fails if its numerator, denominator or condition is misunderstood.
The correction should preserve the p-value in the technical account while withdrawing its invalid interpretation from the public summary. Both steps matter: concealment damages reproducibility, while leaving the faulty headline damages readers’ understanding.
5. Statistical detectability is not the size of the teaching benefit
Suppose the fictional school also reports class means of 78 and 74 on a 100-point algebra quiz. Their descriptive difference is four points. Those two means cannot by themselves reproduce the stated p = 0.04; the data’s variation, participant numbers, allocation and test details are not provided. The means are separate illustrative figures, not a hidden derivation of the p-value.
Whether four points matter depends on what the quiz measures and the target students’ needs. Teachers should examine performance on unfamiliar algebra, sustained retention and variation between learners. An average improvement can conceal some pupils who made little progress or pupils who benefited substantially.
An appropriate effect estimate and uncertainty interval address different questions from the p-value. A useful school report should supply these quantities where justified and avoid claiming that a small p-value makes a tiny improvement transformational.
The Gold Standard asks for practical meaning as well as numerical interpretation. Statistical significance is not a substitute for better learning, nor does a single nonsignificant test prove an educational method useless.
6. The study design decides what can be inferred
If more motivated pupils selected the new class while others remained in the standard routine, apparent benefits may reflect prior performance, home support or teacher differences. A p-value does not remove those possible influences. Randomisation can help in suitable designs, while observational comparisons need their own controls and limits.
Reviewers should look at baseline scores, missing assessments, attendance, group sizes, selection rules and whether students nested in a class were treated appropriately in analysis. A group with a different teacher may also receive different amounts of individual feedback, which could change outcomes independently of the labelled method.
A primary outcome selected before data collection has a different status from whichever result happened to look best after many comparisons. The school should disclose planned and exploratory outcomes rather than presenting an opportunistically selected result as its sole original target.
Any public statement should match the actual outcome and design. If the test measured a two-week quiz, it cannot alone establish long-term examination distinctions for every student or a causal result in a new school system.
7. Twenty independent tests and the risk of a striking result
Imagine twenty separate, valid and independent null-hypothesis tests, all with true null models, each assessed at a 0.05 threshold. The probability that none crosses the threshold is (0.95)^20, approximately 35.8%. The probability that at least one crosses it is 1 − (0.95)^20, approximately 64.2%.
This is a hypothetical illustration under independence and valid-test assumptions. It does not say there is a 64.2% probability that the algebra method is false. Real outcomes often correlate; suitable multiplicity adjustments and careful interpretation depend on the study’s full structure.
The calculation explains why a lone result can look compelling when a reader has not been told about many analyses attempted. Schools should explain the number of comparisons, analysis plan and any corrections or replication that support the finding.
Exploratory studies can still teach us much. Their findings can inform better questions and future tests as long as they are not presented as definitive confirmation merely because a threshold was crossed.
8. The 0.05 line is not a truth boundary
A reported p-value of 0.049 is close to 0.051. A prespecified rule may treat the results differently, but the real evidence changes smoothly rather than suddenly from proved to disproved. Researchers must not turn the threshold into a certificate of truth.
A later comparison may yield a similar estimated effect but a different p-value because of sample size, variance, test design or measurement noise. The sensible response is to examine the complete evidence, not claim that the teaching method physically changed at the decimal point.
The ASA encourages reporting beyond a single index: study quality, estimates, uncertainty, scientific context and transparency all matter. For a school, curriculum decisions also have consequences for pupil workload, lesson time and equal access.
A restrained description is more useful than a binary sticker. ‘Promising preliminary signal’ has an understandable meaning when followed by what was observed and what remains unknown.
9. How the school can correct its public claim
A revised statement could say: ‘Our fictional pilot produced p = 0.04 from a specified analysis. This does not mean there is a 96% probability that the new teaching method is better. We are reviewing effect estimates, uncertainty, study quality and longer-term learning outcomes before making a broader recommendation.’
The project team should retain the original data, statistical code and earlier presentation, together with a dated correction. If old materials still circulate, the correction needs to reach those channels as well. A hidden technical appendix is insufficient when the main slide remains misleading.
An education leader can acknowledge enthusiastic teachers while insisting on careful evidence. The issue is not whether staff tried to help students; it is whether the chosen statistic supports the particular claim placed in front of parents.
The next stage might be a monitored continuation or independent replication. Its governance should identify the target outcomes, learners, costs and conditions under which the evidence would justify expanding the approach.
10. Exercise: interpret the number before judging the method
Give students a card marked ‘p = 0.04’. Ask them to choose between two statements: ‘Results this incompatible with a specified null model occur about four percent of the time under repeated valid tests’ and ‘There is a 96% chance the teaching hypothesis is true’. The first matches the intended conditional interpretation; the second does not follow.
Next introduce fictional mean quiz marks of 78 and 74, then ask whether the means are enough to compute the reported p-value. They are not. Students must identify missing sample sizes, distributions, assumptions, comparison procedures and test details rather than guess them.
Older learners can verify the twenty-test calculation and state the independence assumption. Younger learners can rewrite the overconfident headline into a clear sentence for parents. Neither group needs to decide whether the invented intervention actually works.
The exercise teaches a habit transferable to marketing claims, medical research, public policy and everyday news: check exactly what a number measures before accepting a conclusion based on it.
11. Decision record: keep the result, reject the overclaim
The fictional board can record that the analysis reported p = 0.04 and that the 96% hypothesis-probability statement is unsupported. It should commission a review of the actual test design and the educational significance of the outcomes rather than either blindly adopting or immediately discarding the new routine.
If additional evidence supports a benefit, the school can describe the measured result and its limits. If it does not, the team should revise its teaching plan. Neither outcome erases the value of learning how a statistic can be misinterpreted.
A quality audit should verify that the corrected publication distinguishes prespecified evidence from exploratory analysis, names the effect and uncertainty and does not promise grades that were never studied.
The Gold Standard is the ability to tell parents what p = 0.04 means, what it does not mean and what should happen next. That is stronger than a dramatic but mathematically unsupported 96% guarantee.
Evidence and reporting checklist
- State what the p-value tests and which null model was assumed.
- Report the estimate, uncertainty, group design and missing data.
- Disclose planned versus exploratory outcomes and the number of analyses.
- Avoid converting a p-value into a hypothesis truth probability.
- Assess practical educational value and replication separately.
- Preserve a correction trail for earlier misleading claims.
Frequently asked questions
Is p = 0.04 a 96% chance a teaching method works?
No. A p-value is about the data relative to a specified model, not the probability that the preferred hypothesis is true.
Does a small p-value prove a large benefit?
No. Practical effect size and uncertainty need separate investigation.
Can a result just above p = 0.05 still matter?
Yes. The conventional threshold is not a boundary between truth and falsehood.
Why report multiple comparisons?
Searching many analyses can increase the chance of finding a small p-value, making transparent reporting essential.
Can the school run a further pilot?
It can consider a monitored follow-up under appropriate educational governance, without claiming certainty that the existing data do not establish.
Authoritative references and connected cases
- American Statistical Association · 2016 statement announcement
- ASA Statement on P-Values: Context, Process, and Purpose
- The Gold Standard of Statistics
- The Gold Standard of Education
- Gold Standard Case 209 · Standard deviation versus confidence interval
- Gold Standard Case 216 · UTC deadline and local-time confusion
- Gold Standard · Core Requirements and Evidence
- Gold Standard · Casebook
All study results, groups and examples are hypothetical; the quiz means and reported p-value are not presented as one reproducible raw dataset. The eduKate Gold Standard framework is for evidence literacy, not statistical accreditation.