Did you know that one way of practising can show a better success rate in both easy and difficult questions, yet appear worse when the two groups are combined? A student may calculate every percentage correctly and still reach the wrong conclusion by ignoring how many attempts belong to each difficulty level. This is one reason that the total on a dashboard is not always the most useful learning signal.
Simpson’s paradox is a statistical pattern in which an association seen within separate groups changes direction or disappears when the groups are combined. The immediate answer is that group composition changes the weights in a combined average. To investigate a reversal, calculate the rate for each comparable group, show the numbers of attempts behind each rate, then explain why the combined figure gives a different comparison. A reversal is a prompt to investigate, not proof of a particular teaching cause.
- Name the outcome being measured and the groups used for comparison.
- Keep the count correct and the count attempted for each subgroup.
- Calculate subgroup percentages before producing a single total.
- Check how unequal subgroup sizes change the weight of each subgroup.
- Describe the reversal accurately and avoid concluding that one method caused the result without a suitable study design.
What can go wrong with an innocent overall average?
Imagine a school club comparing two ways of practising a quiz, called Practice A and Practice B. Some attempts use easy questions; others use difficult questions. If far more difficult items are assigned under A and far more easy items under B, the combined success rates describe both success and the different mix of difficulty. They do not isolate which practice approach works better.
A useful diagnostic is to ask a learner to choose the stronger practice approach from the overall totals, then reveal the difficulty breakdown and ask again. A student who says the second view is impossible may be assuming that an overall percentage must preserve every subgroup comparison. It need not.
Hand-checked worked example: two groups reverse the headline
The following numbers are fictional classroom data made only to explain the mathematics. They are not experimental evidence about any actual teaching method or learner. For easy questions, Practice A has 9 correct answers from 10 attempts: 9 ÷ 10 = 90%. Practice B has 24 correct answers from 30 attempts: 24 ÷ 30 = 80%. So A is higher in the easy group.
For difficult questions, A has 18 correct from 30 attempts: 18 ÷ 30 = 60%. B has 5 correct from 10 attempts: 5 ÷ 10 = 50%. A is also higher in the difficult group. So far there is no reversal.
Now combine the groups without changing a single answer. Practice A has 9 + 18 = 27 correct out of 40, which is 67.5%. Practice B has 24 + 5 = 29 correct out of 40, which is 72.5%. Suddenly B has the higher overall percentage, despite A’s higher result in both difficulty bands.
Check every denominator: A made 10 + 30 = 40 attempts; B made 30 + 10 = 40 attempts. These are equally sized overall samples but not equally distributed between the two question types. A’s total consists mostly of difficult questions; B’s mostly of easy ones. The combined rate is a weighted average of the group rates.
Why the arithmetic works: the hidden weights
A’s weighted rate is (10/40 × 90%) + (30/40 × 60%) = 22.5% + 45% = 67.5%. B’s is (30/40 × 80%) + (10/40 × 50%) = 60% + 12.5% = 72.5%. Notice how A weights the 60% difficult rate three times as heavily as the 90% easy rate, while B does the reverse.
This is not a computational mistake. All four subgroup rates and both totals are correct. The problem appears when someone treats the combined figure as if it measured practice quality alone. It also reflects the question mix. Depending on the decision, the relevant comparison might be matched difficulty groups, a standardised question mixture, or a properly designed controlled experiment.
Make a fairer comparison without hiding any results
For an accessible teaching demonstration, choose a hypothetical common mixture of 50% easy and 50% difficult questions for both practices. Under this stated weighting, A would score (90% + 60%) ÷ 2 = 75%, while B would score (80% + 50%) ÷ 2 = 65%. This is an illustrative standardised comparison, not the original observed total and not a new actual test result.
A standardised rate is meaningful only when the groups, definitions and weights are appropriate. A 50/50 mixture is not automatically the true distribution of work in a school or the fairest target. If a course contains 80% difficult material, a different common weighting may be justified. The important habit is to make the weight explicit and explain the purpose of the standardisation.
Do these numbers prove that Practice A causes better learning?
No. Even subgroup comparisons may be confounded. Perhaps students choosing A had more prior knowledge, different teachers, more practice time or other advantages. Or perhaps the ‘easy’ and ‘difficult’ labels conceal differences within each group. The worked example shows how aggregation can reverse an association; it does not establish an intervention’s causal effectiveness.
For a stronger study, a teacher might define equivalent tasks, consider random assignment where appropriate, record baseline skill and plan the analysis in advance. In everyday class reflection, however, the first useful step is simply to avoid overclaiming from two unexplained percentages.
Misconceptions worth diagnosing explicitly
- The overall percentage must lie between the two methods’ subgroup comparisons. Each method’s own weighted average lies between its own subgroup rates, but the ordering across methods can reverse.
- Equal overall sample sizes guarantee fairness. They do not guarantee equal difficulty composition.
- A subgroup with the most attempts is always the most important. Its weight matters, but educational importance depends on the stated question.
- Standardising the weights reveals the one true answer. It produces a comparison under chosen assumptions.
- A reversal proves the data were manipulated. It can arise naturally from uneven group composition.
A practice sequence that builds understanding
Guided check: calculate 9/10 and 24/30, then 18/30 and 5/10. Say which is larger in each category. Supported check: combine correct answers and attempts separately, rather than adding percentages. Show exactly why 90% + 60% cannot be treated as a combined rate without weighting.
Independent check: exchange A’s and B’s attempt allocations: give each 20 easy and 20 difficult attempts while retaining the same hypothetical subgroup rates. With these rates, A would have 18 easy correct and 12 difficult correct = 30/40 = 75%; B would have 16 easy correct and 10 difficult correct = 26/40 = 65%. All counts remain whole numbers in this particular exercise.
Transfer challenge: create a new two-category table in which the overall ranking differs from both category rankings. Verify each cell by hand. Then write three statements: what was observed, what arithmetic explains, and what a fairer next test would need.
How learners, parents and teachers should use this
For a pupil, the progress indicator is a spontaneous question: ‘How many easy and difficult attempts are behind the overall rate?’ For a teacher, display both counts and subgroup percentages on the same worksheet. For a parent, ask what kinds of questions were attempted before interpreting a week-to-week success percentage. The goal is not to distrust every headline but to learn when the headline leaves out a relevant composition difference.
Do not publish individual student performance or identify a struggling group in order to teach the concept. Fictionalised examples can show the entire mathematical mechanism without turning the class into a public ranking exercise.
Frequently asked questions
Is Simpson’s paradox a real mathematical phenomenon? Yes. It is a well-established feature of conditional versus aggregated statistical comparisons. Penn State’s statistics teaching materials describe this type of reversal.
Is the overall rate wrong? No. Here 67.5% and 72.5% are correct overall rates. The mistake would be assigning them a meaning they cannot establish.
Should we always split data into smaller groups? No. Choose relevant, interpretable groups based on the question. Tiny subgroups can create instability or hide uncertainty.
Does a reversal happen only when totals differ? No. The worked example uses forty attempts in both combined totals. The unequal allocation within those totals is sufficient.
Continue learning with clear denominators
For a related lesson on percentages when a class size changes, visit how to compare quiz results when class size changes. See also the eduKate Education Hub and our Clementi mathematics tutorial reference for the progression from identifying a misconception to independently checking evidence.
The core aim of education is not to teach students to repeat a surprising name. It is to give them a reliable habit: before accepting a combined result, ask what was combined and how it was weighted. The arithmetic is only the beginning of the explanation.