Did you know that the study group with the higher final test score may have improved less? Two pupils can work equally hard and finish at different marks because they started at different points. A neat final ranking can hide the very learning progress that parents and teachers want to understand.
A fair comparison of study methods examines learners’ starting knowledge, the learning opportunities they received, and outcomes measured in a comparable way. Baseline means what learners knew before the study period; progress means change on a measure that can legitimately be compared over time. Comparing final marks alone, or raw gains alone, does not by itself prove one method caused better learning. We need both the mathematics and a cautious explanation of what the design can support.
The real educational question is not simply who finished first
Suppose two after-school groups try two approaches to learning a set of unfamiliar concepts. Group A uses practice questions with feedback; Group B makes summary notes. At the end, Group A averages 83 marks while Group B averages 61. It is tempting to declare the first method better. But the groups may not have started at the same knowledge level, and that fact changes the interpretation.
There are at least three questions to separate. What did each learner know before the lessons? How did each learner’s knowledge or skill change on a suitably comparable assessment? And can that change reasonably be attributed to the method rather than group selection, attendance, teacher support, time spent, or ordinary practice effects? The first two can be addressed descriptively. The third requires stronger evidence.
This distinction matters even when a classroom is too small for formal research. A teacher can still use baseline evidence to plan better next lessons; a parent can still observe whether a learner is more independent. Those useful decisions do not require pretending that a small home experiment is a controlled effectiveness study.
Diagnose the mistake with one minute of arithmetic
Ask a learner: “Method A pupils average 80 after teaching, and Method B pupils average 65. Which method helped more?” If they immediately answer A, show that A started at 76 and B started at 42 on the same 100-point skill measure. A increased by 4 points and B increased by 23. Now ask a harder question: “Which method caused more learning?” The responsible answer is that the descriptive gains favour B, but the information is still insufficient to establish the cause.
- Final-score shortcut: assuming the higher endpoint must come from the better technique.
- Gain-score shortcut: treating a larger change as automatically caused by the intervention despite unequal starting groups.
- Scale error: subtracting a mark out of 30 from a mark out of 50 as if both were the same test.
- Practice effect: forgetting that a pupil may improve because they have seen similar questions before.
- Ceiling effect: overlooking that a learner starting near the top has fewer available points to gain.
- Dropout error: comparing only those who completed both assessments without mentioning absent learners.
A completely worked fictional example
All scores in this section are illustrative invented data, not real pupil results, research findings or promises of what tuition will achieve. Imagine that four learners in Group A begin with pretest scores of 70, 72, 74 and 76 out of 100. After a fortnight of Method A, they score 80, 82, 84 and 86 on a suitable follow-up measure using the same meaningful scale. The starting mean is (70 + 72 + 74 + 76) ÷ 4 = 292 ÷ 4 = 73. The ending mean is (80 + 82 + 84 + 86) ÷ 4 = 332 ÷ 4 = 83. Average gain: 83 − 73 = 10 points.
Four learners in Group B begin at 40, 42, 44 and 46 out of 100 and later obtain 58, 60, 62 and 64. Their starting mean is (40 + 42 + 44 + 46) ÷ 4 = 172 ÷ 4 = 43. Their ending mean is (58 + 60 + 62 + 64) ÷ 4 = 244 ÷ 4 = 61. Their average gain is 61 − 43 = 18 points.
Two true descriptions now coexist. Group A has the higher ending mean: 83 versus 61. Group B has the larger mean change: 18 versus 10 points. Neither fact cancels the other. The first describes attained performance on this measure; the second describes improvement on this measure. They answer different questions.
Check the group-gain calculation a second way: Group A individual gains are 10, 10, 10 and 10, averaging 10; Group B gains are 18, 18, 18 and 18, averaging 18. These deliberately clean values make the arithmetic transparent, not realistic variability. Actual pupils rarely progress in identical increments.
Why the larger gain still does not settle the study-method contest
Learners in Group B began with much more headroom on a 100-point assessment. A pupil starting at 96 cannot gain 18 marks on that scale, however much the pupil learns beyond the sampled questions. Conversely, a low first score may reflect unfamiliar instructions, distraction or a chance mistake; a later higher result may partly reflect regression toward the pupil’s usual performance. A raw gain does not tell us how much additional understanding a method caused.
Groups may also differ in motivation, prior teaching, teacher attention, home practice, language proficiency, attendance or the difficulty of questions each learner attempted. If learners chose their own method, the choice itself may be associated with prior skill or confidence. Strong comparisons try to make groups similar at the start—or allocate learners fairly—and use valid measures, sufficient participants and an analysis that respects starting differences.
The US Institute of Education Sciences’ What Works Clearinghouse glossary distinguishes simple gain scores from the stronger design and analytical requirements needed in formal group studies. That is a useful warning for readers: subtracting a pretest from a posttest is informative classroom arithmetic, not proof of causal impact. For everyday teaching, use those gains to ask better questions rather than to announce a winning brand of instruction.
Match the measurement to the skill you care about
A method intended to help pupils recall definitions should be assessed on accurate recall after an appropriate interval. A method intended to strengthen reasoning should be assessed using novel applications that demand explanation. A pupil who can copy an example quickly might still struggle with a new situation. If your test repeats the same examples practised by one group, it may favour familiarity rather than independent transfer.
Two forms of a test should cover comparable content, difficulty and scoring standards. Identical maximum marks are necessary for simple raw-score comparison but not sufficient for equivalent difficulty. Do not merely convert marks to percentages and assume all scales are interchangeable. If the before and after instruments differ materially, avoid a naive subtraction, and explain the limitation.
Also decide when to check learning: immediately after practice, a few days later, and on an unfamiliar problem. The immediate test shows short-term performance; delayed retrieval can reveal what remains accessible; transfer tasks examine whether the learner can apply the idea in a new context. Each outcome may tell a different story about a study approach.
A more careful matched illustration
Consider two pupils at a similar starting point: Learner A scores 52 before Method A and 66 after, a gain of 14. Learner B scores 51 before Method B and 64 after, a gain of 13. This is a closer starting comparison than the earlier groups, and A’s gain is one point higher. But two pupils are too few to conclude that the method is superior; each learner may be affected differently by sleep, task familiarity, effort or ordinary variability.
Now consider a higher-starting pair: Learner C goes from 78 to 84 under Method A, a gain of 6; Learner D goes from 79 to 85 under Method B, also a gain of 6. Across these four illustrative learners, the gains do not support a large dramatic method difference. The important lesson is not which letter wins. It is why grouping, starting level, comparable questions and error analysis matter.
An independent teacher might examine work samples: Could the pupil explain a rule in their own words, correct a previously repeated error and apply the principle without a worked example? Those observations can help refine teaching even without producing a statistically defensible comparison of methods.
A practical learning comparison a small class can honestly carry out
- Choose one narrow learning goal. For example, distinguish an algebraic expression from an equation and justify the distinction.
- Collect a baseline. Give a short diagnostic with the same instructions and fair scoring for all pupils.
- Plan the teaching conditions. Record practice time, assistance, topic sequence and attendance; do not call conditions identical unless they are.
- Use a comparable follow-up. Include new examples so success cannot come solely from remembering an answer pattern.
- Show three views. Starting score, later score and observed change, together with item-level error patterns.
- Check transfer and retention. Include an unfamiliar question and a later recall task.
- State the evidence boundary. Describe what the learners did; avoid declaring a generalised winner from a tiny unrandomised group.
Guided practice: read the scores, then challenge the claim
Step one: do the subtraction accurately
A fictional learner scores 14/20 on the first assessment and 18/20 on a comparable follow-up. The raw gain is 18 − 14 = 4 marks out of 20, equivalent to a change from 70% to 90%, or 20 percentage points. The score increased by 20 percentage points; that is not the same wording as “20 percent more learning”. The percentage change in the score relative to the starting score would be a different calculation and would not directly quantify knowledge gained.
Step two: examine the unequal-baseline trap
A second learner begins at 5/20 and later scores 14/20, a gain of 9 marks. Ask which learner finished higher (the first) and which improved more in raw marks (the second). Then ask whether the second method is necessarily better (not established). The learner must give all three answers, rather than choosing one slogan.
Step three: find the testing flaw
Imagine the first group has practised the exact ten final questions, while the second group sees new versions of the same concepts. Even equal final marks would be hard to interpret. The skill is identifying the unfair measurement before calculating an attractive number.
Step four: make a learner-designed comparison
Let the learner propose two study routines for one topic, including what is measured beforehand, how practice exposure is tracked, when the new task is used and how missed lessons are recorded. Ask them to write two possible conclusions: one narrowly supported by small classroom data and one claim the evidence cannot support. Independent judgement is the assessed outcome.
For parents: look for transfer, not a competition of labels
Parents often ask whether flashcards, note-making, videos, worksheets or tutoring is “best”. It is more helpful to ask what particular difficulty the child has and what evidence would show improvement. A pupil who cannot retrieve a definition may benefit from retrieval practice; a pupil who can recite a definition but cannot use it needs application practice. The right next step depends on the diagnosed gap and how the learner responds.
Keep a small record of a previously missed question type, the correction made, and the next independent attempt. Do not turn the child into an ongoing competition against peers. A useful progress report might say: “The learner now solves unfamiliar two-step equations with correct substitution checks on four of five practice items.” It describes visible work and still leaves room for variability.
For teachers: use evidence to change instruction
The IES review of formative assessment describes assessment as gathering and interpreting evidence so teaching can be adjusted while learning is in progress. A formative check need not become a quasi-experimental claim. If learners remain confused by negative-number substitution, the response is a targeted explanation and new questions—not a declaration that a whole method has failed.
Write down whether gains are shown for the same participating learners, whether missing assessments are clearly marked and whether scoring criteria changed. If a pupil is absent for the later test, do not silently give zero and call it learning loss; nor should that pupil disappear from the class roster without any note. Small bookkeeping decisions can change the story a graph tells.
What genuine improvement looks like
- A learner can compute starting, ending and gain scores with units and test maxima identified.
- They can describe the higher endpoint and larger gain separately when those belong to different groups.
- They can explain at least two alternative reasons for an apparent improvement.
- They can use a new task to check transfer, not merely reproduce a memorised answer.
- They can identify an unmeasured or unequal condition that limits a study-method comparison.
- They can propose a next lesson responding to the learner’s actual error, rather than a ranking of products.
Frequently asked questions
Is percentage gain always fairer than raw marks?
No. Percentages can help when the scoring scale is genuinely comparable, but unequal difficulty or changing constructs remain problems. Relative percentage changes can also exaggerate differences for low starting values. State the measure and why comparison is legitimate.
Can the pupil’s own past score be a useful baseline?
Yes, for tracking progress on comparable tasks. It does not automatically establish that a particular method was the cause; other learning and ordinary practice may have occurred in the same period.
What if a pupil already scores 95/100?
A near-ceiling test leaves little visible room to improve. Use more challenging, conceptually appropriate items or independent transfer tasks instead of assuming that only five points of further learning are possible.
How many pupils do we need to prove one method is better?
There is no universal number that makes an uncontrolled comparison causal. Appropriate design, outcome validity, variation, assignment, attrition and statistical power all matter. A small class can guide its own next steps without making population-wide claims.
Read further and keep the purpose in view
For a related but different measurement issue, see the Core Aim guide to pretest and posttest papers with unequal maximum marks. Explore practical study approaches, the eduKate Education Hub and the diagnosis-first teaching reference for explanation, progressive practice and independent checking.
The core aim of education is to discover what a learner can do next. A score comparison should help us notice new capability, unfinished understanding and the next useful question—not reward the group that happened to arrive with the biggest head start.