Did you know that a class can improve after a new study routine even when the routine is not the only reason for the improvement? The new plan may be genuinely useful, but time, extra practice, growing familiarity with a test or changes elsewhere may also be involved. A report that says ‘scores went up, therefore our programme worked’ skips an important question.
The immediate answer: a before-and-after study compares a measure taken before a change with the same measure after it. If the goal is to describe what happened, those two observations may be enough for a careful description. If the goal is to decide whether the change caused
- Separate the descriptive question (‘what changed?’) from the causal question (‘why did it change?’).
- Write the intervention, population, dates and outcome measure clearly.
- Confirm that the before and after measures are genuinely comparable.
- Measure change in the intervention group using the same unit throughout.
- When suitable, compare that change with the change in a group not exposed to the intervention.
- Check selection differences, shared trends, contamination, missing data and other explanations.
- Report what the design supports, rather than promising success from an attractive difference.
What before-and-after evidence can and cannot show
Suppose students try a new reading routine for four weeks. On the first day they take a short comprehension activity. On the last day, they complete a comparable activity. If the average rises, that is a descriptive observation: scores rose between two time points. It is not yet a demonstration that the reading routine created the entire change. Students may have received other teaching, become more comfortable with the format or taken an easier second assessment.
The logic is counterfactual: to attribute an effect, we want to know what would probably have happened to the same learners during the same period without the new routine. We cannot observe that alternative history directly. A comparison group, careful timing and suitable study methods are ways to estimate it imperfectly. This is why responsible educational research does more than put two bars on a chart.
One complete fictional example: a new reading routine
Invented school-club study: Group A uses a fifteen-minute recall-and-explain routine during a four-week period. Group B continues its usual reading practice. Both groups take a comparable scoring exercise before and after. Assume both scores are expressed on the same 0–100-point scale, larger numbers represent better performance, and the summary statistics are correctly calculated. These are illustrative figures, not actual pupil outcomes or evidence that a specific teaching product works.
- Group A, with the new routine: average before = 60 points; after = 72 points.
- Group B, usual routine: average before = 62 points; after = 69 points.
- Reporting interval: four weeks; both groups evaluated at the start and end.
- Question: does Group A’s change look different from the change occurring during the same period in Group B?
Step 1: calculate each group’s change separately
Group A’s increase = 72 − 60 = 12 points. Group B’s increase = 69 − 62 = 7 points. The first factual statement is that both groups improved. The second is that A’s numerical increase was larger by five points. Do not replace the word ‘points’ with ‘percent’ unless the scores are specifically percentages and the meaning is explained.
Step 2: compare the changes, not just the ending scores
Difference in changes = (72 − 60) − (69 − 62) = 12 − 7 = 5 points. In a simplified teaching example this is the arithmetic behind a difference-in-differences comparison. The end-only difference, 72 − 69 = 3 points, answers a different question because A started two points lower: 60 − 62 = −2 points. The relative before-to-after movement must account for that initial difference.
Step 3: explain what the five-point contrast means
Group A’s average rose by five points more than Group B’s average over this period. It is tempting to call those five points ‘the effect of the new routine’. That causal interpretation needs assumptions: the groups would have followed reasonably comparable trends without the change; the tests measure the same construct on a comparable scale; group membership and other conditions do not distort the comparison; and the observations are reliable. One before measurement alone cannot demonstrate that the underlying trends were parallel.
Step 4: make a cautious conclusion
A sound classroom report might say: ‘In this fictional example, Group A improved by 12 points and Group B by 7 points, a five-point greater increase for Group A. This pattern is consistent with a possible benefit from the routine, but the design does not prove it caused the difference.’ This sentence answers the arithmetic question while keeping the evidence stronger than the rhetoric.
Why a comparison group matters
Imagine a new reading routine begins just as all learners receive a school-wide literacy programme. A one-group before-and-after chart cannot separate the new routine from that larger change. If a reasonably comparable group did not use the routine but still improved, that observation alerts us to background influences. A comparison group therefore provides a more informative reference than assuming that the only thing that changed was the headline intervention.
Yet the comparison must be meaningful. If Group A receives extra tutor time, new books, smaller group instruction and a new routine, while Group B receives none of those supports, the study bundles several differences. A single result cannot isolate the routine’s effect. Similarly, if the groups are in different schools, use different tests or include learners with sharply different needs, selection and measurement differences may explain part of the contrast.
The hidden question: were the groups already moving differently?
Two groups can show similar starting scores while following different learning trajectories. For example, imagine A’s earlier averages were 50, 55 and 60, while B’s earlier averages were 60, 61 and 62. Before the new routine starts, A is already improving faster. Comparing only the final pre-test to the post-test would risk crediting the intervention for momentum that existed beforehand. Earlier comparable measurements can reveal such pre-existing trends.
With a very small study, it may not be possible to estimate a stable trend, and noisy test scores can make the change look larger or smaller than it is. A comparison group is useful, not magical. Random assignment, where feasible and ethically appropriate, can reduce some selection problems; careful repeated measures, suitable controls and transparent reporting are also important. The strongest design depends on the question, resources and ethical constraints.
Learner diagnostic: mistakes that tell a teacher what to reteach
- Post-score shortcut: the learner subtracts 69 from 72 and calls three points ‘the improvement’. Ask them to calculate A’s actual before-after change.
- Single-group overclaim: the learner says all twelve points must come from the routine. Ask what changed for B in the same period.
- Wrong comparison: the learner subtracts the two starting scores but never compares changes. Ask which difference tests the actual question.
- Measurement mismatch: the learner compares a 10-question quiz with a 50-question test as raw scores. Ask for comparable scores or a valid common scale.
- Selection blind spot: the learner assumes A and B are interchangeable because their averages are close. Ask about composition, teaching and earlier trends.
- Certainty inflation: the learner treats a five-point contrast as proof. Ask what the design cannot rule out.
- Missing-data blindness: the learner includes only pupils who completed the final test and ignores why others are absent. Ask how loss to follow-up might change the result.
These errors reflect different reasoning gaps. An arithmetic mistake needs a two-column change table; a causal mistake needs an alternative-explanation discussion; a measurement mistake needs an assessment audit. Good tutoring identifies which misunderstanding is present instead of assigning twenty more generic percentage problems.
A simple study-design checklist before collecting data
- Define one outcome: specify whether the target is an assessment score, reading time, recall accuracy or another measure.
- Plan comparable assessments: keep construct, scoring and difficulty sufficiently aligned or explain differences.
- Define groups: record how learners enter each group; do not imply random assignment unless it actually happened.
- Document other changes: note tutor time, extra homework, resources, schedule disruptions and outside support.
- Protect participants: use consent, appropriate permissions and privacy-preserving, pseudonymous records.
- Keep all eligible results: document missing scores without treating them automatically as zero.
- Decide reporting language in advance: say what counts as a descriptive finding and what extra evidence would support a causal claim.
Progressive practice and independent transfer
Stage 1 — one group, descriptive only
A fictional group’s mean rises from 40 to 46 points. The observed increase is 6 points. Ask: ‘Can we say the scores rose?’ Yes. ‘Can we conclude a teaching change caused six points?’ Not from these observations alone. The student must state at least two other plausible influences.
Stage 2 — compare two changes
Group X rises from 50 to 58 (+8). Group Y rises from 51 to 55 (+4). The difference in changes is 4 points. A correct explanation describes how each group’s score moved; it does not merely compare 58 and 55. Then ask what evidence would make the comparison more trustworthy.
Stage 3 — notice a misleading improvement
A club’s after-score is 75 versus a before-score of 65: +10. Its comparison club rises from 60 to 74: +14. The first club’s change is 4 points smaller than the comparison change (10 − 14 = −4). The student should recognise why the statement ‘our club improved ten points, therefore our method was superior’ is unsupported.
Stage 4 — separate effects of test difficulty
Suppose the first assessment has simple passages and the second has much harder passages, with both reported as percentages. A different score might reflect changed difficulty rather than changed comprehension. Ask the learner how to demonstrate comparability, perhaps by using appropriately parallel forms or a sound common measurement scale. A neat subtraction cannot repair a badly matched instrument.
Stage 5 — design a defensible small investigation
Students propose an ethical, low-stakes class investigation into a revision aid. They specify a question, groups, measurements, assignment process, duration, potential confounders and a privacy plan. A peer must identify a plausible alternative explanation. The final report uses conditional language and respects school policies rather than claiming that a short exercise proves educational effectiveness.
What parents and teachers can observe
Parents need not run an experiment on their children to benefit from this reasoning. When a programme claims ‘the average improved after four weeks’, ask whether the same learners were measured consistently, what else happened during the period and whether a comparable group or longer trend was examined. These are questions about evidence quality, not an accusation that improvement is unimportant.
Teachers can mark a short report with distinct checks: arithmetic of each change, fair comparison, identification of confounders, treatment of missing data and calibrated wording of the conclusion. A strong learner might improve from an unsupported ‘proved’ to ‘observed’ and explain what additional design would be needed. That is measurable progress in scientific and statistical literacy, independent of any claimed mark increase.
Limitations, ethics and caution
The difference-in-differences calculation here is an introduction, not a full causal-estimation procedure. With real data, analysts examine group comparability, prior trends, uncertainty, varying exposure, cluster effects, assessment reliability and possible spillovers. A comparison group can still be systematically different. Asking a pupil to adopt a new routine is not a substitute for an approved study design, and nobody should withhold needed educational help solely to create a convenient control group.
The fictional figures are group averages, not results for real children. They do not include individual score distributions, sample sizes, error bars or confidence intervals; therefore no claim of statistical significance is supported. We should not infer that all individuals improved from a rising group average. An ethical report protects learner identities and states limitations alongside results.
Frequently asked questions
If scores rise, isn’t that evidence the programme worked?
It is evidence of improvement over the measured period. Causal attribution requires addressing alternative explanations, ideally with a suitable design and comparison.
Does a comparison group make a result automatically fair?
No. Groups must be meaningfully comparable, measured consistently and checked for confounding and different pre-existing trends.
Why compare changes instead of only the final scores?
Because groups can start from different levels. Subtracting each group’s starting score helps reveal how each moved over the period, though it does not by itself solve all bias.
What is difference-in-differences in simple words?
It is the change in one group minus the change in another group. Interpreting that number as an effect requires additional design assumptions.
Do we have to test every school idea scientifically?
No. Many classroom changes are adopted on practical, ethical or pedagogical grounds. But when making a strong causal claim, the quality of the evidence should match the strength of that claim.
The deeper educational aim
The point of research literacy is not to reject every promising improvement. It is to celebrate accurate observations while asking which conclusion they actually support. A learner who distinguishes ‘improved after’ from ‘improved because of’ can interpret school projects, public claims and personal study experiments more responsibly. That is a transferable habit of intellectual honesty.
For connected reading, consult the Core Aim articles about comparing study methods with different starting levels and independent experimental trials, the eduKateSG Education Hub, the specialist comprehension guide for reading-skills context and the Clementi teaching reference for progressive diagnosis and practice.
For authoritative research-design context, see the US Centers for Disease Control and Prevention journal article on interpreting before-and-after studies, comparison groups and confounding. All group scores and changes above are invented and arithmetically checked, and no real learner outcome is implied.