eduKateSG Learning Node Series · 0098

How Improvement Measurement Works | Use Measures to Learn Without Letting the Metric Become the Mission

A school changes the way it teaches reading. Scores rise. Is the change working?

Perhaps. Or the cohort was different. Or the assessment was easier. Or only already-strong readers improved. Or teachers spent unsustainable amounts of time producing the result. Or the score moved once and then returned to baseline. Or the process that supposedly caused the improvement was never used consistently.

Measurement for improvement begins where the headline number stops. Its purpose is not to prove that leaders were right. Its purpose is to help a team learn whether a specific change is producing better performance, for whom, under what conditions, at what cost and with what unintended consequences.

A measure is useful when it changes the next decision.

The 50-Second Read

• Measurement for improvement is different from measurement for accountability, certification or research.
• Start with an explicit improvement aim and a theory about what should change.
• Use a small family of measures: outcome measures, process measures and balancing measures.
• Track data over time; one before-and-after number hides variation and timing.
• Use measures close enough to the work that teams can receive signals quickly.
• Measure implementation as well as outcomes, or you cannot tell whether the idea failed or was never really delivered.
• Segment data when averages hide important groups or conditions.
• Never reward a number without checking the behaviour it creates.
• Use qualitative evidence alongside quantitative measures to understand why the signal moved.
• Do not collect data merely because the system can produce it.
• Every measure should have a decision attached to it.
• The goal is better learning, not a prettier dashboard.

Canonical Owner Boundary

This Learning Node owns measurement used inside an improvement cycle to learn whether a change is producing better system performance over time. How Education Works | Educational Measurement owns the broader problem of turning scores into defensible evidence. How Data-Informed Instruction Works owns classroom evidence used for the next teaching move. How Evidence-Informed Decision Making Works owns the wider synthesis of research, local data, judgement and values. How Control Charts Work owns the statistical logic of detecting meaningful system change. This page owns the practical measurement architecture surrounding improvement work.

1. Measurement Has Different Jobs

A national examination, a research trial, a teacher’s exit ticket and a school improvement dashboard all use data, but they are not doing the same job. Research may prioritise strong causal inference. Accountability may prioritise comparability. Improvement measurement prioritises timely learning inside a changing system.

Confusing these purposes creates bad design. A measure built for annual accountability may be too slow to guide a weekly improvement cycle.

2. Begin With the Aim

“Improve learning” is not an operational aim. Better aims specify the population, direction, magnitude and timeframe: reduce the proportion of Secondary 1 students who fail to retrieve key algebraic procedures after two weeks; increase the percentage of feedback tasks that lead to a successful revision; reduce unresolved reading errors by the end of each instructional cycle.

The aim determines what evidence can matter.

3. Make the Theory Visible

Before collecting data, state how the proposed change is supposed to improve the outcome. If teachers use short retrieval at the start of lessons, students should practise recalling previously learned material, gaps should become visible sooner, corrective action should happen faster and delayed retrieval should improve.

That chain suggests process measures, outcome measures and possible failure points. Without a theory, measurement becomes a fishing expedition.

4. Outcome Measures Tell You Whether the Destination Is Moving

Outcome measures capture the result the team ultimately cares about: reading fluency, attendance, successful problem solving, writing quality, retention, behaviour incidents, or another relevant educational outcome.

Outcomes matter, but they are often delayed and noisy. They tell you what happened more readily than why.

5. Process Measures Tell You Whether the Change Reached the Work

If the intervention depends on weekly feedback-and-revision cycles, measure whether those cycles actually occurred and with what quality. If the intervention depends on small-group reading sessions, measure reach, frequency and completion.

Without process evidence, disappointing outcomes are ambiguous. The theory may be wrong, or the intended process may never have happened.

6. Balancing Measures Ask What Else You Broke

A change that improves one metric can damage another part of the system. More practice questions may raise short-term accuracy while reducing time for extended reasoning. Additional intervention sessions may improve reading while causing students to miss science repeatedly. Faster marking may reduce teacher workload while making feedback less actionable.

Balancing measures keep improvement honest by tracking plausible side effects.

7. The IHI Family-of-Measures Logic

The Institute for Healthcare Improvement’s Model for Improvement is influential far beyond healthcare because it combines a clear aim, measures and small tests of change. Its guidance distinguishes outcome, process and balancing measures and encourages teams to plot data over time rather than rely on isolated snapshots.

Schools should borrow the logic rather than the clinical context: measure the result, measure the mechanism and watch for unintended consequences.

8. Plot Data Over Time

A pre-test and post-test can hide the journey. Weekly or fortnightly observations show whether improvement began before the change, followed the change, fluctuated with implementation quality, or disappeared after initial enthusiasm.

IHI’s run-chart guidance emphasises this simple principle: improvement happens over time, so time should be visible in the evidence.

9. One Point Is an Event; a Pattern Is Evidence

Schools often overreact to one unusually good or bad result. Variation is normal. The important question is whether the pattern has changed.

Control charts and run-chart rules help teams distinguish common fluctuation from signals that the system may actually have shifted.

10. Practical Measures Must Be Close to Practice

The Carnegie Foundation describes practical measures as measures that can be collected, analysed and used inside everyday work, providing timely signals about whether changes are leading to improvement. Its Practical Measurement for Improvement resources emphasise relevance, timeliness and usefulness to practitioners.

This matters in schools because a perfect metric that arrives after the decision is not practically useful.

11. Use the Smallest Measure That Can Answer the Question

Improvement teams routinely overbuild measurement systems. They collect every variable because it might become interesting later. The result is staff burden, incomplete data and dashboards nobody uses.

Start with the smallest measure capable of sending a useful signal. Add complexity only when a real decision requires it.

12. Measure Frequently Enough to Learn

If a team tests a change every week but receives outcome data once per year, the learning cycle is disconnected. Use nearer-term process and intermediate outcome measures while waiting for slower, higher-stakes outcomes.

Frequency should match the speed of the process you are trying to understand.

13. Do Not Confuse Ease of Collection With Importance

Digital systems generate abundant data: clicks, logins, completion rates, timestamps and submission counts. These are attractive because they are automatic. They may still be weak proxies for learning.

Measure what the theory requires, not merely what the platform exports.

14. Beware Proxy Drift

A useful proxy can gradually become the target. “Students should revise in response to feedback” becomes “every book must show green pen.” The visible trace replaces the learning mechanism.

Regularly ask whether the metric still represents the outcome or process you actually care about.

15. Goodhart’s Warning Belongs in Every Dashboard

When a measure becomes a target, people adapt to the target. That adaptation can be productive or perverse. Attendance targets can improve follow-up—or encourage coding practices that make the number look better. Homework completion targets can strengthen routines—or produce easier assignments that inflate completion.

The answer is not to stop measuring. It is to design measures with behaviour in mind and use multiple forms of evidence.

16. Segment Before the Average Hides the Problem

An intervention can raise the school average while leaving one group unchanged or worse. Segment by relevant year level, prior attainment, class, attendance pattern, language profile or implementation condition when doing so serves the improvement question and respects privacy.

Variation is often where the learning lives.

17. Look for Denominators

“Forty students improved” means little unless we know forty out of how many. “Ten behaviour incidents” means something different in a school of 300 than a school of 3,000, and something different across ten days than a year.

Rates, proportions and exposure counts prevent scale from misleading interpretation.

18. Annotate Changes on the Timeline

If a run chart changes, teams should know what else changed around the same time: examination season, teacher absence, timetable changes, new materials, a holiday, a coaching cycle or a cohort shift.

Annotation does not prove causation, but it makes interpretation more disciplined.

19. Combine Quantitative and Qualitative Evidence

A number tells you that something moved. A short interview, lesson observation, student work sample or teacher note may help explain why. IHI explicitly recommends using qualitative as well as quantitative information in improvement work.

Schools should treat these sources as complementary, not competing.

20. Measure Implementation Quality

Binary measures such as “used/not used” can be too weak. A questioning routine used mechanically may have different effects from the same routine used skilfully. A reading intervention delivered at half the intended dosage is not equivalent to full delivery.

Reach, dosage, quality and responsiveness are often useful implementation dimensions.

21. Measure Cost in Human Time

Schools frequently treat staff time as free because it does not appear as a new invoice. It is not free. If a change consumes four extra hours per teacher each week, that cost belongs in the evaluation.

A practice that raises outcomes slightly while exhausting the people delivering it may not be sustainable improvement.

22. Build Decision Thresholds Before Seeing the Result

Teams are vulnerable to motivated reasoning when they decide what counts as success after seeing the data. Define useful thresholds in advance where possible: what result suggests continue, adapt, investigate or stop?

Thresholds need not be rigid, but pre-commitment improves honesty.

23. A Dashboard Is Not a Learning System

Dashboards display information. Improvement requires interpretation, hypotheses, decisions, action and return. A school can possess beautiful analytics and remain incapable of learning from them.

Every dashboard meeting should end with changed action or a clearly stated reason for no change.

24. Cross-Domain Comparison: Manufacturing Quality

Manufacturing learned long ago that inspecting finished products is not enough. Stable quality depends on understanding the process that produces the output and detecting variation before defects accumulate.

Schools should not reduce learning to factory output, but the measurement lesson transfers: late outcomes cannot tell you enough about the process that created them.

25. Cross-Domain Comparison: Sports Performance

A sports team does not evaluate improvement only by the season-ending table. Coaches use training loads, technical indicators, video, recovery data and match performance because the final score is too sparse to guide daily work.

Education similarly needs near-term measures without mistaking those measures for the ultimate purpose.

26. Cross-Domain Comparison: Software Observability

Reliable software systems use multiple signals—latency, error rates, traffic, logs and traces—to understand whether a change improved or damaged the system. No single metric explains the whole state.

The transferable idea is triangulation around a theory, not data volume for its own sake.

27. Carnegie’s Improvement Principle

The Carnegie Foundation’s Six Improvement Principles includes a direct proposition: improvement at scale requires measurement, alongside problem specificity, attention to variation, system understanding, disciplined inquiry and networked learning. Its current practical-measurement resources make the educational application explicit.

The important word is improvement. Measurement is not the destination. It is the navigation system.

28. A Practical Measurement Stack

• Aim: what specifically are we trying to improve?
• Theory: what mechanism should create the change?
• Outcome measure: is the desired result moving?
• Process measure: is the key mechanism happening?
• Balancing measure: what important harm or trade-off might increase?
• Implementation measure: who received the practice, how often and with what quality?
• Time series: what does the pattern look like before and after changes?
• Segmentation: for whom and under what conditions is performance different?
• Qualitative evidence: what explains the pattern?
• Decision rule: what will we do if the signal moves?

29. Failure Mode: Measure Everything

The team creates forty indicators. Staff spend more time feeding the system than changing practice. Attention fragments and missing data grow.

Repair: keep only measures tied to a live theory or decision.

30. Failure Mode: Measure Only Outcomes

Results disappoint, but nobody knows whether the intervention was used. Leaders blame the idea or the staff without evidence.

Repair: measure the mechanism and implementation conditions.

31. Failure Mode: Measure Only Activity

Training attendance is 98 per cent. Lesson plans are uploaded. Meetings occurred. None of these proves that student learning improved.

Repair: connect activity measures to outcomes through an explicit causal theory.

32. Failure Mode: Punish the Signal

If bad data automatically trigger blame, people learn to improve the number rather than the system. The most valuable early warning disappears.

Repair: separate learning measures from high-stakes accountability where possible and make the purpose explicit.

33. Questions for an Improvement Team

• What exact decision will this measure inform?
• How quickly does the team receive the signal?
• Does the measure represent the outcome or merely a convenient proxy?
• What process should move before the outcome moves?
• What unintended consequence should we watch?
• Which groups are hidden by the average?
• What changed at the same time as the data?
• What qualitative evidence could explain the pattern?
• How much staff time does collection consume?
• Could rewarding this number create gaming?
• What result would make us adapt, continue or stop?

34. The Missing-Node Scan

If a school has plenty of data but repeatedly cannot tell whether a change is working, the missing node may be measurement architecture rather than measurement volume.

Look for symptoms: annual data used to steer weekly work, dozens of metrics without decision rules, outcomes without process evidence, activity without learner outcomes, averages without segmentation, dashboards without annotations, and targets that generate visible compliance but weak learning.

The cure is not more data. It is better questions connected to a smaller set of usable signals.

35. The Return Path

A school introduces a new revision routine. The first temptation is to wait for examination results. Instead, the team defines the aim, records how often students complete genuine revision, samples whether revisions improve the original work, tracks a delayed performance measure and watches teacher workload as a balancing measure.

After four weeks, completion is high but quality is flat. Student interviews reveal that many pupils are copying corrections without rebuilding the underlying answer. The team changes the routine. Two weeks later, revision quality begins to rise. Delayed performance follows later. Workload remains manageable.

The measurement system did not prove that the original plan was successful.

It did something more valuable.

It helped the school notice that the plan was incomplete while there was still time to improve it.

Improvement measurement works when numbers remain servants of learning: close enough to the work to guide action, broad enough to catch harm, and humble enough to change when the system teaches us something new.