Canonical question: How can an education programme appear to succeed when researchers or schools measure many outcomes and highlight whichever score improves most?
Audience: Parents, teachers, tutors, school leaders, curriculum teams, learners and readers of research reports.
Short answer: An education initiative may influence reading speed, comprehension, writing accuracy, confidence, attendance, quiz performance, delayed recall and many other measures. That breadth can be valuable. But when organisers measure everything, look at the results and then announce only the most flattering outcome, they can create a persuasive success story even if the overall evidence is weak. A sound evaluation distinguishes the main learning outcome from supporting or exploratory outcomes before seeing the results; reports favourable, unfavourable and uncertain findings; and interprets the pattern against measurement quality, comparison conditions and practical importance. One bright number is a clue, not a verdict.
The gateway: three test results and a very confident brochure
A Secondary 1 English programme promises to improve independent writing. Six weeks later, a report shows a striking result: pupils’ average scores on a vocabulary matching quiz rise more than expected. The brochure says the programme has transformed English performance. Yet sentence construction improved only a little, the unfamiliar writing task showed no clear difference, and an oral explanation measure was not reported. A parent sees the bold vocabulary result and assumes the child will now write stronger essays.
The difficulty is not that vocabulary is unimportant. Vocabulary knowledge can support comprehension and expression. The difficulty is that a result in one measured skill has been promoted into a larger claim. If the programme’s stated goal was independent writing, the most relevant evidence is whether learners can produce better writing on suitable tasks. A vocabulary gain may support a theory about how improvement might occur. It does not settle the full outcome.
Now make the example more complicated. Suppose the team measured twelve outcomes and chose to publicise the one with the largest apparent gain. A surprisingly high result somewhere becomes more likely when many outcomes are inspected. Researchers may call this selective outcome reporting, outcome switching or, in a broader family of problems, multiplicity. The exact term depends on what was planned and how the results were analysed. In everyday education, the underlying problem is understandable: the brightest number receives all the attention, even though the original question had a different centre.
This article has a narrow purpose. It is not a generic explanation of statistical significance, a guide to test design or a condemnation of school dashboards. It explains how selecting a favourable outcome after measuring many outcomes can change the meaning of a programme claim, and how to keep curiosity without confusing a discovery with confirmation.
The 50-second explanation
Imagine a school programme that is assessed using reading comprehension, reading speed, vocabulary recognition, spelling, writing length, writing quality, attendance and learner confidence. If one result looks unusually good, it is tempting to announce the programme works. But those results answer different questions, and looking across many measures creates more opportunities to find an apparently impressive result by chance or by test alignment. The key safeguard is to decide in advance which outcome most directly represents the programme’s main promise. Report all the other results too, including disappointing ones, but distinguish planned confirmation from additional exploration. Check whether each measure is valid, independent enough, fair and educationally meaningful. A programme can have a real benefit in one area without proving broader learning gains. The responsible question is not “Did any number improve?” but “Did the programme improve the learning outcome we said mattered, compared with a credible alternative?”
Stage 1: begin with the sentence the programme is supposed to make true
Schools frequently choose initiatives before making their outcome claim precise. “Improve literacy” sounds admirable but can mean decoding, vocabulary, comprehension, fluency, grammar, essay composition or willingness to read. A method that helps one of these skills may not affect the others immediately. The first act of a sound evaluation is to write a sentence with an educational verb and a particular performance: “After eight weeks, pupils can identify the main claim and provide two relevant pieces of evidence in an unfamiliar passage without prompts.”
That sentence tells us what to teach and what to measure. It also protects learners from programmes that change their headline promise whenever the first test proves disappointing. A comprehension programme cannot automatically redefine success as improved app engagement simply because pupils logged in frequently. Engagement may be helpful, but it is not a substitute for the promised reading skill.
For a mathematics programme, the main outcome might be accurate solving and explaining of unfamiliar percentage problems. For a science programme, it might be designing a fair experiment and justifying the controls. For an English vocabulary routine, it might be accurate use of target words in new written contexts after a delay. Choose one outcome or a small, clearly ordered set that matches the educational purpose. Do not choose it because it produced the highest score after the fact.
Stage 2: see why several useful measures create several interpretation problems
A school may be right to measure multiple things. Learning is not a single number. Fluency can improve before comprehension; confidence can change before independent performance; attendance can matter even when test scores are stable. A rich evaluation therefore gathers evidence about mechanism, delivery, experience and academic outcomes. Multiplicity becomes risky when those results are treated as interchangeable tickets to the same success claim.
Suppose a writing programme produces little improvement in essay quality but helps pupils identify supporting evidence more accurately. That may be a meaningful intermediate gain. It invites the question of whether later instruction can convert evidence selection into better writing. The measured improvement should be celebrated within its scope. It should not be advertised as proof of better essays if essay evidence is absent.
The same principle applies in reverse. A programme might improve independent mathematics performance but not confidence immediately. We should not call that academic improvement invalid merely because a separate survey failed to move. The aim is to read the pattern. Some outcomes are primary targets, some explain the pathway, some identify harms or costs, and some generate new hypotheses. A well-ordered set of measures produces understanding; a bag of interchangeable metrics produces publicity.
Stage 3: learn what happens when the best of many numbers is selected
Imagine ten unrelated quiz measures, each with ordinary measurement fluctuation. Even if a programme had no real effect, several outcomes might move up while others move down. Which movement looks most exciting depends partly on the chance variation of those particular measures. Select the largest improvement and the chosen number may overstate what would happen in a fresh cohort.
This is one reason multiple comparisons matter in formal research. The more opportunities investigators have to declare success, the more carefully they should control the risk of false-positive findings when making confirmatory claims. Statistical methods for handling multiplicity exist, but they cannot replace a meaningful research question. Nor does a school need to learn every correction formula before asking whether outcomes were selected in advance.
There is also a less technical issue: many measures do not have the same relevance. A programme for persuasive writing could generate a significant change in reading speed, but that does not automatically establish an effect on argumentative reasoning. A parent should not be asked to infer a different construct from the one actually assessed. Multiplicity is both a statistical problem and a problem of meaning.
Stage 4: distinguish planned outcomes from post-hoc discoveries
An outcome selected before the study begins is sometimes called prespecified. Prespecification does not make its result true, but it reduces the freedom to choose whichever result becomes most convenient. A secondary outcome can also be prespecified. An exploratory outcome is measured to help generate new questions; it can be valuable, but a dramatic exploratory result should ordinarily invite replication rather than be presented as definitive confirmation.
Consider a science intervention. Its planned primary measure is the ability to explain controlled variables on a novel experiment. Secondary measures include correct use of technical vocabulary and completion of practical worksheets. After the lessons, pupils improve in vocabulary but not on the novel experiment. The honest conclusion is that the intervention showed a vocabulary benefit in the assessed setting while providing less evidence for the main reasoning objective. The teaching team may refine the intervention and investigate how to support transfer.
What would be misleading is silently changing the report to say the programme was designed mainly to improve vocabulary. Schools can change educational goals through thoughtful review. They should record when that change occurred and not describe a newly chosen target as though it had always been the test of success.
Stage 5: identify the unit of meaning before choosing a statistic
In English writing, one task might receive separate scores for ideas, organisation, vocabulary, grammar and language accuracy. Those are legitimate dimensions, but they can be correlated. If the programme improves the overall quality of arguments, several subscales might move together. Counting every moved subscale as an independent demonstration of success could exaggerate the amount of evidence. Conversely, collapsing every subscale into an unexplained total could hide a clear weakness.
A carefully designed outcome hierarchy can help. Primary: independent writing quality on a novel prompt, scored with a rubric that covers the agreed learning objective. Supporting: evidence selection and paragraph coherence. Process: lesson attendance and completion of practice. Safety or burden: pupil stress and teacher time. Each is relevant, but each has a different role in the decision.
The What Works Clearinghouse uses outcome domains and study protocols to guide how education findings are reviewed. A school does not need to reproduce its technical methods, but it can borrow the underlying discipline: know which construct each measure represents and do not let the analysis redefine the construct according to the most attractive available number.
Stage 6: a worked Primary 6 vocabulary example
A small group of Primary 6 pupils studies one hundred advanced words. The teacher uses examples, contrasts, retrieval and short writing tasks. At the end of the month, recognition accuracy rises sharply. Parents are delighted, and the teacher has reason to be pleased. Pupils can identify more meanings than before.
But imagine the school’s broader claim is that the programme helps students choose precise vocabulary in PSLE composition. To evaluate that claim, the teacher must examine unfamiliar writing tasks, appropriateness of word choice, grammar and clarity. A pupil who selects the correct definition of reluctant may still use it awkwardly in a sentence. Another may write a perfectly accurate paragraph without using the advanced word at all, because the simpler word was more appropriate.
The teacher can report both findings: recognition improved on a defined list; transfer into independent writing still needs attention. That is educationally actionable. The next lessons may practise choosing the right word for a purpose, not merely recalling meanings. A programme improves when evidence identifies the distance between skills. It becomes less trustworthy when the strongest proximal score is treated as proof of every distant outcome.
Stage 7: a worked Secondary 1 mathematics example
Imagine a tutor in a three-pupil mathematics group uses four different types of practice: worked examples, independent questions, verbal explanations and error analysis. The pupils take a short test immediately afterward and an unfamiliar test two weeks later. They complete a confidence check and a time-to-completion task. The programme seems to help speed and confidence, while the delayed unfamiliar test shows a modest change.
The tutor should not throw away the speed and confidence evidence. Faster procedures can release attention for harder problems, and increased willingness to attempt questions can matter. But if the advertised goal is independent mathematical reasoning, the delayed unfamiliar questions deserve special weight. A learner who finishes familiar problems quickly yet cannot explain why the method works has not reached the same goal as a learner who reasons accurately on novel tasks.
A useful report states what each measure showed and what the tutor will change. Perhaps the class needs more time on representations and fewer near-identical exercises. Perhaps one pupil is still unstable with fractions. The right learning decision emerges from the pattern, not the biggest available percentage increase.
Stage 8: a worked Secondary Science example
A school introduces a practical-investigation notebook. Pupils record their hypotheses, variables, observations and conclusions. After several weeks, notebook completion rises substantially, science vocabulary scores improve a little, and performance on unfamiliar experimental-design questions is unchanged. Leaders could easily announce success by highlighting notebook completion. Yet the programme was intended to improve scientific reasoning, not merely to make the notebook look tidy.
What does this result mean? The notebook system may have improved routines without yet changing reasoning. Teachers might need to model why changing two variables makes a conclusion weaker. Pupils may be copying headings without understanding the relationship among variables. The measure of experimental design could also be too difficult, too short or insufficiently sensitive. A thoughtful evaluation investigates those possibilities.
The distinction between process and outcome is crucial. A complete notebook may create the conditions for learning, but completion is not necessarily evidence of conceptual understanding. The same applies to learning-platform logins, worksheet counts and time spent in class. Useful measures become misleading when their role is silently upgraded.
Stage 9: why some results are easier to move than others
Outcomes differ in proximity to the taught material. A lesson on specific vocabulary words is likely to affect recall of those words before it changes performance on a broad writing assessment. A study-skills intervention may change learners’ planning habits before producing detectable examination gains. A classroom-discussion routine may improve the quality of oral explanations before affecting standardised scores. The closer measure is not inherently inferior; it simply answers a narrower question.
The practical danger is selecting whichever distance makes the intervention look best and then using the result to imply broader educational value. If the measure is an immediate check of practised content, say so. If the measure tests independent transfer, explain that too. When a programme claims long-term learning, include a later assessment appropriate to that claim rather than only the test administered directly after teaching.
The Institute of Education Sciences highlights outcomes that are valid and appropriately aligned, while warning against assessments that are overly aligned to an intervention. That principle is particularly useful when outcome selection and test familiarity interact: the easiest measure to improve may not be the fairest measure of what the programme promised.
Stage 10: do not turn measurement error into educational meaning
A quiz of five items can produce large percentage swings when one answer changes. A survey response can shift because the pupil’s mood changed. A writing rubric may show variation between markers. An examination cohort can include different starting abilities from the previous one. None of these issues makes the measures useless. They do mean that each score carries uncertainty.
Suppose confidence rises from three to four on a five-point scale. Was the change meaningful? We need to know how the question was phrased, whether the same pupils answered, what the scale represents and whether their independent work changed. Suppose a score rises eight marks on one paper. Were the papers comparable? Did the test cover the same content and difficulty? What was the baseline? Simple measurement questions often matter more than an impressive second decimal place.
When many imprecise outcomes are measured, the chance of finding an impressive-looking movement increases. Use reliable measures where possible and acknowledge weaknesses when precision is limited. Do not ask a single small test to carry a large claim about a child’s future achievement.
Stage 11: read confidence, motivation and enjoyment as different outcomes
Children can enjoy a lesson without mastering its content. They can dislike an uncomfortable but productive exercise and later benefit from it. They can feel confident while holding misconceptions. They can also learn successfully yet remain uncertain about their performance. Motivation, enjoyment, confidence and achievement are connected in complex ways; they are not synonyms.
If a programme mainly promises to improve school belonging, a belonging measure may be central. If it promises to improve algebra, belonging can be a useful supporting outcome but should not replace algebra evidence. If it promises both, specify both and identify which result is primary for the decision. That clarity avoids shaming educators for attending to wellbeing while preventing wellbeing scores from being misrepresented as academic effects.
Ask whether any outcome could indicate a cost. More homework completion may come with excessive evening workload. Higher test scores might reflect narrow drilling that displaces other subjects. Improved classroom order may conceal less pupil initiative. A balanced report considers benefits and possible trade-offs, not only the greenest metric.
Stage 12: why a statistically noteworthy result may not be the important one
In formal evaluation, statistical significance and practical importance are different ideas. A very large study can identify a small difference with considerable precision, while a small school study can produce an apparently large difference with wide uncertainty. Neither the word “significant” nor the absence of it tells us by itself whether the programme is worth the cost.
Consider a reading intervention that improves a short recognition measure by a tiny amount but requires several hours of extra staff time each week. Even if the result is precise, school leaders must ask whether it is educationally meaningful and whether the same resources would help learners more elsewhere. Conversely, a promising improvement in a difficult-to-measure skill may warrant further exploration even if an early small pilot is too imprecise for a strong claim.
Avoid reducing the decision to whether one p-value passes a familiar threshold. Examine the size, uncertainty, relevance, cost and durability of the outcome. Also examine whether it was selected beforehand or emerged from many tests. These dimensions answer different questions and should be communicated separately.
Stage 13: the special danger of changing the outcome after seeing a disappointing result
Sometimes a programme team begins with a clear intention but faces pressure after the data arrive. The main outcome shows no difference. A secondary measure looks positive. A report draft begins to describe the secondary measure as the programme’s central success. This can happen through optimism rather than deliberate misconduct: people remember what excites them and rationalise the earlier plan.
The repair is procedural. Keep a dated outcome plan. In a formal study, register relevant questions, measures and analysis intentions before outcomes are known. In an ordinary school pilot, retain the staff-meeting note or evaluation document that identified the main outcome. Any subsequent change should be recorded and explained. A revised goal can be legitimate, but it should be treated as a new question, not an earlier confirmed one.
Transparency protects innovators as well as readers. A team can say, “Our reading-comprehension target was not clearly met. We found an unexpected improvement in vocabulary recognition and will test that finding in a subsequent programme.” That is a useful discovery. The data are not wasted merely because the original claim was not demonstrated.
Stage 14: report a matrix, not a highlight reel
An understandable outcomes table can contain four columns: planned educational question, measure used, observed pattern and interpretation limit. For a writing programme, the first row might read: independent writing quality; unfamiliar essay; little clear difference; small sample and marking uncertainty. The second might read: evidence selection; brief independent task; moderate improvement; may be a useful intermediate gain. The third might read: vocabulary recognition; matching quiz; large improvement; closely aligned to taught items.
This style is not glamorous, but it makes the decision visible. A parent sees what improved and what did not. A teacher can decide which step needs strengthening. A school leader can assess whether a small additional trial is warranted. The report also prevents a good outcome from swallowing an equally important disappointing one.
To make the table honest, report the number of learners assessed on each measure, not only percentages. Some pupils may have missed a writing assessment but completed a short quiz. If outcome-specific participation differs, the apparent pattern may partly reflect who supplied data. The separate problem of missing learners matters here, though it remains a distinct mechanism with its own canonical owner.
Stage 15: choose a primary outcome that deserves to be primary
Prespecifying a primary outcome is only useful if it actually represents the educational aim. A poorly chosen primary measure can make a programme appear ineffective even when learners gain something important, or effective when the assessed skill is trivial. Outcome planning therefore needs subject expertise, not merely methodological discipline.
Ask three questions. First, does the measure capture a skill the programme promises to change? Second, can pupils perform the task independently without copying the lesson’s exact examples? Third, is the measure appropriate for the students’ starting level and the timeframe? For a Secondary 4 Additional Mathematics intervention, an overly broad final-year paper may be insensitive to a short unit on trigonometric identities. A focused but novel set of reasoning problems could be more suitable for the immediate question, with broader examination performance examined later.
Also consider fairness. A writing measure should not reward vocabulary complexity at the expense of accurate communication when accuracy is the actual learning target. A science measure should not penalise a pupil’s English unnecessarily when scientific reasoning is central. Good assessment and good outcome governance depend on each other.
Stage 16: the difference between outcome switching and sensible teaching adaptation
Teachers must adapt teaching when evidence reveals a misconception. If pupils show that they cannot identify a paragraph’s topic sentence, a teacher may change tomorrow’s lesson. That is responsive instruction, not the same as manipulating a completed evaluation. The distinction lies in what claim is being made and when it is made.
A teacher can legitimately set a new goal midway through a course, document the reason and begin measuring progress toward that goal. But an evaluation claiming that a specific original intervention caused a particular outcome should be clear about whether the intervention or outcome definition changed. Otherwise readers cannot tell what was tested. Versioning the programme and its outcomes is a simple safeguard.
For parents, the practical question is whether a provider changes objectives transparently. If a child cannot yet manage basic fraction operations, a tutor may reasonably prioritise foundations over advanced examination practice. The tutor should explain the change and show evidence of progress toward the revised goal. Flexibility in teaching is valuable; retroactively rewriting success criteria is not.
Stage 17: examine the pattern across outcomes as a possible mechanism
Outcomes can do more than certify success or failure. Their pattern can explain what a programme changes first. Suppose a learner starts attempting unfamiliar maths questions but still makes procedural errors. That suggests greater willingness to engage, without proving mathematical mastery. Suppose writing evidence selection improves but coherence does not. That suggests the pupil can choose relevant material but may need help connecting it to a claim.
A cautious mechanism explanation asks which changes would need to occur before the distal goal moves. In science, understanding variable control may be a prerequisite for designing fair experiments. In vocabulary, accurate recognition may precede fluent and appropriate use. In mathematics, automaticity with fractions may reduce the working-memory load of later algebra. These are plausible learning pathways, not universal sequences guaranteed for every pupil.
To test a proposed pathway, collect evidence at suitable stages and look for transfer to tasks beyond the taught examples. Do not mistake a post-hoc story for causal proof. A coherent explanation can guide the next lesson even when it remains a hypothesis about why improvement occurred.
Stage 18: the learner route — use a few measures that answer different questions
Students often track progress by the last mock examination mark. That is understandable, but it is not enough to diagnose learning. A helpful personal dashboard might include one target-skill measure, one transfer measure and one process indicator. For example: can I solve equations involving fractions without help; can I apply the same principle in a new word problem; did I complete the planned short practice sessions?
The process indicator helps with routines, but it should not replace the learning measures. Completing ten worksheets can still leave a misconception. A transfer item may show genuine progress even if a familiar timed quiz is disappointing. Look for a pattern across sessions rather than searching for the single number that makes you feel best.
Do not turn the dashboard into a source of anxiety. One or two short independent checks are enough for many purposes. The objective is to select better next steps, not to prove personal worth. Learners have different starting points and different rates of change. Accurate feedback is useful because it identifies what to practise, not because it produces a flattering metric.
Stage 19: the parent route — decode programme evidence in ordinary language
When a tuition provider or educational product displays a dramatic gain, ask which skill the result actually measures. Was that skill the programme’s main promise before the trial? What happened on other relevant measures? Was the task the same as the practised material? How many learners supplied the result, and over what period? Did the provider compare them with similar learners not receiving the specific support?
Parents need not demand a randomised trial before choosing every lesson. Decisions also depend on the child’s needs, the tutor’s responsiveness, affordability and fit. The aim is to avoid purchasing a broad promise supported only by a narrow outcome. If your child struggles to write coherent explanations, a vocabulary-recognition score is encouraging but does not directly answer whether the writing difficulty has been solved.
Ask the provider how it will assess independent performance after several weeks and how it will revise the teaching plan if the main goal is not progressing. A specific answer signals that outcomes are used to improve learning rather than merely to decorate marketing.
Stage 20: the teacher route — choose an outcome ladder before teaching
Write three levels of evidence. Near is the skill practised in the lesson, such as identifying evidence in a model paragraph. Transfer is the skill applied in an unfamiliar context, such as choosing evidence in a new text. Durable is the skill demonstrated after a meaningful delay with less prompting. Not every unit requires formal measurement at all three levels, but the ladder makes clear which claims belong to each level.
Identify one learning target as primary for the instructional cycle. Select a fair task, a simple rubric and a consistent way of collecting work. Keep process measures such as attendance separate. If an unexpected benefit appears, welcome it as an additional observation but label it appropriately. Explain to colleagues which outcome was planned and which emerged during teaching.
At the end, decide what to change. If near performance improves but transfer does not, add varied examples and independent application. If transfer is strong but delayed performance weakens, plan spaced checks. If comprehension improves but pupils feel overwhelmed, revise pacing and support. The pattern supports a teaching decision because it is read in relation to the purpose, not because one number has been selected for praise.
Stage 21: the school leader route — protect evidence from marketing pressure
School leaders are often asked to turn pilot results into budget choices. A clear outcome hierarchy helps avoid two extremes: accepting the brightest number uncritically or dismissing every complex result as inconclusive. Before the pilot begins, require a short outcomes plan identifying the main learning claim, supporting outcomes, how each will be assessed and what result would influence the decision.
Afterward, review the complete outcome set with attention to learner needs and opportunity cost. Do not insist that every measure improve; education initiatives often affect specific skills. But do insist that the report explain why the primary measure matters and that it disclose relevant disappointing results. A programme that genuinely improves a narrow skill may be worth adopting for that purpose. It should not be sold as a comprehensive academic solution.
When commissioning formal evaluation, seek advice on analysis plans, multiplicity and reporting standards. The EEF provides protocol, study-plan, analysis-plan and reporting resources that can support this discipline. Smaller schools can borrow the transparency principle without reproducing an expensive research infrastructure.
Stage 22: what the evidence publishers actually establish
The Education Endowment Foundation publishes resources for evaluation design, prespecified protocols, statistical analysis plans and reporting. Its public evaluation infrastructure reflects the importance of stating questions and analysis intentions before results are interpreted. The What Works Clearinghouse provides procedures and study review protocols that define outcome domains and guide research synthesis. These are foundations for accountable evidence use, not proof that every study with multiple outcomes is flawed.
The Institute of Education Sciences’ Standards for Excellence in Education Research emphasise sound measures that are valid, reliable, appropriate to the intervention and not overly aligned to its teaching tasks. They also encourage examining outcomes beyond the immediate intervention period. The OECD’s April 2025 teaching report underlines the complexity of classroom practice and the different domains in which effective teaching may operate. Together these sources support careful distinctions among outcomes, educational goals and implementation.
None of these publishers endorses a universal rule that a school must assess exactly one skill. Multi-outcome evaluation can be excellent when the hierarchy, purposes and limitations are clear. The target is selective reporting and overclaiming, not curiosity.
Stage 23: failure modes and practical repairs
Failure: the objective is broad and the headline is narrow. Repair: write a specific outcome claim before evaluation and preserve the distinction between the claim and supporting measures.
Failure: twelve outcomes are measured but only the biggest is published. Repair: report the full set in a simple table and identify which findings were planned or exploratory.
Failure: a practised quiz stands in for independent learning. Repair: include unfamiliar application appropriate to the goal, and avoid overstating what near-task performance demonstrates.
Failure: confidence is treated as mastery. Repair: regard confidence as a separate outcome and check independent work.
Failure: a null primary result is replaced by a positive secondary result without explanation. Repair: retain the original plan, describe the secondary result as additional evidence, and test it anew before making a strong claim.
Failure: one percentage from five questions becomes the programme’s success rate. Repair: examine reliability, number of items, baseline difficulty and uncertainty.
Failure: the report contains statistical language but not an educational decision. Repair: state what the result means for the actual learner, the next teaching step, time cost and alternative choices.
Failure: improvement on some measures is ignored because every indicator did not move. Repair: judge the specific targeted outcome fairly rather than demanding a magical programme that changes everything.
Stage 24: a one-page decision framework for a school team
At the top, write: “We were trying to improve…” and complete the sentence in terms of learner performance. In the next space, record the primary measure, why it fits the goal, and whether it was selected before outcomes were known. In a third space, describe the comparison or benchmark and its limitations. Record the number of eligible and assessed learners.
Create a small section for secondary results: near practice, transfer, delayed performance, wellbeing and workload, including results that did not improve. Mark exploratory observations separately. In the final section, answer three questions: what can we reasonably claim, what remains uncertain, and what action follows? The action might be to retain a targeted routine, redesign the next lessons, run a stronger evaluation or stop an expensive programme with little useful evidence.
Consider a fictional Primary 5 class. A reading routine improves vocabulary recognition but not comprehension of a new passage after four weeks. The decision is not simply success or failure. Teachers may keep the vocabulary component, change comprehension instruction and repeat the unfamiliar-passage check. The framework turns outcomes into a practical diagnosis instead of a binary brand verdict.
The same one-page format works for families reviewing tuition and for tutors planning next-month instruction. Transparency does not require complexity; it requires respecting what each measure can actually tell us.
Stage 25: the ethics of communicating school results
Education decisions influence children’s time, confidence and resources. A selective outcome claim can direct a family toward a programme that does not address its child’s difficulty, or divert school funding from stronger alternatives. The ethical responsibility is therefore more than avoiding fabricated numbers. Accurate communication must also preserve the relationship between the numbers reported and the outcomes promised.
This responsibility cuts both ways. A cautious report should not erase useful improvement merely because it is narrow. If a routine helps children recognise academic vocabulary, say so and explain why that skill may matter. If independent writing has not yet improved, say that too. Readers can handle nuance when it is presented plainly.
Teachers also deserve protection from the demand to produce constant success stories. A programme that reveals a persistent misconception has generated valuable information. A pilot showing that an intervention is too burdensome can save instructional time. Findings need not be spectacular to improve education. The honest test is whether the evidence helps make the next decision more intelligent and humane.
Frequently asked questions
What is selective outcome reporting?
It occurs when a report’s choice of which measured outcomes to present or emphasise depends on how favourable the results appear. Not every decision to focus on a particular measure is improper; the concern arises when readers are misled about the original question or the complete pattern of findings.
Is it wrong to measure many educational outcomes?
No. Multiple outcomes can reveal mechanisms, side effects and important differences among skills. The problem is treating every outcome as an interchangeable chance to declare success or concealing unfavourable results. Clear hierarchy and complete reporting are the safeguards.
What is a primary outcome?
A measure identified as central to the evaluation’s main research question. It should represent the educational target and ideally be specified before results are known. Its value comes from relevance and planned interpretation, not from automatically being the largest numerical improvement.
What is the difference between a secondary outcome and an exploratory finding?
A secondary outcome is additional evidence relevant to the study, often planned beforehand. An exploratory finding emerges as an area for further investigation, frequently from a broader search. Either can be valuable, but a striking unexpected finding usually warrants independent confirmation before a strong general claim.
If vocabulary improves but writing does not, did the programme fail?
It depends on what the programme promised. If the goal was vocabulary recognition, it may have achieved that narrow goal. If the goal was improved independent writing, vocabulary improvement alone does not show that the broader target was reached. It may identify a useful intermediate step.
Does statistical significance settle the matter?
No. Statistical significance is one part of interpreting certain studies. We also need effect size, uncertainty, outcome relevance, study design, implementation and educational costs. A statistically noteworthy result on a poorly chosen outcome can still be a poor basis for a school decision.
Should a parent request every test result from a tutor?
Not necessarily every routine classroom exercise. Ask for an understandable summary of progress against the child’s agreed learning goals, evidence from independent tasks, and a clear explanation of what remains difficult. A useful answer is often a small set of meaningful work samples rather than a giant spreadsheet.
Why not choose whichever result matters most after the programme finishes?
You may discover a genuinely important new question, but selecting it after seeing the results changes the evidential status of the claim. Report it as a discovery or revised goal and test it prospectively. Do not silently present it as the objective the programme had all along.
Can a school change a goal when it discovers prerequisite gaps?
Yes. Responsive teaching should change when student needs become clearer. Document the new goal and measure it appropriately. The issue arises when a completed evaluation’s original success criterion is rewritten after its result is known.
How should parents interpret a score that doubled?
Check the number of items, baseline score, test comparability and skill measured. A rise from one to two correct answers out of five is mathematically a doubling but may not demonstrate broad mastery. Ask whether the learner can apply the skill independently to new questions.
Are engagement and wellbeing less important than test scores?
No. Their importance depends on the programme’s purpose. They are different outcomes and deserve appropriate measures. The mistake is claiming an academic effect from a wellbeing measure, or a wellbeing effect from an academic score, without supporting evidence.
What should a principal demand from an evaluation report?
A clearly stated learning objective, outcome hierarchy, measures, complete findings, relevant comparison, assessed sample, uncertainty, time and cost implications, and an honest statement of what the report cannot establish. A one-page decision summary can make that information accessible.
Sources, research basis and internal links
- Education Endowment Foundation — Evaluation guidance and resources. Evaluation design, protocol, analysis-plan and reporting resources; consulted October 2026.
- What Works Clearinghouse — Handbooks and study review protocols. The importance of planned outcome domains, study review standards and disciplined interpretation; consulted October 2026.
- Institute of Education Sciences — Standards for Excellence in Education Research. High-quality, appropriately aligned measures and documentation of intervention and comparison conditions; consulted October 2026.
- OECD — Unlocking High-Quality Teaching, 3 April 2025. On the different goals and practices involved in effective teaching.
- US Department of Education — Selecting Evidence-Based Practices for Tiers 1, 2 and 3. Guidance on the need to assess evidence quality and relevance, not simply appealing headlines.
- eduKateSG — How X Works hub. The existing education mechanism library.
- eduKateSG — How Formative Assessment Works. The separate owner for adapting instruction to ongoing evidence.
- eduKateSG — Secondary 1 Mathematics Tutor Clementi. A local learning context for illustrations, not a source of programme efficacy claims.
Reading safeguard: All scores, classes and outcomes in worked examples are hypothetical. They demonstrate reasoning, not research findings about a named student, school or provider. The lesson is to state which outcome matters first, then make the claim no stronger than its measure.