Canonical question: Why can the most promising education pilot deliver a smaller improvement when another school tries it?
Audience: School leaders, teachers, tutors, education researchers, parents, learners and anybody being shown a remarkable first set of results.
Short answer: A pilot can be useful and still overstate the improvement a school should expect next time. When many ideas, classes, teachers, sites or measures are tried, the apparently best result is often a mixture of genuine benefit and unusually favourable circumstances. Choosing a winner using that unusually favourable result means the next implementation may look disappointing even if nothing has gone wrong. This is a selection problem, not proof of dishonesty or evidence that innovation is futile. A sound response is to distinguish feasibility from effectiveness, record every candidate considered, test the selected approach again, and decide whether it still helps ordinary learners under ordinary conditions.
The reader’s gateway: the class that changed everything
Imagine a secondary school choosing between several approaches to improve written explanations in science. One department tests four lesson routines. The fourth class appears to improve dramatically. Pupils write longer, include more relevant terms and seem less hesitant. At the next staff meeting, a slide announces the winning routine. By the next term, every class is asked to adopt it. Then the average improvement is smaller than the pilot promised. Some pupils are still struggling; some teachers say the routine takes too long. Was the pilot false? Were teachers unwilling? Did the pupils become less motivated?
Those explanations are possible, but there is an earlier question worth asking: how did this particular class become the example everybody believed? If the school inspected four routines, twelve classes and three scoring methods, it had many opportunities to discover something impressive by chance. The strongest initial observation was not selected at random. It was selected precisely because it looked strongest. That choice changes how we should interpret the number.
The mechanism matters well beyond formal research. A parent may select a tutor because three former pupils made spectacular gains. A learner may choose a revision app because its best testimonials look extraordinary. A ministry may see a promising small programme, while a school considers expansion across different teachers and timetables. Selection is everywhere. It is often rational to explore many options. What becomes unsafe is treating the highest observed result as the expected result for everyone else.
This article is an explanation of selection, replication and scaling. It is not a generic guide to randomisation, a ranking of tuition brands or an argument that all pilots fail. The useful question is how to move from encouraging early signals to decisions that remain fair when the first excitement has gone.
The 50-second explanation
Suppose ten schools try different ways to improve reading. Every result contains a true programme effect, measurement noise, variation among pupils and teachers, and perhaps a lucky timetable or unusually committed team. The school with the highest observed gain is likely to have both a genuinely useful approach and some favourable noise. If leaders select that school as the model and expect its exact gain everywhere, they have counted the lucky portion as though it belonged permanently to the method. Researchers call this family of problems selection bias or, in some settings, the winner’s curse. Results often move closer to an underlying average when repeated; that tendency is commonly called regression to the mean. The cure is not to ignore the promising pilot. It is to document what was tried, identify why the routine should work, repeat the test with independent measures and different learners, and only expand as confidence and delivery capacity grow.
A small diagram in words: what gets selected?
An observed improvement may be thought of as real effect + background change + measurement error + favourable or unfavourable context. That is a conceptual decomposition, not a formula that allows a teacher to calculate each component from one spreadsheet. In a well-designed comparison, some background explanations can be reduced. In an ordinary class pilot, many remain mixed together. If an organiser then chooses whichever observation is largest, the selected observation tends to contain disproportionately favourable contributions from those uncertain parts.
Imagine six small-group lessons designed to improve percentage problems. Each is reasonable. Each is taught competently. Learners complete pre- and post-questions, though the questions are not equally difficult and the children are not matched perfectly. One routine produces an observed rise of sixteen marks; others produce six, eight, seven, eleven and nine. The sixteen is intriguing. But before printing a brochure that promises a sixteen-mark gain, we need to ask whether that group started at an unusually low baseline, whether it received extra revision elsewhere, whether one exam paper happened to match the lesson, and whether we would have noticed a different class had the scores been arranged differently.
The winning number might represent real superiority. It might also include an unusually large positive error. It is impossible to settle that by admiring the number more closely. We need new evidence.
Stage 1: define the decision before celebrating the score
A pilot has different possible jobs. It can ask whether a timetable is feasible, whether teachers can deliver a routine, whether pupils engage with it, whether a measure captures its intended learning outcome, or whether the routine improves achievement relative to what would otherwise have happened. Those are related but different questions. A three-week trial with a handful of enthusiastic pupils can answer feasibility questions surprisingly well. It is rarely strong enough to forecast the exact improvement an entire school will obtain.
Before running a pilot, write a sentence beginning, “We will use the evidence to decide whether…” The ending determines the appropriate measurement. “Whether parents will accept a Saturday session” calls for attendance, family feedback and accessibility. “Whether weaker readers can decode a new passage more accurately after six weeks” calls for defined reading measures and a credible comparison. “Whether the approach should become the default for every Secondary 1 class” requires evidence across classes and teachers, not merely a celebration of the first cohort.
The Education Endowment Foundation’s programme pipeline separates stages of early development, pilot work, efficacy, effectiveness and scale-up. That does not mean every school must commission a major trial. It means a responsible claim should not outrun the stage of evidence available. A workable classroom idea is a success at the development stage even when we cannot yet announce its average impact.
Stage 2: count the invisible attempts
A published success story usually features what happened. It rarely begins with the many ideas that did not produce a dramatic slide. A school may have tried different group sizes, practice sequences, quiz frequencies, teacher explanations or parent reminders. If twenty such choices were explored and the strongest was reported as though it was the only planned test, readers see a distorted view of the search process.
This is not the same as accusing teachers of manipulating pupils. Iteration is valuable. A science department ought to notice an effective analogy and adjust instruction. The problem occurs when an exploratory search is retold as a single decisive proof. Record what was tried, including routines that seemed weak or inconclusive. Keep a decision log: date, learner group, measure, version, time cost, change from the previous version and reason for continuation. Even a plain document can prevent an accidental rewrite of history.
For families, ask an uncomplicated question when shown an impressive claim: “Is this result from everyone who used the programme, or from the best-performing group?” The answer may be reassuring, but it may also reveal that the advertised number is a highlight rather than a representative expectation. An honest education provider can say, “Here is one example of a very strong improvement; outcomes vary, and this is not a typical forecast.”
Stage 3: understand why the best result is unusually fragile
Suppose the true contribution of a learning routine is modest but positive. Small samples produce variation: some contain learners already ready to accelerate; others contain students who miss lessons or need more prerequisite work. Every cohort also meets different topics and teachers. Even with an identical approach, the observed result moves. Selecting the maximum among those moving observations tends to select an upward fluctuation alongside any genuine advantage.
A simple analogy is drawing several handfuls of marbles from a mixed bag. If we choose the handful containing the largest proportion of blue marbles, the next handful drawn from the same bag will probably have fewer blues. Nothing about the bag changed. We changed our expectations by selecting an extreme observation. In learning, there are additional complications: teachers respond to pupils, tests are imperfect and classrooms are not interchangeable. The analogy explains the selection problem, not the full causal story.
Regression to the mean does not guarantee that every large gain will shrink. Some innovations really are exceptional. Nor does a subsequent smaller result prove the first estimate was false. A better phrasing is that selecting an exceptional observed gain gives us a specific reason to expect some shrinkage when we repeat the evaluation, unless strong independent evidence shows otherwise.
Stage 4: separate the method from the unusually helpful people
A pilot sometimes succeeds because a highly skilled teacher invented the routine and knows precisely how to adjust it. When other teachers reproduce the printed worksheet, they are not reproducing that teacher’s judgement, timing or relationships. A method can be more than a document. The active ingredient may include diagnostic questioning, responsiveness to mistakes, classroom norms or the time needed to rehearse examples.
Consider a Primary 6 vocabulary routine that asks pupils to use new words in precise sentences. A veteran tutor notices that one pupil confuses reluctant with resistant, another cannot choose a suitable context, and a third uses the word correctly but writes a sentence without evidence. The tutor adjusts the next example for each child. When the routine is copied as a list of twenty sentences to complete independently, those adjustments disappear. A weaker result does not mean vocabulary practice is useless. It may mean the pilot relied on skilled interaction that the scaled version did not contain.
Write down what teachers actually did, not just what the lesson pack said. Which decisions were essential? Which can be trained? Which need specialist knowledge? Which are too expensive to reproduce? A programme that only works when an exceptional individual runs it may still be worthwhile, but its scale-up claim must explain the staffing constraint.
Stage 5: avoid treating enthusiasm as a fixed resource
Novelty can bring extra attention. Teachers prepare more carefully because colleagues will observe the pilot. Families respond promptly to reminders because they know they are participating in something new. Pupils may enjoy a special name, device or invitation. These are genuine conditions that can support learning, but they may not remain unchanged over several years.
The answer is not to subtract enthusiasm as though it were meaningless. Motivation and relationships can be part of effective teaching. Instead, ask whether the programme requires special attention that the school can continue to provide. If the innovation needs thirty minutes of additional teacher preparation per student each week, its first result is incomplete until that resource is made explicit. A school can sustain some intensive routines but should choose them knowingly.
A parent choosing extra English help faces the same issue. The first month may include additional consultations, sample feedback and frequent contact. What will the normal weekly service look like in month four? Good questions address both learning benefit and ordinary delivery, rather than only the attractive launch experience.
Stage 6: distinguish a better test score from better transferable learning
A pilot can look strong because the assessment is close to the practice. Pupils may learn the exact structure of an exercise and gain confidence. That is useful. Yet it is different from being able to handle unfamiliar contexts independently. If a science pilot teaches one set of experimental-design questions and then assesses near-identical items, its result may reveal successful rehearsal rather than broad reasoning transfer.
A fair evaluation uses measures that match the intended outcome without merely reproducing the teaching materials. For mathematical proportion, include familiar ratios and novel word problems. For English writing, score an unfamiliar prompt with a clear rubric, ideally without scorers knowing which group used the new routine. For vocabulary, include recognition, accurate use in writing and a later application rather than only the original matching exercise. These are examples of stronger inference, not a prescription that every topic needs the same test format.
The Institute of Education Sciences emphasises sound outcome measures that are valid, reliable, appropriate to the context and not excessively aligned to the intervention. This matters because a promising method should improve the learning we care about, not merely the number it was trained to optimise.
Stage 7: ask what would have happened without the pilot
Students often improve with time, school teaching, exam preparation, reading experience or maturation. When an intervention begins just as those influences gather strength, a simple before-and-after comparison can credit the programme with everything that changed. The mistake is especially tempting when the first cohort starts after a disappointing test. Some subsequent rebound may have happened even without the new method.
A meaningful comparison asks what similar learners would probably have achieved under ordinary arrangements. Random assignment can strengthen that comparison when feasible and ethical. A matched comparison can be informative when its limitations are transparent. A school might alternatively use repeated measures and carefully document external changes while recognising that causal certainty is limited. Small tutors cannot always run formal experiments; they can still be accurate about what their evidence supports.
Do not confuse this with our neighbouring topic on fair programme comparisons. The specific problem here is that a promising result was selected as a winner from a larger search. Even a reasonably designed comparison can be exaggerated when its most favourable result is singled out among many trials. Comparison integrity and winner selection are two different gates.
Stage 8: check whether the chosen pupils were unusually ready
Pilots often recruit volunteers, motivated teachers or pupils whose families actively opt in. Their willingness can make delivery easier and improve attendance. That is not bad. It does change whom the result describes. If a reading programme was tested among parents who could support nightly practice, we should not assume the same schedule is accessible to every family.
A useful pilot report therefore describes selection. Who was invited? Who accepted? Who was excluded by travel, cost, language, timetable or eligibility rules? Which learners completed the intervention? Did support requirements differ between a highly engaged site and a typical one? The answer helps educators anticipate adaptations without overstating the original result.
There is an important distinction between reach and effect among recipients. A routine may be excellent for the pupils who regularly attend but fail to reach children facing transport difficulties. Another routine may show a slightly smaller improvement among attenders yet serve far more of the intended population. Schools need both numbers to understand overall value. Do not turn this discussion into a new article about participation funnels: here the point is whether selecting a particularly cooperative pilot created a misleading forecast.
Stage 9: choose replication that actually tests the claim
A second run with the same charismatic teacher, same class type and same worksheet may show that the first result was not a complete accident. It does not answer how the programme travels. Independent replication changes at least one important condition while preserving the core idea: another teacher, another class, a different school, a later cohort or a new assessment. A thoughtful sequence might first repeat the routine with comparable learners to check stability, then try a less familiar context to investigate transportability.
Replication should not become an obstacle course designed to defeat an otherwise useful method. If the intended service is one specialist tutor teaching three learners, its success does not require proof that every teacher can teach thirty learners with no adaptation. Generalisation should be judged against the actual claim. A local specialist promise can be narrow and honest. A claim about every Singapore school needs a broader test.
Before replication, specify the outcome that will count. Otherwise the second team may simply search again for a new winning number. The discipline is to test the earlier expectation, not to create an entirely new highlight reel each time.
Stage 10: scale delivery before scaling publicity
Many programmes fail at expansion because their real resources were hidden inside a small pilot: the founder taught every lesson, researchers supplied materials, teachers volunteered overtime or families received individual support. Once expanded, those contributions cannot be distributed across a hundred classrooms without a plan. The education question is not merely “Does this routine work?” It is also “Can ordinary staff deliver its essential components with the available time, training and safeguards?”
Build a scale-up budget that separates teacher time, student time, administrative work, training, assessment, materials, ongoing coaching and opportunity cost. The last item matters. A new programme that improves one target but displaces essential reading or mathematical practice may have a different net value from its advertised single-outcome gain. Decisions should consider the full school day.
The EEF programme pipeline is useful precisely because it moves through increasing evidence and implementation demands rather than treating a pilot success as automatic permission for universal rollout. A school can copy the logic without copying a large research bureaucracy: small trial, explicit review, independent repeat, wider test and continuing monitoring.
Stage 11: what a realistic school pilot dashboard should contain
The dashboard should start with purpose and population, not the biggest green number. Show the learner group, starting profile, attendance, teaching time, baseline measure, comparison approach and the preselected primary outcome. Then show a range of results, including uncertainty and any adverse outcomes. If data are limited, state that openly. An honest graph with wide uncertainty is more useful than a precise-looking average whose meaning is unknown.
A second page should report delivery: which teachers taught the programme, what training they received, which components were completed and what adaptations occurred. A third should report future feasibility: costs, timetabling, workload and family accessibility. One eye-catching example may be presented as a story, but it must be labelled as an example rather than as the expected effect.
For a small tuition centre, the equivalent may be a two-page internal note. It does not need academic jargon. What did the learner struggle with? What did we teach? What evidence changed? What would we check next month? What cannot we tell from one student’s improvement? These questions discipline the story without stripping away its humanity.
Stage 12: a worked English example, from promising to credible
A school introduces a six-week routine for editing paragraphs. Pupils first identify a claim, then select evidence, then explain why the evidence supports the claim. The pilot teacher chooses a persuasive writing topic the class knows well. End-of-unit scripts look stronger and a small group improves substantially on the teacher’s rubric. The staff team is excited, which is appropriate: they have discovered a plausible instructional idea.
But suppose the school also tested peer comments, model answers and timed rewriting in different groups. The selected routine was simply the one with the highest visible gain. Moreover, its assessment prompt resembled the lesson examples. To strengthen the conclusion, leaders can retain the routine, define the distinctive teaching moves, select an unfamiliar writing prompt, use the same rubric across groups and train an independent scorer to apply it. They can test another teacher with a group that did not volunteer for a special pilot.
If the follow-up effect is smaller but consistent, that is not necessarily failure. The revised finding may be more decision-ready because it estimates what a typical classroom can reproduce. A truthful improvement of moderate size is often more valuable than an impressive number that vanishes when ordinary conditions return.
Stage 13: a worked mathematics example, from best class to fair expectation
Imagine a Secondary 1 mathematics tutor in a three-pupil group near Clementi has tried multiple explanations for negative-number operations. One explanation uses money owed and money held; another uses a number line; a third starts with formal sign rules. A pupil finally understands subtraction using the number line, and the next quiz jumps from 40 to 80 per cent. The tutor is right to celebrate. The pupil has made a meaningful step.
The dangerous leap is advertising, “Our number-line method doubles everyone’s score.” The pupil may have entered the session after already learning part of the idea in school. The quiz may contain fewer questions or easier distractors. The number-line explanation may have been exactly right for this particular misconception. None of those possibilities makes the learning moment less real. They make the general claim less defensible.
A better routine records the exact misconception, what the pupil could not explain before, what changed during tutoring and whether the pupil can solve a fresh negative-number problem a week later. Another learner with a different misconception may need a different representation. Good tuition is a sequence of diagnoses and responsive decisions, not a competition to discover one miraculous worksheet.
Stage 14: a science example that reveals the hidden denominator
A Science department compares twelve small changes to practical lessons. One approach results in five pupils producing sophisticated explanations of fair testing. The headline says, “Five pupils reached distinction quality.” Yet twenty-eight pupils participated, several were absent at final assessment, and the strongest five had already shown unusually good scientific vocabulary. The success story contains a numerator but hides the denominator.
To understand the programme, examine all intended learners and the changes in their work. Look at the weakest explanations, not only the best. Did students learn to distinguish the changed variable from the measured variable? Could they spot a confounding factor in an unfamiliar experiment? Did the programme support learners who initially lacked reading fluency? A scaled practice should be evaluated against the population it promises to help.
This example touches missing-data and subgroup issues, each of which deserves its own owner. In this article the relevant question is narrower: why did the pilot leader choose that programme from the twelve tried? The most dramatic handful of scripts did not necessarily identify the approach with the strongest average benefit.
Stage 15: the parent route — how to read impressive improvement stories
Parents do not need a statistics qualification to recognise the selection problem. Ask whether the story describes the typical learner or a particularly successful one. Ask what the starting level was and how improvement was measured. Ask whether the result is from a small pilot, routine delivery across many terms, or an independently checked comparison. Ask what happens when a pupil’s needs differ from the featured example.
Pay attention to how providers answer uncertainty. A trustworthy educator can explain that children progress at different rates, discuss realistic goals, describe the frequency of learning checks and specify which parts of progress depend on practice outside lessons. Claims of guaranteed grade jumps warrant scrutiny because no tutor controls every influence on a learner’s outcome.
Also consider the opportunity cost. A shiny programme may be expensive and time-consuming. If the child needs foundational decoding, a general motivational product may not address the immediate problem. If mathematics errors stem from unstable fractions, the next lesson should repair fractions before promising advanced examination strategies. The best question is not, “Which success story looks biggest?” It is, “Which mechanism is relevant to this child, and how will we know if it is helping?”
Stage 16: the learner route — do not judge your own strategy by one exceptional week
Students perform a version of winner selection whenever they change study techniques according to the highest recent quiz mark. A learner uses flashcards on Monday, watches explanations on Tuesday, practises exam questions on Wednesday and gets a high score on Thursday. The learner concludes that Wednesday’s strategy must be responsible. Perhaps it helped. Perhaps Thursday’s questions matched knowledge already held, or the learner slept better, or school revision covered similar ground.
Instead, run a modest personal learning experiment. Select one specific target, such as using quotations accurately in literature answers. Practise a defined routine for several sessions. Use different but comparable prompts and record whether you can work independently after a delay. If you alter the routine, change one important feature at a time when practical. Keep the aim stable enough that the evidence makes sense.
This is not a command to become obsessed with measurement. Children should have time for exploration and play. It is a method for making important learning decisions without overreacting to one remarkable performance or one disappointing paper. Treat both as information, then look for patterns.
Stage 17: the teacher route — keep discovery, improve the inference
Teachers are skilled at making changes mid-lesson. They should not abandon that responsiveness because evaluation is difficult. The goal is to distinguish improvement work from evidence of effectiveness. During improvement work, try representations, invite student explanations, notice confusion and make adjustments. During evaluation, record enough of the process to understand what the observation can and cannot establish.
A practical four-column record can help: lesson question, intervention change, learner response and possible alternative explanation. For example: “How do pupils explain a graph’s gradient?”; “Used a distance-time motion story”; “Seven of nine now interpret units correctly”; “School taught speed in Science the previous week.” The final column does not destroy the success. It prevents false certainty.
After several lessons, decide whether the idea is ready for an independent check. Can another colleague teach it using a clear description? Does it work on a different topic or novel question? Have we spent more time on this routine than the old one? Could the group benefit even more from the alternative we displaced? Asking these questions creates a professional culture in which evidence improves practice instead of simply confirming enthusiasm.
Stage 18: the leadership route — protect the right to find a smaller effect
An institution creates bad evidence when it rewards only the biggest outcomes. If leaders expect a pilot to justify a decision already announced, staff may hesitate to report that benefits were uneven or costs higher than expected. Responsible leadership treats a well-run pilot with a small or null effect as useful knowledge. It prevents wasted time and helps refine the next version.
Start with a decision charter: the learner need, the proposed mechanism, who bears the cost, what evidence will trigger continuation, and what would trigger redesign or discontinuation. Make those thresholds realistic. A pilot intended to test feasibility might continue despite limited attainment data if delivery is safe and acceptable. A claim of academic effectiveness should not be approved simply because teachers enjoyed the experience.
When a first result is spectacular, commission a second analysis or an independent replication rather than scaling publicity at once. When a second result is smaller, ask what was learned about selection, context and implementation rather than punishing the team. The public deserves stable, usable improvement, not a succession of irresistible pilot headlines.
Stage 19: what research supports — and what it cannot settle for your school
The EEF explicitly structures its programme evidence pipeline from development and pilots to stronger evaluations and scale-up. Its guidance makes clear that scaling involves increasing confidence in effects and capacity to deliver. The US Institute of Education Sciences’ Standards for Excellence in Education Research call for clear documentation of mechanisms, implementation, comparison conditions, outcome measures, costs and distal outcomes. These principles explain why one highly selected pilot result cannot carry an entire expansion decision.
Neither source provides a universal correction factor such as “subtract five marks from every promising pilot.” The amount by which an estimate shrinks depends on sample size, variation, number of attempts, selection rule, measurement quality and whether the original method genuinely was superior. Even sophisticated statistical correction cannot rescue missing records of what was tried. In ordinary school settings, transparent repetition and careful comparison are often more useful than pretending to compute a precise adjustment from incomplete evidence.
A further caveat: some educational innovations improve when scaled because wider use allows teachers to refine materials, share training and stabilise delivery. Winner selection is a warning about confidence, not a law that large programmes must deteriorate. A good evidence system remains open to improvement and to disappointment.
Stage 20: common failure modes and their repairs
Failure: the school’s best class becomes the benchmark for every class. Repair: present the full range of outcomes and distinguish the showcase case from the typical result. Explain why the best class was chosen. If selection followed a broad search, flag the uncertainty explicitly.
Failure: an early pilot score is treated as a causal estimate. Repair: identify background improvement, comparison conditions and alternative explanations. Choose a future evaluation suited to the decision. A feasibility result is not an impact result.
Failure: a method is reduced to a worksheet. Repair: document the teacher moves, student prerequisites, timing, coaching and monitoring required to preserve its mechanism. Test delivery by ordinary staff, not only the inventor.
Failure: the programme keeps changing while its first claim stays fixed. Repair: version the materials and state which version each estimate describes. Updating a method can be wise; treating evidence for an earlier version as full proof of a later one is not.
Failure: only successful learners appear in the results. Repair: report who was eligible, invited, enrolled and assessed. Explain missing or excluded cases rather than quietly replacing the intended population.
Failure: the school is embarrassed by smaller replication effects. Repair: treat smaller, more stable effects as better information. A careful adjustment can save years of misplaced effort and support more realistic family expectations.
Stage 21: a four-week micro-pilot protocol
Week one is for the question. Define the precise learning difficulty and write down what would count as improvement. Choose a measure that reflects the target skill and collect baseline work. Record which learners are included, which ordinary lessons continue and what new resources the intervention demands.
Week two is for delivery. Teach the routine consistently enough that it has a nameable form, but retain professional discretion to respond to learner needs. Record deviations and attendance. Do not make a marketing claim in the middle of the process merely because one pupil has a breakthrough.
Week three is for a fresh check. Use a novel task rather than relying solely on practised items. Gather evidence from all intended learners where practical. Note what other teaching and revision occurred. Ask pupils what they could explain without help rather than treating enjoyment as sufficient proof of learning.
Week four is for a decision note. Identify what the pilot suggests about feasibility, acceptability and learning; what remains uncertain; whether a smaller independent repeat is worthwhile; and what extra resources wider delivery needs. If the evidence is weak, the next action may be to improve the measure rather than expand the programme. The protocol is a practical school adaptation, not an officially validated four-week trial design.
Stage 22: an audit card for the next remarkable claim
When a very large improvement appears, read the story in a particular order. First establish the original educational need. Then identify the exact intervention, not its marketing name. Next determine the number of candidates or versions tested before a winner was chosen. Ask whether the comparison was credible and the outcome independent of the teaching material. Identify the pupils and teachers who contributed to the reported result. Examine the time, staffing and assessment cost. Finally, ask what happened in the first independent repeat.
That order matters because it prevents a compelling result from becoming the starting assumption for every subsequent explanation. If you begin with “Why is this programme so successful?” you may never discover that the data support only “What did this promising pilot show?” Good enquiry asks a weaker but answerable question before a stronger but unsupported one.
The audit card also helps schools communicate with families. A responsible message could be: “The pilot suggests pupils can learn the new explanation routine and teachers can deliver it. The achievement estimates are preliminary because the group was small and selected. We will test unfamiliar work with another cohort before recommending a wider rollout.” That is confident and useful; it is not evasive.
Frequently asked questions
Is the winner’s curse the same as regression to the mean?
Not exactly. Winner selection describes how an extreme result becomes the chosen example. Regression to the mean describes a statistical tendency for extreme observations to be less extreme on later measurements under relevant conditions. Winner selection often creates the circumstances in which regression to the mean matters. Neither term automatically identifies how much of a school’s result was due to the actual teaching.
If a pilot is small, should we ignore it?
No. Small pilots can reveal whether teachers can run the lesson, whether learners understand the instructions, whether families can attend and whether the proposed measure is practical. They can generate promising hypotheses. The mistake is treating them as precise forecasts for an entire population.
What if the successful pilot had no comparison class?
It can still support a descriptive claim: these learners improved during the period. It cannot, by itself, tell us exactly how much improvement the programme caused. Ordinary teaching, maturation, other revision and changes in the test could also matter. Use cautious language and plan better evaluation if the decision warrants it.
Do repeated tests eliminate selection bias?
Repeating a selected claim in a genuinely new cohort can improve confidence. Repeating the entire search for a fresh best result is weaker, because another winner will emerge by the same selection process. The new test should specify what is being checked before the results are inspected.
Can a private tutor responsibly publish student success stories?
Yes, when consent and privacy are respected, the evidence is accurately described, and an exceptional case is not represented as a guaranteed or average outcome. A learner’s story is valuable as an illustration of a mechanism, but it is not a substitute for representative evidence.
Does a smaller effect at scale prove teachers implemented badly?
No. Delivery may differ, but early estimates may also have been unusually favourable or measured on easier tasks. The learner population, available support, comparison condition and result measurement may change. Diagnose these possibilities instead of assuming teacher failure.
Is the best pilot always a mistake to select?
No. We often should investigate the most promising approach. The error is not selecting it; it is forgetting that the selection itself changes what the observed gain means. Selecting a candidate for further testing is sensible. Promising its exact first gain everywhere is not.
How is this different from the question of missing learners?
Missing learners concern whether outcomes omit people who did not complete a programme or test. The winner’s curse concerns choosing unusually strong observed results from multiple candidates. Both can inflate expectations, but they arise through different mechanisms and need different checks.
What should a parent ask before paying for a new education product?
Ask what skill it claims to change, how it was evaluated, whether the advertised result is typical, how teaching is adapted, what independent work the learner must do, and when progress will be reviewed. A provider able to describe its limitations may be more trustworthy than one offering an extraordinary number without context.
What is the strongest single question for school leaders?
“Would we still have chosen this programme if the most spectacular group had been one of the ordinary ones?” That question exposes how strongly the decision relies on an extreme observation. It does not answer the whole evaluation, but it is a good opening test.
Sources, further reading and links to neighbouring eduKateSG guides
- Education Endowment Foundation — Programme pipeline of evidence generation. Explains the distinction between early development, pilots, efficacy, effectiveness and scale-up; consulted October 2026.
- Education Endowment Foundation — Evaluation guidance and resources. Research design, study plans, reporting and implementation resources; consulted October 2026.
- Institute of Education Sciences — Standards for Excellence in Education Research. On mechanisms, counterfactual conditions, sound measures, implementation, costs and longer-term outcomes; consulted October 2026.
- What Works Clearinghouse — Handbooks and study review standards. How rigorous education evidence is reviewed; current reference page consulted October 2026.
- OECD — Unlocking High-Quality Teaching, 3 April 2025. Discusses how teaching practices interact with context and implementation, without asserting that one technique fits every classroom.
- eduKateSG — How X Works hub. Broader mechanism explanations.
- eduKateSG — Secondary 1 Mathematics Tutor Clementi. A local small-group learning context; separate service owner, not a source of causal-effect estimates.
- eduKateSG — How Randomisation Works. The neighbouring owner for random assignment; this article instead owns selecting an unusually successful pilot from many attempts.
Editorial note: Illustrative scores, classes and schedules in this article are constructed teaching examples, not results from a real named school, pupil or trial. A pilot is permission to learn more, not permission to promise the biggest observed gain forever.