One class observes that compost-treated seedlings appear to have darker leaves. Another class studies new plants and finds no colour difference. Who is right? Both may have recorded their observations sincerely, yet the results do not establish the same conclusion. The gold standard of school scientific replication disagreement interpretation is to compare the actual questions, methods and evidence behind divergent studies, explain what the disagreement means and plan a further test only where it can reduce a specific uncertainty.
Quick answer: Keep original and replication datasets separate; identify exploratory versus pre-planned outcomes; compare experimental units, baseline conditions, observation rules, instrument limits and missing data; describe the observed differences without inventing a biological mechanism; and judge conclusions within the evidence. A negative replication does not automatically prove that the first result was fabricated or that the effect is impossible.
Quick navigation
- When two studies disagree, neither automatically becomes worthless
- A repeat observation is not necessarily an independent replication
- Exploratory and confirmatory observations have different roles
- Worked example: two fictional seedling datasets
- Read the actual pattern before calculating an impressive mean
- Arithmetic summaries are correct only within their meaning
- What can be concluded from a non-repeating effect?
- Observer definitions and lighting can generate apparent disagreement
- Random variation remains plausible with small samples
- Different baseline states can produce genuine context differences
- The independent treatment unit is the unit that received assignment
- Missing outcomes and excluded cases need visible reasons
- A third study should answer a specific unresolved question
- Do not change the target after the follow-up is disappointing
- A negative replication can strengthen scientific literacy
- Transfer example: two water-cooling studies
- Transfer example: filtration can create non-comparable outcomes
- Three learning pathways: repair, stabilise and extend
- A ninety-minute classroom lesson
- Break-tests for overconfident conclusions
- Frequently asked questions
- MOE context and connected Gold Standard reading
When two studies disagree, neither automatically becomes worthless
A school class observes that compost-treated seedlings appear to have darker leaves. Another class studies new seedlings under a defined colour-scoring procedure and finds no difference between treatment and control. The first class feels challenged; the second class claims it has disproved the idea. Neither reaction is a complete scientific interpretation. The two observations may be different because the physical phenomenon varies, the methods differ, the first result was selected after looking at many features, or natural variation obscures a smaller effect.
Replication disagreement is information about evidence, not an accusation. A careful investigator asks what each stage genuinely measured, what units were independently studied and whether the two sets of observations could reasonably be expected to match. If one class photographed leaves in afternoon sunlight and the other viewed them under fixed classroom lighting, their colour classifications might differ for reasons unrelated to the compost treatment.
Students learn something valuable when they resist choosing a winner immediately. The gold standard is to compare study designs and results, explain plausible differences and propose a test that would distinguish alternative explanations. A repeated result strengthens an idea only when the repetition supplies relevant new evidence under suitable conditions.
A repeat observation is not necessarily an independent replication
Measuring one seedling’s height three times can reveal disagreement between readings. It does not create three separately grown seedlings or three independent allocations of compost. Re-scoring the same photograph with another observer can test observer consistency, but it does not supply new biological growth evidence. Students should identify whether a new study introduces genuinely new experimental units and a meaningful opportunity for the hypothesis to succeed or fail.
A close replication attempts to test a similar question under closely matched conditions. A broader replication may change the setting to investigate whether an explanation transfers to another environment. Both are potentially valuable, but they support different inferences. A colour effect appearing in one plant variety but not another may reveal a real context-dependent process rather than an error.
Do not define replication by similar titles alone. Two groups both calling a project ‘Does compost improve plants?’ may measure different outcomes. One might use stem-height gain, another leaf count and a third subjective greenness. Their results can be separately valid yet fail to provide a direct replication of the same physical claim.
Exploratory and confirmatory observations have different roles
An exploratory observation can occur while a class is investigating another question. Pupils studying stem-height gain may notice a possible difference in leaf colour, then examine whichever photographs seem most striking. That can generate a hypothesis, but it was not the main measure defined in advance. A confirmatory follow-up chooses a specific colour outcome before collecting its own results.
The distinction matters because selecting one eye-catching feature from many potential observations can exaggerate how surprising it seems. If the first class noticed colour only because height, leaf size and other features did not provide a dramatic pattern, the selected result should not be treated as equivalent to a new test that fixed the colour measure before observation.
A strong report preserves both stages: ‘We noticed a possible colour pattern in the first exploration; a separate class tested that question under pre-specified rules.’ The question is then whether the new evidence strengthens, weakens or changes confidence in the idea. A disappointing follow-up should remain visible rather than be recast as proof of some different outcome.
Worked example: two fictional seedling datasets
Consider two independent illustrative classroom stages, each involving six base-medium and six compost-medium seedlings. The first stage is exploratory. After noticing a visual difference, students assign leaf-colour scores from old photographs using a simplified ordinal scale: category 1 means lighter-looking, category 2 intermediate and category 3 darker-looking under the chosen reference. Because the reference and observation method were not pre-specified, the stage-one result warrants particular caution.
Stage two uses new seedlings in independently prepared pots. It fixes the colour category definitions and observation date before results appear, attempts consistent lighting and records assignments and uncertain classifications. The table below contains invented instructional scores, not measured pigment concentrations or real findings. It is intended to teach comparison between an initial apparent signal and a follow-up that does not reproduce the same pattern.
The two stages retain separate identities. The class should not merge the most favourable treated leaves from each stage into a combined display and silently omit less favourable results. Nor should it describe repeated scores from one plant as independent treatment assignments. Each study’s original group size, physical units, outcome status and observation procedure need to remain understandable.
| Study stage | Growing medium | Fictional leaf-colour scores | High category (3) |
|---|---|---|---|
| Exploratory | Base | 2, 1, 2, 2, 1, 2 | 0 of 6 |
| Exploratory | Compost | 2, 3, 2, 3, 2, 3 | 3 of 6 |
| Independent follow-up | Base | 2, 2, 2, 1, 2, 2 | 0 of 6 |
| Independent follow-up | Compost | 2, 2, 1, 2, 2, 2 | 0 of 6 |
Read the actual pattern before calculating an impressive mean
In the first exploratory stage, the base-medium scores are 2, 1, 2, 2, 1 and 2. Four are category 2 and two are category 1, with no category-3 scores. The compost scores are 2, 3, 2, 3, 2 and 3: three are category 2 and three are category 3. This is the visible pattern that prompted interest. It is a descriptive difference in the category distribution.
In the second, pre-planned follow-up, the base group scores 2, 2, 2, 1, 2 and 2, while the compost group scores 2, 2, 1, 2, 2 and 2. Each group has five intermediate and one lighter category, with no category-3 scores. The follow-up therefore shows matching frequency distributions under its chosen method, unlike the initial observation.
A simple table of category counts often communicates this result more accurately than a single average. When visual scores are ordinal, a difference between categories 1 and 2 need not correspond to the same measurable biological difference as the gap between 2 and 3. The categories are a defined classification system, not a laboratory measure of chlorophyll.
Arithmetic summaries are correct only within their meaning
The exploratory base scores total 10, giving a coded mean of 10 ÷ 6 ≈ 1.67. Its compost scores total 15, giving a coded mean of 15 ÷ 6 = 2.50. In the follow-up, both group totals are 11, so both coded means are approximately 1.83. These calculations are arithmetically correct for the fictional category codes and make the difference between study patterns visible.
However, a gap of 0.83 coded category points in the first stage does not mean a physically measured increase of 0.83 units of leaf pigment. The score is ordinal and was assigned under a simplified method. Reporting many decimal places would add a veneer of precision without supplying more evidence about the actual plant chemistry or uncertainty.
The conclusion should say what the categories show, not what the pupils wish they measured. The second class did not reproduce the first stage’s higher frequency of dark-category leaves. That is a meaningful finding about the follow-up’s evidence, but it is not a universal statement that compost cannot influence any leaf property.
What can be concluded from a non-repeating effect?
The most defensible combined conclusion is that one exploratory observation suggested a colour association, while a separately defined follow-up did not show the same association under its method. Thus the evidence for a consistent effect is limited. The follow-up narrows confidence in the original suggestion, but cannot settle every possible plant variety, growing condition or true underlying effect.
Do not dismiss the first students as dishonest simply because their result did not repeat. Their photographs may accurately show what they saw, even if lighting or selection caused a misleading impression. Do not declare the second class infallible either. A small new sample, coarse categories and natural variation can limit its ability to detect a subtle effect. Both studies need appropriate evidence evaluation.
A strong student conclusion states the observations separately, identifies one or two material limitations and suggests which new evidence would answer the unresolved question. Generic phrases such as ‘the results may be wrong because of human error’ are much weaker than naming a specific comparison problem, such as inconsistent leaf-colour scoring under different lighting.
Observer definitions and lighting can generate apparent disagreement
Leaf appearance depends on illumination, camera exposure, white balance, angle and the reference used by the observer. A dark-looking leaf in one photograph may look intermediate in another without undergoing biological change. A valid follow-up needs a consistent visual procedure and clear examples for category boundaries if it uses such a simplified measure.
Ask two students to classify representative new images independently before discussing scores. If the observers disagree frequently, improve the definitions or record uncertain cases rather than force agreement by always choosing the category favourable to compost. A simple observer-agreement check can be useful, though perfect agreement does not by itself prove the category measures the biological property the class cares about.
Where appropriate, keep treatment labels hidden from the scoring observers to reduce expectation effects. The responsible teacher must still know all information necessary for safety and proper handling. Blinding can help observer fairness, but it does not replace valid apparatus, independent experimental units or a meaningful scientific question.
Random variation remains plausible with small samples
Each fictional group contains six seedlings. Such small samples can naturally differ in starting state, development or leaf appearance. Even a fair random assignment does not guarantee perfectly matched groups. A notable difference in one small study may be less apparent in the next simply because different plants were selected and allocated.
Students should not equate the presence of an arithmetic difference with strong evidence of an effect. Ask whether the sizes and variation of the categories support a broad claim, and whether the original result was selected after many outcomes were considered. In an advanced research context, uncertainty can be quantified through appropriate statistical designs, but a classroom should not invent an accuracy percentage from a few scores.
The answer is not automatically to run studies until one achieves the desired pattern. Repetition should have an explicit purpose and a suitable method. A negative replication is a legitimate result to preserve alongside a positive initial observation, not an obstacle to be discarded.
Different baseline states can produce genuine context differences
Seedlings may begin with different leaf sizes, colours, varieties, root development and exposure histories. If one class selects only strong seedlings and another includes plants at several developmental stages, the resulting leaf-colour distributions may differ regardless of treatment. A relevant baseline description helps readers see how comparable the groups were at the start.
Similarly, the treatment itself may not be operationally identical across classrooms. A named compost mixture can differ in composition, amount or interaction with the base medium. An intended treatment comparison requires sufficiently defined inputs and current safe teacher-approved procedures. A lack of detail about what was actually applied makes replication claims weaker.
A follow-up can record the aspects that matter without attempting to control every possible environmental factor. Specific plausible conditions such as plant variety, lighting and observation period deserve priority over a long speculative list. Scientific discipline means focusing on relevant competing explanations.
The independent treatment unit is the unit that received assignment
Six seedlings in separate independently assigned pots can provide six treatment units per group in the simplified classroom design. Six leaves cut or observed from one treated seedling do not create six independent treatments. If six seedlings in one tray all receive the same medium assignment, their treatment experiences share the tray. Students should recognise that the number of readings may exceed the number of independent units.
When comparing studies, look for these structural differences. The first class might have used one shared tray per treatment, while the second used individual pots. A table listing six seedlings per group would then conceal materially different experimental designs. Counting observations as though they were independent can exaggerate confidence in a result.
This principle transfers to water-cooling experiments, classroom surveys and sensor networks: repeated readings from one device or one location do not automatically represent independent comparisons. The gold standard identifies the relevant unit before treating more data rows as stronger replication evidence.
Missing outcomes and excluded cases need visible reasons
A seedling that dies before colour scoring should not simply vanish from the treatment group. Its absence could be related to growing conditions and thus change how survivor-only results should be interpreted. A lost photograph, on the other hand, is a recording problem whose relation to leaf colour may be different. Document the original number assigned and the reason each planned outcome is missing.
If two groups both have five measured leaves but began with six independent plants, the final equal denominators do not guarantee the selection remained fair. The class must examine which cases were lost and whether missingness might depend on the treatment or outcome. A valid zero or lowest category also differs from a missing classification.
Do not put a guessed colour score into the raw table because a graph needs a complete set of bars. Any hypothetical sensitivity scenario should be clearly described as hypothetical. The priority is to preserve honest evidence and understand its limits.
A third study should answer a specific unresolved question
After divergent results, running an identical third study without reviewing methods may leave the cause of disagreement unknown. A better plan identifies plausible explanations and decides what evidence would distinguish them. If lighting or observer categorisation differed, choose consistent viewing conditions and validated scoring examples. If baseline plant state differed, record relevant starting characteristics and use an appropriate allocation.
The third study should define its main outcome, independent units, time frame and handling of uncertainty before pupils see new scores. A three-stage project is not a majority vote in which two positive outcomes automatically defeat one negative one. Different studies may have different evidentiary quality, and repeated observations on the same units can provide less independent information than their row counts suggest.
A meaningful design says: ‘We will examine a new set of comparable seedlings using an agreed colour classification under consistent light, with independent treatment allocation.’ Its aim is to reduce ambiguity, not guarantee that a particular result appears.
Do not change the target after the follow-up is disappointing
Suppose the follow-up shows no leaf-colour difference but reveals a slight difference in stem thickness. It is appropriate to record that as a new exploratory observation. It would be misleading to declare that the follow-up ‘confirmed the compost effect’ by switching from colour to thickness after inspection. The original hypothesis and measured outcome must remain visible.
The same issue can occur in an educational presentation when pupils select only the largest-looking difference from a long list of measured features. If the class never stated which outcome would answer its question, its headline may be driven by chance and preference. Pre-specified outcome definitions help protect the interpretation.
Students should understand that curiosity is not being punished. The new stem-thickness idea can inspire another study with a meaningful measure and suitable instrument. Exploration and confirmation are complementary stages, provided they are reported honestly.
A negative replication can strengthen scientific literacy
A class may feel discouraged when a carefully conducted follow-up finds no difference. The teacher can show why this remains valuable: the result helps evaluate how dependable the original apparent pattern was. It may suggest that the effect is small, context-dependent, sensitive to measurement or not reliably detectable under the present method.
The conclusion should not exaggerate in the other direction. ‘We found no difference in these groups using this score’ is narrower than ‘compost has no effect on plants’. Some effects might require different valid measurement methods or more independent evidence. A scientific statement is strongest when its scope matches the actual question and observations.
The class can acknowledge the negative follow-up in its final project report and discuss whether further investigation is warranted. Honest null or conflicting results are better educational outcomes than an invented universal success story.
Transfer example: two water-cooling studies
One group reports greater cooling when a fan moves air near a vessel, while another group observes little difference. Before claiming a contradiction, compare starting temperatures, vessel shapes, water volumes, fan settings, observation intervals and thermometer resolution. Two investigations can use the word ‘cooling’ while measuring different outcomes or operating under different physical conditions.
If the second group uses a thermometer displaying only whole degrees, it may not reliably detect a small difference that a more suitable instrument reveals. If the first group puts one vessel by an open window, airflow and location may be confounded. The correct evaluation examines each study’s design and actual measured evidence before asserting that one result falsifies the other.
A useful new test would address the most material difference in method. Repeating the original setup without identifying the uncertainty may produce another ambiguous outcome. The aim is comparison that can distinguish explanations, not repeated performances of a memorised demonstration.
Transfer example: filtration can create non-comparable outcomes
A class investigating a filter may measure water clarity, while another class measures the volume of filtrate collected over ten minutes. The first could report clearer water, and the second report slower flow. These are not directly contradictory results; they address different performance dimensions. An inference about filter effectiveness requires specifying what ‘effective’ means in the context of the question.
A genuine replication would use a comparable definition of the relevant outcome, apparatus and observation period. If one study treats clearer water as success and the other treats faster flow as success, the results cannot simply be pooled or ranked without a justified objective. Pupils should learn to separate distinct questions rather than force a single winning product.
This transfer problem demonstrates the wider lesson: apparent disagreement may come from different physical measures rather than a difference in reality. Exact words, units and timing can determine whether two experiments test the same claim.
Three learning pathways: repair, stabilise and extend
Repair: Show two photographs of similar leaves under different light and ask why their apparent colours may differ. Present one simple result from each classroom and ask learners to describe only what was observed, without deciding which class is more honest. Introduce the words ‘same question’ and ‘same method’ before technical terms about replication.
Stabilise: Let pupils count the scores in each fictional group and calculate the coded means while recognising the ordinal nature of the scale. Ask them to write two separate sentences: one for the exploratory stage and one for the follow-up. Their conclusion should state that the original higher dark-category frequency was not repeated in the new sample.
Extend: Ask stronger learners to design a third study that addresses lighting, observer calibration, starting leaf condition and independent units. They can also discuss selective outcome reporting and the limited inference of small samples. The key assessment is whether they can choose a specific uncertainty to investigate rather than repeat a vague claim.
A ninety-minute classroom lesson
Begin with ten minutes presenting the apparently contradictory observations. Spend fifteen minutes distinguishing exploratory from pre-specified confirmatory work. Allow twenty minutes to calculate and compare the fictional score distributions. Use fifteen minutes to examine lighting, observer rules, independent treatment units and sample size.
Give students another fifteen minutes to design a focused new investigation, ten minutes to write an evidence-based conclusion and five minutes to transfer the reasoning to water cooling or filtration. The timetable is an illustrative teaching design, not a mandatory MOE lesson prescription. Teachers should use safe, approved investigations or clearly labelled fictional data.
A strong exit answer names the observed disagreement, explains at least one plausible source of non-comparability, and proposes evidence that would meaningfully reduce uncertainty. Reproducing the preferred numerical result is not the teaching objective.
Break-tests for overconfident conclusions
Does re-scoring the same photograph provide new biological replication evidence? No. Can a positive exploratory pattern be treated as the same kind of evidence as a pre-specified independent follow-up? Not automatically. If observers use different light and category definitions, must a difference be biological? No. If the groups contain repeated leaves from just one plant, is the sample size as large as the leaf count suggests? No.
Further tests include a dead plant silently removed from one group, a new study that changes its outcome after seeing a disappointing result, an ordinal score described as exact pigment concentration and a third study whose outcome is selected merely to win a two-to-one vote. For each challenge, the pupil should identify the relevant design or inference error, not simply say that the experiment was unfair.
Perhaps the most important question is: ‘If the third study also finds no difference, how should the original surprising observation be described?’ The strongest answer preserves the original fact as an observation, acknowledges that independent support remains limited and describes any further test only if it could answer a meaningful remaining question.
Frequently asked questions
Does a failed replication mean the first class cheated? No. Natural variation, observer differences and method problems can produce different results. Review the evidence before alleging wrongdoing. Can re-measuring the same sample count as replication? It may test measurement consistency, but does not necessarily add independent experimental units.
Are coded colour scores precise measurements of pigment? No. An ordinal visual category is not a biochemical concentration. Why is a pre-specified outcome important? It protects against choosing whichever result looks favourable after observing the data. Does a matching pair of studies prove a universal effect? Not automatically; scope, method and variation still matter.
What should we do if replication is negative? Report it, compare methods and limits, and decide whether a focused further study could reduce uncertainty. How many replications are enough? No universal school number guarantees certainty. How can parents help? Ask what each class actually measured and which differences could explain the conflicting conclusions.
MOE context and connected Gold Standard reading
The Singapore MOE Primary Science Teaching and Learning Syllabus emphasises evidence gathering, comparing data, formulating explanations and communicating justifications. Replication-disagreement reasoning builds on those inquiry skills; it is not a claim that every Primary learner must perform professional meta-analysis or specialised statistical hypothesis testing.
Related eduKate guides separate the adjacent learning steps: Exploratory Findings Confirmation Planning explains how to design the follow-up; Pre-Specified Outcome Definitions preserves the original outcome; Observer Agreement Checks evaluates scoring consistency; and Scientific Sampling Bias explains selection limitations.
Final principle: Two studies can disagree and both remain scientifically useful. A careful learner compares questions, methods, samples and uncertainty before choosing what further evidence is needed.