How Evidence Works | How Claims Become More or Less Believable

Data, Observation, Evidence and Conclusion Are Different Layers

An observation is something detected or recorded. Data are representations of observations. Evidence is data or information used in relation to a specific claim. A conclusion is the judgement drawn after interpreting that evidence.

World → observation → measurement/record → data → interpretation → evidence relative to a claim → conclusion.

Collapsing these layers creates false certainty. A number in a spreadsheet is not automatically evidence for the conclusion we care about. We still need to know what produced the number, what it measures, how uncertain it is and why it bears on the claim.

Evidence Is Relational: the Same Information Can Be Strong for One Claim and Weak for Another

A student’s repeated success on ten familiar questions is strong evidence that those particular procedures can currently be executed under familiar conditions. It is weaker evidence for independent transfer to unfamiliar questions, long-term retention or conceptual understanding.

This is why evidence quality cannot be judged without the claim. The question is never only “How good is this data?” but “How much should this data change belief in this exact claim?”

Provenance Is Part of Evidence Quality

Evidence becomes easier to trust and correct when its provenance is visible: where it came from, who produced it, how it was collected, which transformations were applied, and whether the original record can be inspected.

For research, provenance may include instruments, sampling, protocols, analysis code and source datasets. For historical evidence, it may include authorship, date, custody and context. For a student’s work, it may include whether the task was independent, timed, prompted or corrected during completion.

Without provenance, apparently precise evidence can lose much of its meaning because the route from world to record is hidden.

Independence Is Not Binary — Evidence Can Share Hidden Dependencies

Two sources are not fully independent merely because they are published in different places. They may use the same dataset, repeat the same witness, depend on the same measurement instrument, share the same methodological bias or cite one original report.

This matters because dependent evidence should not be counted as though every item were a new observation of the world. Ten articles repeating one unverified claim do not create ten independent confirmations.

Count evidence by independent routes to the world, not by the number of times the same signal is repeated.

Triangulation Is Strongest When Different Methods Have Different Failure Modes

Triangulation means approaching a claim through more than one evidential route. Its value is greatest when the methods fail differently.

A student’s capability might be examined through written work, oral explanation, delayed retrieval and unfamiliar transfer. Agreement across all four is more informative than four near-identical worksheets because each route exposes different failure modes.

In Science, converging measurement technologies can increase confidence when they rely on different assumptions. In History, independent documentary, archaeological and material evidence may corroborate one another. In everyday judgement, behaviour across different contexts can be more informative than one self-report.

Measurement Uncertainty Travels Into the Evidence

Measurements are estimates produced through instruments, procedures and models. Resolution, calibration, sampling, observer judgement and environmental conditions all contribute uncertainty.

NIST’s measurement guidance treats uncertainty as part of the measurement result rather than an embarrassing afterthought. The same principle belongs in education. A score of 74 is not infinitely precise evidence of capability. One paper samples a subset of content under one set of conditions and contains measurement and sampling variation.

The correct response is not “measurements are unreliable, therefore ignore them.” It is to preserve the uncertainty while still using the signal appropriately.

Evidence for Causation Needs More Than Association

When the claim is causal, the evidence must help answer a counterfactual: what would have happened if the proposed cause were absent or different?

Randomisation is powerful because, when well implemented, it helps make comparison groups similar on both observed and unobserved factors before the intervention. But causal evidence can also come from other designs when randomisation is impossible, provided the assumptions and alternative explanations are made explicit.

A useful causal audit asks about temporal order, plausible mechanism, confounding variables, selection effects, comparison groups, dose or exposure, repeated patterns and whether another explanation predicts the same result.

Evidence Hierarchies Are Useful Only When Matched to the Question

It is tempting to rank evidence with one universal pyramid. But the best design depends on the claim. A randomised trial may be excellent for estimating an intervention’s causal effect. It may be inappropriate for establishing the date of a historical event, proving a mathematical theorem, describing a rare side effect, understanding implementation failure or determining how a person experienced an event.

Instead of asking for the universally “highest” evidence, ask:

• What kind of claim is being made?
• What evidence design could answer that claim?
• What assumptions does that design require?
• Which failure modes remain?

Sample Size, Effect Size and Precision Answer Different Questions

A large sample can estimate a very small effect precisely. A small sample can contain a large apparent effect with wide uncertainty. Statistical significance alone therefore does not tell us whether an effect is important, stable or relevant to the decision.

A fuller reading asks about the estimated magnitude, confidence or credible interval, sample design, missing data, baseline differences, multiplicity of tests and whether the analysis was specified before the result was seen.

For ordinary readers, the practical translation is simple: How large is the effect, how uncertain is that estimate, and would the difference matter in the real world?

Replication, Reproduction and Converging Evidence Solve Different Problems

Reproducibility asks whether the same data and analysis can generate the reported result. Replicability asks whether a new study collecting new data can obtain a consistent result under sufficiently similar conditions. Generalisability asks whether the finding travels to different populations, settings or times.

These should not be collapsed. A result can be reproducible from the original data yet fail to replicate in new samples. It can replicate in one population yet generalise poorly elsewhere.

The National Academies’ work on reproducibility and replicability is useful here because it treats scientific confidence as cumulative and method-dependent rather than as a binary stamp placed on one paper.

Evidence Synthesis Must Preserve Heterogeneity

Systematic reviews and meta-analyses can combine evidence across studies, but a pooled average can hide meaningful variation. Effects may differ by age, baseline capability, implementation quality, subject, dosage, context or outcome measure.

A useful synthesis therefore asks not only “What is the average effect?” but also “How variable are the results, what explains the variation, and which conditions resemble the case we care about?”

Publication and Selection Processes Can Distort the Visible Evidence

The evidence we can see is not always a random sample of all evidence produced. Positive, surprising or clean results may be more likely to be written up, accepted, shared or remembered. Organisations may also report successes more readily than failed trials.

Pre-registration, registered reports, protocol publication, trial registries and transparent reporting are different attempts to reduce the gap between the evidence generated and the evidence eventually visible.

Absence of Evidence Becomes Evidence of Absence Only Under Specific Conditions

Not finding something is informative only when the method had a reasonable chance of detecting it if it were present.

If a highly sensitive test finds no signal in a well-sampled setting, absence carries weight. If nobody looked carefully, the absence tells us little. In education, failing to observe transfer is meaningful evidence against a transfer claim only if the task actually offered a fair opportunity to demonstrate transfer.

No observed signal + strong detection opportunity can be evidence; no observed signal + weak detection opportunity is mostly ignorance.

Population Evidence and Individual Evidence Should Be Combined, Not Confused

Population research provides useful priors: what tends to happen on average and under which conditions. Individual evidence tells us what is happening for this learner, patient, worker or system now.

For education, the strongest practical route is often:

Use research to choose plausible interventions → apply carefully → measure the individual’s response → adapt from the receipt.

This avoids both extremes: ignoring research because “every child is different” and applying population averages as though they determine every child.

A High-Resolution Evidence Audit

1. Claim: What exact proposition is being supported or weakened?
2. Observation: What actually happened in the world?
3. Measurement: How was that observation converted into a record?
4. Uncertainty: What error, sampling or model uncertainty enters through that measurement?
5. Provenance: Can the chain from source to current evidence be reconstructed?
6. Relevance: Why should this information change belief in this claim?
7. Independence: How many genuinely separate routes produced the evidence?
8. Triangulation: Do methods with different failure modes converge?
9. Alternative: What competing explanation predicts the same observation?
10. Causality: If the claim is causal, what approximates the missing counterfactual?
11. Magnitude: How large is the observed effect or difference?
12. Precision: How uncertain is that magnitude?
13. Bias: What sampling, reporting, incentive or publication processes may distort the visible evidence?
14. Replication: Does the finding survive new data or methods?
15. Generalisability: Does it travel across people, settings and time?
16. Synthesis: Does a pooled result hide important heterogeneity?
17. Individual receipt: What does this specific person or system do after the evidence-based action is applied?
18. Disconfirmation: What result would lower confidence in the claim?
19. Claim calibration: Is the final wording no stronger than the evidence permits?

Evidence Boundary: Strong Evidence Is a Property of the Claim–Method Relationship, Not the Prestige of the Source Alone

The US Department of Education’s What Works Clearinghouse shows how an evidence system can make standards explicit rather than treating every study equally. NIST’s measurement uncertainty guidance illustrates why a measured value must carry information about uncertainty. The National Academies’ Reproducibility and Replicability in Science distinguishes several ways confidence can be tested across analysis and new evidence.

These systems differ because they answer different kinds of claims. That is the deeper lesson: evidence quality is not one universal ladder. It is the fit among claim, measurement, design, provenance, alternatives, uncertainty and intended use.

Connect Evidence to the Wider eduKateSG Mechanism Estate

• How Reasoning Works — how evidence becomes premises, likelihood shifts and conclusions.
• How Assessment Works — how learner performances become evidence about capability.
• How Decision-Making Works — how evidence interacts with values, thresholds and consequences.
• How Knowledge Works — why reliable knowledge needs provenance and correction.
• How Science Works — the larger system through which empirical claims remain answerable to the world.