Two credible studies disagree.
Which one is wrong?
Sometimes neither.
Evidence conflict is a state in which credible evidential routes support materially different conclusions, requiring diagnosis of claim, measurement, population, method, time, dependency and context before confidence can be reconciled.
This is the third pillar beneath How Evidence Works. The master owns alternatives and uncertainty broadly. This page owns the conflict-resolution problem: what should we inspect when good evidence does not line up?
Quick Read
Conflicting evidence should not be resolved by counting papers or choosing the preferred authority. First check whether the items answer the same claim. Then compare populations, measurement definitions, intervention dose, implementation quality, timing, analysis and source dependence. Apparent contradiction often becomes conditional truth: an effect occurs for some learners, at some dose, under some conditions, but not universally. Genuine unresolved conflict should remain visible rather than being compressed into false certainty. The goal is to discover which variable separates the routes and design the next observation to discriminate between them.
Start by Aligning the Claim
Study A asks whether a programme improves immediate test scores.
Study B asks whether it improves one-year retention.
Different results are not necessarily contradictory because the claims differ.
Measurement Differences Can Create Conflict
One study measures self-report.
Another measures observed behaviour.
Both may be measuring legitimate but different constructs.
Population Differences Can Reverse an Effect
An intervention helps novices but not advanced learners.
Averaging across different populations can hide this conditional structure.
The fourth sibling, Transportability Boundary, owns how evidence travels across populations and settings.
Timing Can Explain Disagreement
Short-term gain and long-term fade are compatible.
So are early costs followed by later benefits.
Always compare observation windows.
Implementation Can Differ Even Under the Same Label
Two schools both report using “the same programme.”
One delivers 30 minutes daily with trained staff.
The other delivers 15 minutes irregularly.
The label is shared; the exposure is not.
Dependency Can Make Agreement Look Stronger Than Conflict
Five supporting studies share one dataset.
One conflicting study uses a genuinely independent route.
The first sibling, Evidence Dependency, owns the shared-upstream problem.
Conflict Can Reveal Claim Inflation
If evidence supports a result only under certain conditions, the original universal claim may have been too broad.
The second sibling, Claim–Evidence Distance, owns how far the conclusion travels beyond observation.
Conflicts Should Be Decomposed, Not Averaged Away Automatically
A pooled average can be useful.
But if half the contexts show strong benefit and half show no effect, the average may describe nobody particularly well.
Look for heterogeneity and effect modifiers.
One Strong Disconfirmation Can Be Highly Diagnostic
If a theory predicts an effect in every case, one credible counterexample may force the claim to shrink or acquire a boundary condition.
Do Not Resolve Conflict by Prestige Alone
Authority can help assess method and track record.
It does not eliminate the need to inspect why the evidential routes differ.
The Next Experiment Should Target the Difference
If studies differ mainly by age group, test age directly.
If they differ mainly by dose, vary dose.
If they differ mainly by measurement, collect both measurements in the same sample.
Conflict is useful when it tells us what distinction the next observation should isolate.
A Practical Evidence Conflict Matrix
CLAIM: Method improves mathematics performance ROUTE A: Primary students, 8 weeks, standardised test, +6 points ROUTE B: Secondary students, 4 weeks, classroom quiz, no effect CHECK: - same population? NO - same duration? NO - same outcome? NO - same implementation? UNKNOWN - independent samples? YES NEXT QUESTION: Does effect depend on age, duration or measurement?
A 20-Lens Evidence Conflict Audit
- Are the claims identical?
- Are outcomes defined the same way?
- Are populations comparable?
- Are settings comparable?
- Are time horizons comparable?
- Is exposure/dose comparable?
- Was implementation faithful?
- Are measurements comparable?
- Are samples independent?
- Do routes share upstream evidence?
- Are analyses answering the same estimand?
- Could selection explain the difference?
- Could publication bias distort the visible pattern?
- Is one claim broader than the evidence?
- Is heterogeneity being averaged away?
- What variable best separates the results?
- What new observation would discriminate?
- Should confidence split by context?
- What remains genuinely unresolved?
- Does the final synthesis preserve disagreement where the evidence has not yet converged?
Final Thought: Disagreement Is Often a Map of the Missing Condition
When credible evidence conflicts, the mature response is not to choose a favourite result. It is to find the condition under which each result makes sense—and let the next test target that boundary.
EVIDENCE · FOUR PILLAR LEGS
Return to How Evidence Works, or continue through Evidence Dependency, Claim–Evidence Distance and Transportability Boundary. Return to the How X Works Hub.