Series: How to Prepare For — Global Examination Performance Edge Articles
Advanced Article P028
The evidence does not line up neatly.
One source says the policy worked.
Another shows a weak outcome.
One experiment supports the hypothesis.
Another produces an anomaly.
One graph rises.
Another measure falls.
One witness remembers the event one way.
Another remembers it differently.
One calculation suggests a route.
A boundary condition makes that route look incomplete.
This is not necessarily a broken question.
It may be the question.
Advanced examinations often stop rewarding the learner who can find one supporting fact and begin rewarding the learner who can reason when evidence disagrees.
This article owns that edge condition.
It follows How to Prepare for an Exam When the Question Contains More Information Than You Need | Separate Signal From Distractor Data. That article asks which information matters. This one begins after several pieces matter and point in different directions.
It routes around the broader evidence, source-analysis, critical-thinking and “what evidence would change your answer” owners by claiming a narrower examination-performance job: constructing a defensible answer when relevant evidence conflicts.
The 50-second answer
When an exam question contains conflicting evidence:
- State exactly what conflicts instead of saying “the evidence is mixed.”
- Check whether the sources measure the same thing.
- Check whether they refer to the same population, time, place or condition.
- Separate observation from interpretation.
- Compare evidence quality, not just evidence count.
- Ask whether one result is an anomaly, boundary case or measurement problem.
- Generate at least two explanations that could account for the disagreement.
- Look for a condition under which both findings could be true.
- Use chronology and mechanism to test causal claims.
- State uncertainty at the level the evidence justifies.
- Build a judgement that explains the strongest counterevidence rather than ignoring it.
- Under time, prioritise the contradiction that most changes the conclusion.
The central rule is:
Conflicting evidence is not something to hide from your answer. It is information about where your answer needs finer conditions.
Alicia, Tricia and Kai Kai see the contradiction
Alicia chooses the source that supports the answer she first preferred.
Tricia lists both sources and writes, “Therefore the evidence is inconclusive.”
Kai Kai asks:
Do these sources actually make incompatible claims?
One measures short-term output.
The other measures long-term retention.
The contradiction shrinks.
Both can be true:
the intervention can improve immediate performance while producing little durable retention.
The better answer is not “Source A wins.”
It is a conditional model explaining why the results differ.
What this article owns
- Conflict parsing: stating precisely which claims disagree.
- Comparability: checking whether evidence concerns the same construct and conditions.
- Quality weighting: comparing reliability, relevance and directness.
- Rival explanations: generating mechanisms that could explain disagreement.
- Reconciliation: finding conditions under which apparently conflicting evidence can coexist.
- Counterevidence handling: integrating rather than hiding inconvenient facts.
- Calibrated judgement: matching conclusion strength to evidence strength.
- Timed synthesis: producing this reasoning within an examination.
First principle: define the contradiction
Do not write:
“The sources disagree.”
Write:
Source A reports X under condition C, while Source B reports not-X or a weaker X under condition D.
The more precisely the conflict is stated, the easier it becomes to explain.
Second principle: apparent conflict may be a construct mismatch
Two sources may use the same everyday word while measuring different things.
“Success” may mean score, retention, participation, profit, survival, approval or speed.
Before comparing results, compare what was measured.
Third principle: compare population
Evidence from beginners may not directly describe experts.
Evidence from children may not directly describe adults.
Evidence from one industry, ecosystem or historical population may not transfer automatically to another.
Population differences can reconcile findings.
Fourth principle: compare time scale
Immediate effects can differ from delayed effects.
Short-run costs can coexist with long-run benefits.
Early improvement can disappear.
Delayed effects can emerge after the measurement window.
Put evidence on a timeline.
Fifth principle: compare context
A mechanism may work under one set of conditions and fail under another.
Temperature.
Resources.
Institutional rules.
Prior knowledge.
Market conditions.
Source purpose.
Context is not an excuse to avoid judgement. It is a variable to test.
Sixth principle: separate observation from explanation
Two sources can agree on what happened and disagree on why.
That is a different conflict from disagreeing about the observation itself.
Mark:
Observation.
Interpretation.
Causal claim.
Then compare like with like.
Seventh principle: count is not weight
Three weak pieces of evidence do not automatically outweigh one strong piece.
Compare:
- relevance;
- measurement quality;
- sample or coverage;
- directness;
- independence;
- consistency with mechanism.
Evidence synthesis is weighted, not democratic.
Eighth principle: independence matters
Five sources repeating the same original claim are not five independent confirmations.
Ask whether the evidence streams share a source, dataset, method or assumption.
Ninth principle: direct evidence and proxy evidence differ
A proxy can be useful.
But if one source measures the target directly and another measures a correlated indicator, their evidential roles differ.
Do not treat them as interchangeable.
Tenth principle: measurement error can create conflict
Ask whether instruments, coding, sampling, translation, marking or classification could explain the difference.
Do not invoke measurement error merely to dismiss inconvenient evidence.
Use it when the question provides a reason.
Eleventh principle: anomalies deserve investigation, not automatic deletion
An outlier may be noise.
It may also reveal a hidden condition.
Ask:
What would have to be true for this anomaly to be informative rather than accidental?
Twelfth principle: contradictions can expose boundary conditions
A rule that seemed universal may work only above a threshold, within a range or under a particular assumption.
Conflicting evidence can therefore improve the model.
The boundary question
Under what conditions would Source A be expected, and under what conditions would Source B be expected?
Thirteenth principle: search for moderators
A moderator changes the strength or direction of a relationship.
Examples:
- prior knowledge;
- age;
- dose;
- time;
- environment;
- implementation quality;
- baseline risk.
A hidden moderator can reconcile conflicting results.
Fourteenth principle: search for mediators
A mediator explains the pathway through which an effect occurs.
If one source measures the first stage and another measures the final outcome, the missing middle may explain the apparent contradiction.
Fifteenth principle: chronology constrains causality
A proposed cause must precede the effect it is claimed to produce.
When sources disagree about causation, map the sequence.
Temporal impossibility can eliminate one explanation.
Sixteenth principle: correlation evidence should not silently become causal evidence
One dataset may show association.
Another experiment may test causality.
They are not directly contradictory if they answer different questions.
Identify the inference each design permits.
Seventeenth principle: mechanism can arbitrate between findings
If two outcomes differ, ask which proposed mechanism can generate both under their respective conditions.
A model that explains both is often stronger than one that explains only the preferred result.
Eighteenth principle: generate rival explanations
Do not stop with the first reconciliation.
Generate at least two plausible explanations for important conflicts.
Then ask what evidence would distinguish them.
The rival-explanation table
| Explanation | Predicts A? | Predicts B? | Extra evidence needed |
|---|---|---|---|
| Different populations | Yes | Yes | Matched subgroup data |
| Measurement error | Possibly | Possibly | Reliability check |
| Time-scale effect | Yes | Yes | Repeated measurement |
Nineteenth principle: ask what evidence would change the judgement
Before committing, identify the evidence that would make the rival explanation stronger.
This protects against motivated reasoning and connects to the existing How to Think Properly | Decide What Evidence Would Make You Change Your Answer owner.
Twentieth principle: counterevidence should change something
If you mention counterevidence but your conclusion remains exactly as strong as before, ask whether you actually integrated it.
Counterevidence may change:
- confidence;
- scope;
- conditions;
- causal claim;
- recommended action.
Twenty-first principle: concession is not surrender
A strong answer can say:
Evidence B weakens the universal version of the claim, but Evidence A remains stronger for condition C.
This is more precise than either ignoring B or abandoning all judgement.
Twenty-second principle: “inconclusive” is sometimes correct—but must be earned
Do not use “mixed evidence” as an escape from analysis.
Explain why the evidence cannot discriminate sufficiently.
What remains unknown?
What additional evidence would resolve it?
Twenty-third principle: calibrated language matters
Use language that matches the evidence:
demonstrates.
strongly supports.
suggests.
is consistent with.
does not rule out.
is insufficient to establish.
Precision in confidence is part of reasoning.
Twenty-fourth principle: do not average incompatible quantities
Students sometimes try to reconcile disagreement by averaging numbers that measure different constructs, populations or scales.
Before combining evidence, establish comparability.
Twenty-fifth principle: aggregation can hide subgroup conflict
An overall effect can conceal opposite effects in subgroups.
If the question provides subgroup data, inspect it before relying on the aggregate.
Twenty-sixth principle: subgroup evidence can also mislead
Do not fragment the data until a tiny subgroup appears to support the preferred answer.
Use subgroup analysis only when the distinction is relevant and sufficiently supported.
Twenty-seventh principle: base rates matter
A striking individual case may conflict with the general pattern without overturning it.
Ask whether the question is about the individual case or the broader probability.
Twenty-eighth principle: case evidence and population evidence have different jobs
A case can reveal possibility and mechanism.
A larger dataset can estimate frequency or average effect.
Do not ask one type of evidence to do the other’s job.
Twenty-ninth principle: source purpose can explain emphasis
Historical, media and policy sources may emphasise different facts because of audience, purpose and position.
That does not automatically make one false.
Use provenance to interpret the disagreement.
Thirtieth principle: incentives can affect evidence—but require evidence themselves
A source may have reasons to frame events selectively.
Do not infer distortion solely from dislike of the source.
Connect purpose or incentive to specific omissions, emphases or contradictions.
Thirty-first principle: contradiction can be generated by definitions
Two studies may define “failure,” “poverty,” “mastery,” “recovery” or “success” differently.
Compare operational definitions before comparing results.
Thirty-second principle: thresholds can create apparent disagreement
One source may classify 49 as failure and another may treat it as near-mastery.
The underlying measurement can be similar while the category differs.
Inspect thresholds.
Thirty-third principle: model assumptions can create different answers
Two mathematical, economic or scientific models can produce different predictions because they assume different boundary conditions or simplifications.
State the assumptions before deciding which model fits the question.
Thirty-fourth principle: sensitivity analysis is a reasoning tool
Ask whether changing a plausible assumption changes the conclusion.
If the answer flips easily, the judgement is fragile.
If it remains stable across reasonable assumptions, confidence increases.
Thirty-fifth principle: robustness is stronger than one-path support
A conclusion supported by several independent evidence types can be more robust than one supported by repeated versions of the same measurement.
Look for convergence across methods where the task permits it.
Thirty-sixth principle: triangulation does not mean counting agreement
Different methods have different biases.
The value of triangulation is that independent weaknesses may not align.
Explain why the evidence streams complement one another.
Thirty-seventh principle: contradictory evidence can improve an essay
Do not hide the difficult source in the final paragraph.
Use it to sharpen the thesis.
A strong thesis often becomes conditional:
X is the stronger explanation under conditions A and B, although C limits how far the claim can be generalised.
Thirty-eighth principle: contradictory evidence can improve a scientific explanation
An anomalous result may reveal uncontrolled variables, measurement limits, threshold effects or an incomplete mechanism.
Use it to propose the next test.