HOW INTELLIGENCE WORKS · EVIDENCE WEIGHTING · eduKateSG
How Intelligence Decides Which Evidence Should Change Belief, and by How Much
Evidence rarely arrives in equal pieces. Some observations are direct, some noisy, some duplicated, some independent, some highly diagnostic and some barely relevant. Evidence weighting is the intelligence process that decides how much each piece should alter the current map.
Claim → evidence set → relevance → reliability → independence → diagnosticity → contradiction → weighted update → calibrated confidence.
This article belongs to the How Intelligence Works series. The canonical owner of evidence itself remains How Evidence Works. This pillar isolates one narrower intelligence job: integrating unequal evidence without losing provenance, dependence or uncertainty.
The Unequal-Evidence Problem
Ten weak observations do not automatically outweigh one strong measurement. Ten websites repeating one source do not become ten independent confirmations. One dramatic anecdote may be vivid but unrepresentative. One carefully controlled result may be highly informative but narrow.
Intelligence therefore needs more than accumulation. It needs a way to compare evidential force.
Evidence quantity and evidence weight are not the same variable.
1. Relevance Comes Before Weight
Before asking how strong a piece of evidence is, intelligence should ask whether it bears on the claim at all.
A perfectly accurate measurement can be useless if it measures the wrong variable. A student’s handwriting may be directly observable but irrelevant to whether the mathematical model is correct. A survey from another population may be high quality yet only weakly applicable to the current case.
Relevance determines whether the evidence belongs in the update.
2. Reliability Changes Evidential Force
| Evidence feature | Main question |
|---|---|
| Measurement quality | How noisy or biased is the observation process? |
| Source competence | Was the source able to observe or assess this claim well? |
| Source incentives | What pressures could distort reporting? |
| Method transparency | Can the route from observation to claim be inspected? |
| Replicability | Does the result recur under appropriate repetition? |
| Freshness | Is the evidence current enough for the present decision? |
Reliability is not a moral label attached permanently to a source. A highly reliable source in one domain may be weak in another. Weight belongs to the source-plus-claim relationship.
3. Independence Prevents Duplicate Evidence From Becoming Artificially Heavy
Several pieces of evidence are stronger when they arise through genuinely different routes. If five articles all copy one report, they provide one evidential lineage, not five independent confirmations.
Intelligent weighting therefore tracks dependence: shared datasets, shared instruments, shared assumptions, shared witnesses or shared source chains.
Repetition increases confidence only to the extent that the repetitions bring new information.
4. Diagnosticity Matters More Than Drama
Some observations are powerful because they discriminate among competing explanations. Others are compatible with almost everything.
If two hypotheses both predict a common symptom, observing that symptom adds little separation. If only one hypothesis predicts a particular finding, that finding carries greater diagnostic weight.
This links evidence weighting to discrimination and model selection: strong evidence is often evidence that changes which map remains plausible.
5. Contradictory Evidence Should Not Be Averaged Away Automatically
When evidence conflicts, the intelligence task is not simply to count votes. Conflict may reveal measurement error, population differences, temporal change, hidden subgroups or a model whose scope is too broad.
Strong weighting preserves the contradiction long enough to ask why the sources disagree.
Synthesis should combine maps where justified without smoothing away conflicts that still contain information.
6. Evidence Weighting in Mathematics
Formal proof has a distinctive evidential structure. A theorem does not become proven because many examples support it. One valid proof can establish a general result under stated assumptions, while one valid counterexample can disprove a universal claim.
Examples still matter for discovery, intuition and testing candidate rules. Their evidential job is different from proof.
This distinction teaches a general intelligence lesson: evidential force depends on the logical relationship between the evidence and the claim, not merely on the number of observations.
7. Evidence Weighting in Science
Scientific inference combines measurement quality, design, sample, controls, replication, prior work and alternative explanations.
A result from a well-controlled experiment can carry strong causal information within its conditions. An observational study may represent the wider world better while leaving more confounding. Replication across methods can strengthen confidence by reducing dependence on one design.
The job is not to declare one evidence type universally superior. It is to match evidential weight to the claim being asked.
8. Evidence Weighting in Everyday Reasoning
Everyday life contains mixed evidence: memory, testimony, documents, patterns, direct observation and consequences.
A useful discipline is to ask what would have to be true for this evidence to be misleading. Was the observer in a position to see? Could memory be reconstructed incorrectly? Is the source repeating someone else? Does the anecdote represent the wider case?
This does not require permanent scepticism. It requires proportionate updating.
9. Evidence-Weighting Failure Atlas
| Failure | What happens | Repair |
|---|---|---|
| Vote counting | Source quantity substitutes for evidential force | Weight relevance and independence |
| Authority capture | Status replaces claim-specific evidence | Inspect competence and method |
| Drama bias | Vivid evidence dominates representative evidence | Restore base rates and sampling |
| Duplication inflation | Copied sources appear independent | Trace provenance |
| Contradiction erasure | Conflict is averaged away | Model why evidence differs |
| Precision illusion | Exact-looking numbers receive excessive trust | Inspect measurement uncertainty |
| Confirmation weighting | Supporting evidence is weighted more heavily than disconfirming evidence | Use symmetric criteria |
10. Evidence Weighting and Calibration Are Different
Evidence weighting asks how much a particular observation should alter the map. Calibration asks whether the system’s resulting confidence matches its actual accuracy over time.
The companion article How Intelligence Works | Calibration owns confidence-performance fit.
Poor weighting often produces poor calibration, but the two are not identical.
11. Teams Need an Explicit Evidence Table
Teams can disagree because they weight the same evidence differently without realising it. One member trusts direct field observation; another trusts historical data; another prioritises a controlled test.
Making evidence classes explicit improves collective reasoning: source, date, method, relevance, reliability, dependence and what claim each item actually bears on.
Disagreement becomes more productive when the team can see whether it disagrees about evidence, weight, model or value.
12. Institutions Encode Evidence Weights Into Standards
Courts, laboratories, regulators, schools and professional bodies often formalise which evidence classes count for which decisions.
These standards create consistency and protect against arbitrary judgement. They can also become stale or too rigid when new methods emerge.
Institutional intelligence therefore preserves both evidential standards and a route for reviewing those standards when the world changes.
13. Artificial Intelligence and Evidence Weighting
AI systems can retrieve, summarise and compare evidence, but fluent synthesis can hide unequal source quality and dependence.
Reliable AI-supported reasoning should preserve source identity, dates, direct versus derivative evidence, contradictory findings and confidence boundaries. A model should not turn ten copied webpages into the appearance of ten independent confirmations.
The strongest system separates retrieval from generation and makes the evidential path inspectable enough that a human can challenge the weighting.
14. The Evidence Weighting Audit
- Claim: What exact proposition is being updated?
- Relevance: Does this evidence bear on that claim?
- Reliability: How trustworthy is the observation process?
- Independence: Is this genuinely new information?
- Diagnosticity: Does it distinguish among competing explanations?
- Scope: Under which population, time and conditions does it apply?
- Contradiction: Which credible evidence points the other way?
- Provenance: Can the evidence be traced to its origin?
- Update: How much should confidence change?
- Calibration: Does the resulting confidence survive later outcomes?
15. CivDJ Reading: Not Every Channel Enters at the Same Gain
In the CivDJ frame, evidence arrives through different Masters, tools and records. Evidence weighting controls the gain rather than flattening every channel to the same volume.
Provenance, authority, freshness, independence and diagnosticity determine how strongly a channel may alter the final mix. Contradictory channels remain audible until the Tumbler can explain the difference.
A faithful mix does not silence weak evidence; it keeps weak evidence weak and strong evidence traceable.
16. Return to the Scale
Evidence does not arrive already labelled with how much belief it deserves.
Intelligence must ask what the evidence measures, where it came from, whether it is independent, what alternatives it discriminates and what uncertainty remains.
The goal is not to turn judgement into one mechanical score. It is to make the reasons for changing the map visible enough to inspect, challenge and calibrate.