HOW INTELLIGENCE WORKS · MODEL DISCRIMINATION · eduKateSG
How Intelligence Chooses the Observation That Forces Competing Models to Make Different Predictions
Model discrimination is the intelligence process that designs or selects evidence specifically because rival models disagree about what should happen. When several explanations already fit the known facts, the next useful observation is not merely more evidence. It is evidence that separates the candidates.
Competing models → find where their predictions diverge → choose the most separating observation or intervention → collect the result → eliminate, weaken or revise models → repeat until the useful distinction is resolved.
This article belongs to the How Intelligence Works series. Hypothesis Testing owns testing candidate explanations generally. Diagnosticity Reasoning owns judging how much already-observed evidence favours one explanation over another. Identifiability Reasoning owns whether the target can be uniquely recovered at all. Model discrimination owns the next control job: which observation should we choose because the models predict different outcomes there?
More Evidence Is Not Always More Informative
If two models make nearly identical predictions in the region already observed, collecting more data from that same region may only strengthen both models together. Model discrimination looks for the point where their predictions pull apart.
The purpose is not to collect the largest dataset. It is to collect the evidence with the greatest power to separate the models.
A decisive test is valuable because the rivals cannot all survive the same result.
1. Rival Models Must Be Made Explicit
Discrimination begins with at least two candidate models stated clearly enough to produce different predictions somewhere. Vague verbal explanations are difficult to discriminate because they can often be repaired after any result.
A useful model says what should happen under specified conditions.
2. The Best Test Targets Prediction Divergence
Suppose two theories fit all existing observations. One predicts a sharp change when condition X is introduced; the other predicts no change. Condition X has high discriminatory value.
The strongest test is often where predicted outcomes are farthest apart relative to expected noise.
3. Model Discrimination Is Not Confirmation Hunting
A weak test asks where the favoured model looks impressive. A strong test asks where the leading alternatives disagree.
This shifts experimentation from proving one story to exposing the contrast among stories.
4. Discrimination Can Be Observational or Interventional
Sometimes naturally occurring cases already occupy the separating region. In other settings, the investigator must manipulate a variable, alter a stimulus or create a boundary case.
The job is not necessarily to intervene. The job is to reach evidence on which the models cannot all say the same thing.
5. Model Discrimination in Mathematics
Two candidate functions can fit the same known points and diverge outside the observed interval. Evaluating a carefully chosen new point can distinguish them more efficiently than adding many points where they overlap.
The principle generalises: choose the constraint that removes the largest part of the remaining solution space.
6. Model Discrimination in Learning Diagnosis
A student repeatedly answers percentage problems incorrectly. One model says the learner does not understand percentage as a multiplicative relationship. Another says the concept is sound but base-value selection is unstable.
A good diagnostic item holds the percentage simple while changing which quantity is the base. The item is chosen because the two diagnoses predict different errors.
7. Boundary Cases Are Often Highly Discriminating
Models that behave similarly in routine cases may separate at extremes, transitions, reversals or unusual combinations.
Counterexamples, edge cases and perturbations are useful when they expose hidden structural differences rather than merely create difficulty.
8. Sequential Discrimination Builds a Decision Tree
One test may not identify the final model. Its result can remove several candidates and determine which test should follow.
The process becomes adaptive: each observation changes the best next question.
9. Model-Discrimination Failure Atlas
| Failure | What happens | Repair |
|---|---|---|
| Confirmation targeting | Tests are chosen where the preferred model already looks strong | Choose points of maximal rival disagreement |
| Overlap sampling | More data are collected where models predict the same outcome | Move to a separating region |
| Vague-model immunity | A theory can explain every possible result after the fact | Demand pre-specified predictions |
| Noise blindness | Models differ less than measurement noise | Increase precision or choose a stronger contrast |
| Single-shot obsession | One imperfect test is expected to settle everything | Use sequential discrimination |
| Cost blindness | The most separating test is too expensive or risky | Balance information gain with cost and reversibility |
| Post-result rewriting | Predictions are changed after seeing data | Archive predictions before testing |
10. Model Discrimination and Diagnosticity Are Different
Diagnosticity evaluates evidence already in hand: which explanation does this observation favour? Model discrimination acts earlier by choosing evidence because the alternatives would respond differently.
One interprets evidence. The other designs the evidence path.
11. Teams Should Ask What Result Would Change the Debate
When meetings repeat the same arguments, the useful question is often: what observation would make one side update more than the other?
A disagreement becomes scientifically productive when it can be converted into rival predictions.
12. Institutions Need Discriminating Pilots, Not Decorative Pilots
Pilots sometimes demonstrate that a chosen programme can operate. A discriminating pilot instead compares plausible mechanisms, implementation choices or policy theories under conditions where their expected outcomes differ.
The pilot then reduces uncertainty about what should scale.
13. Artificial Intelligence and Active Model Discrimination
AI systems can generate many plausible explanations. Their next capability is not simply ranking them by verbal plausibility but proposing tests that separate them.
A stronger agent identifies prediction disagreement, estimates test cost and risk, uses tools or experiments where authorised, and updates the candidate set after the result.
14. The Model Discrimination Audit
- Rivals: What models remain plausible?
- Predictions: What does each model predict under the same condition?
- Divergence: Where are those predictions most different?
- Noise: Is the difference large enough to observe?
- Cost: What will the test consume in time, money or risk?
- Reversibility: Can the test be bounded safely?
- Pre-registration: Are predictions recorded before the result?
- Outcome: Which models would each possible result weaken?
- Sequence: What test becomes best after this result?
- Stop rule: When is the remaining distinction good enough for action?
15. CivDJ Reading: Solo the Channel Where the Masters Disagree
In the CivDJ frame, several Masters may produce nearly identical room mixes under ordinary conditions. To discriminate them, solo or perturb the channel where their internal routings imply different returns.
Do not compare Masters where they sound the same. Move to the bar where their arrangements diverge.
16. Return to the Observation That Splits the Models
Model discrimination turns disagreement into experimental geometry.
It asks where rival models stop making the same prediction and then routes attention toward that point.
The mature mind does not merely ask for more evidence. It asks for the evidence that changes which model remains possible.