eduKateSG · Judgement · The Gold Standard of Education
Learn what every compatible preference model says—and what only some models permit
Choose the explanation you need, or follow all twelve chapters in order.
Full chapter index · Independent practice · Judgement subject guide
Robust ordinal regression does not force one precise value function from sparse preference statements. It constructs the set of value functions compatible with those statements and declared assumptions. An option is necessarily preferred when it outranks another under every compatible model; it is possibly preferred when at least one compatible model supports the relation.
That distinction turns model ambiguity into a result rather than hiding it. This guide builds the constraint set, checks consistency, works a transparent example, and shows when a representative value function is helpful without confusing it with the robust conclusion.
Continue from The Gold Standard of Judgement within The Gold Standard of Education: learning that transfers into life and the How X Works learning library.
Full chapter index
See the mechanism · Chapters 1–2
Build the worked model · Chapters 3–5
Test what changes · Chapters 6–8
Check evidence and limits · Chapters 9–10
Practise and continue · Chapters 11–12
CHAPTER 1 OF 12 · See the mechanism
1. Start from a decision problem and preference information
List the alternatives, evaluation criteria, directions, scales and decision owner. Then record preference information separately: pairwise comparisons, rankings of reference alternatives, intensity statements or criterion trade-offs. Evaluations describe options; preference statements constrain how evaluations are aggregated.
Use only statements the decision maker can defend. “A is at least as good as B” is an inequality, not a demand for an exact weight. Date the elicitation and note whether uncertainty belongs to performance data, preference information or both.
Keep indifference and incomparability separate. Indifference says the decision maker judges two options equally valuable under the elicitation language; incomparability says the present information does not establish an order. Robust output often contains unresolved pairs. That is a faithful account of what was learned, not a computational defect demanding an arbitrary tie-break.
Evidence check: for Start from a decision problem and preference information, preserve the observation before the interpretation. Record alternatives, criteria, scales and decision owner. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Suppose an additive value model uses non-negative criterion weights that sum to one, with marginal value functions obeying chosen monotonicity rules. Each declared preference becomes a constraint. All weight and function combinations satisfying every constraint form the compatible set.
The set can be large, which is the point. If no compatible model exists, the information or assumptions are inconsistent. If many exist, a single fitted model understates ambiguity. Robust ordinal regression asks relational questions over the whole feasible set.
Evidence check: for Define the compatible-model set, preserve the observation before the interpretation. Record compatible value functions and assumptions. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
For an additive linear teaching model, “reference option R is at least as good as S” becomes U(R) ≥ U(S), where U is the weighted sum of normalised criterion values. A strict preference can include a small discrimination margin. Monotonicity and normalisation provide additional constraints.
Keep the encoding visible. A mistaken criterion direction or scale normalisation can change feasibility. If statements come from several people, do not pool them silently; collective compatibility requires an explicit aggregation or deliberation rule.
Evidence check: for Translate statements into constraints, preserve the observation before the interpretation. Record each elicited statement as a visible constraint. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Consider four fictional options on two benefit criteria. A=(80,50), B=(65,70), C=(50,85), and D=(40,40), with values divided by 100. Three compatible weight vectors illustrate remaining ambiguity: (0.7,0.3), (0.5,0.5), and (0.3,0.7).
Their scores are: model 1 gives A=.71, B=.665, C=.605, D=.40; model 2 gives A=.65, B=.675, C=.675, D=.40; model 3 gives A=.59, B=.685, C=.745, D=.40. A, B and C exchange order. Each, however, exceeds D in all three models—and if the feasible set preserves positive monotone weights, domination makes that conclusion robust.
| Option | Values (g1,g2) | w=(.7,.3) | w=(.5,.5) | w=(.3,.7) |
|---|---|---|---|---|
| A | (.80,.50) | .710 | .650 | .590 |
| B | (.65,.70) | .665 | .675 | .685 |
| C | (.50,.85) | .605 | .675 | .745 |
| D | (.40,.40) | .400 | .400 | .400 |
Evidence check: for Work several compatible models, preserve the observation before the interpretation. Record every option score under each weight vector. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Necessary preference X ≽ᴺ Y means U(X) ≥ U(Y) for every compatible value function. Possible preference X ≽ᴾ Y means the inequality holds for at least one. Necessary relations are conservative; possible relations show what the present information has not ruled out.
In the example, A over B is possible under model 1, while B over A is possible under models 2 and 3. Neither is necessary. A over D is necessary under the stated model class. Do not convert “possible” into “plausible probability”; it is an existence statement, not a frequency.
Evidence check: for Read necessary and possible relations, preserve the observation before the interpretation. Record all-model versus at-least-one-model quantifiers. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
To test whether X is possibly preferred to Y, seek any compatible model with U(X) ≥ U(Y). To test whether X is necessarily preferred, search for a compatible counterexample with U(Y) > U(X). If no counterexample exists under the encoded assumptions, necessity follows.
This formulation supports audit. Preserve the model class, constraints, solver tolerance and witness model for each possible relation or counterexample. A graph of necessary edges can summarise stable conclusions while a graph of possible edges shows unresolved comparisons.
The two tests are not symmetric shortcuts. Finding one witness establishes possibility, but failing to find one may reflect infeasibility, a solver limit or an incorrectly encoded constraint. For necessity, the analyst deliberately searches for a reversal. Record the optimisation status and, when a strict margin is used, test whether the conclusion survives a reasonable change in that margin.
Transitivity also needs care. Necessary preference under a transitive value model is transitive, but a visual summary can omit edges for readability. Possible preference need not provide one globally consistent ranking: A over B may be witnessed by one compatible model and B over C by another. Never splice separate witnesses into a fictional single model.
Evidence check: for Compute relations as optimisation questions, preserve the observation before the interpretation. Record solver witness, counterexample and tolerance. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 7 OF 12 · Test what changes
7. Diagnose inconsistency rather than smoothing it away
An empty compatible set can come from contradictory preference statements, evaluation error, an overly restrictive additive model, or an incorrect direction. Identify a minimal or interpretable conflicting subset and return it to the decision maker.
Before revising judgements, run small mechanical checks: criterion units, benefit versus cost direction, duplicate reference options and exact versus rounded evaluations. Then relax one assumption at a time. If feasibility returns only after permitting criterion interaction, that observation is evidence about model adequacy, not proof that the newly flexible model is true.
Do not quietly drop the least convenient statement. Ask whether a comparison was tentative, context-specific or mistranscribed. If non-additive interaction is the issue, changing the model class is a substantive choice that should be justified, not a numerical repair.
Evidence check: for Diagnose inconsistency rather than smoothing it away, preserve the observation before the interpretation. Record minimal conflicting information and model fit. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
When criterion values are intervals or scenarios, decide whether a relation must hold across every admissible evaluation or merely some. That quantifier is separate from the quantifier over compatible preference models. Mixing them produces overconfident language.
Run scenario-specific relations and report which edges persist. If A necessarily outranks D under every evaluation scenario and every compatible model, the claim is stronger than necessity under a single midpoint table. If the edge breaks only under one measurement range, that directs further evidence gathering.
Evidence check: for Add imprecise evaluations carefully, preserve the observation before the interpretation. Record evaluation intervals and quantifier order. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 9 OF 12 · Check evidence and limits
9. Use a representative value function without replacing robustness
A representative function can support explanation, visualisation or a concrete ranking. Select it by a declared rule—such as centrality or maximal discrimination—after computing robust relations. Then check that its ranking respects the necessary relation.
One representative model cannot show the width of the compatible set. Present it beside necessary and possible relations, plus examples of materially different compatible models. This prevents readers from mistaking a communication aid for the unique preference structure.
A good explanation uses the representative model to walk through one comparison, then immediately shows a compatible countermodel wherever the relation is not necessary. If two representative-selection rules produce different complete rankings but preserve the same necessary edges, report the stable edges first. Decision makers can then distinguish a robust action from a convenient narrative ordering.
Do not optimise the representative function to make a preferred option look dominant. The selection rule belongs in the protocol before results are viewed. If communication requires a simpler model, quantify what robust information it omits and keep the full relation matrices available for review.
| Output | Quantifier | What it supports | What it does not support |
|---|---|---|---|
| Necessary relation | All compatible models | Stable comparison | Probability of correctness |
| Possible relation | At least one model | Unresolved possibility | Likely preference |
| Representative model | One selected model | Explanation and illustration | Unique true ranking |
| Counterexample model | One feasible witness | Why necessity fails | Population frequency |
Evidence check: for Use a representative value function without replacing robustness, preserve the observation before the interpretation. Record representative ranking beside robust relations. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Robust ordinal regression explores all models compatible with ordinal information. Stochastic multicriteria acceptability analysis samples distributions over uncertain evaluations or preferences and reports acceptability frequencies. Those frequencies require probability assumptions; necessary and possible relations require feasibility assumptions.
Neither method removes judgement. Both depend on model class, criteria, data and elicitation. Use robust relations when the question is what sparse statements logically entail; use stochastic analysis when defensible distributions and frequency-style summaries matter. They can complement one another.
Evidence check: for Compare nearby uncertainty methods, preserve the observation before the interpretation. Record method boundary versus stochastic acceptability. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Return to the four-option example and add the statement that the first criterion is at least as important as the second, so w1 ≥ w2. Model 3 is no longer compatible. Recompute every relation rather than deleting one row from a final ranking. B may still exceed A under equal weights, while A exceeds B at 0.7/0.3, so the pair remains unresolved.
Now strengthen the statement to w1 ≥ 0.65 and solve again. A exceeds B throughout that weight interval, creating a necessary relation for this linear model. The new conclusion comes from added preference information, not from better option data.
Evidence check: for Practise a changed-context query, preserve the observation before the interpretation. Record the added weight constraint changed context. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Archive the option table, normalisation, criterion directions, model class, every elicited statement, feasibility result, necessary and possible matrices, counterexample models, solver settings and the wording shown to the decision maker. Frequently asked questions: Is necessary preference the same as statistical significance? No. Does possible mean likely? No. Must the result be a complete ranking? No. Can one representative function replace the set? No.
Continue with ELECTRE credibility, discordance and distillation to compare a different outranking logic. Technical starting points include the paper on selecting a representative value function for robust ordinal regression, the study of robust ordinal regression with imprecise evaluations, and the MaRDI method entry. Cite the exact variant used because assumptions differ across implementations.
Evidence check: for Build the evidence route and continue, preserve the observation before the interpretation. Record the exact necessary-relation claim supported. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.