THE GOLD STANDARD OF JUDGEMENT · SPECIALIST STUDY
Aggregate interacting criteria without pretending every contribution is independent
Weighted averages add criterion contributions independently. That can mislead when evidence sources overlap, capabilities reinforce one another or two weaknesses create a bottleneck. A Choquet integral replaces ordinary additive weights with a capacity defined on sets, allowing the value of a coalition to differ from the sum of its parts. The flexibility is useful, but it increases elicitation, interpretation and overfitting risks. Good judgement makes the capacity, scale, decision attitude and robustness tests inspectable.
This guide belongs to the Judgement subject guide, the Gold Standard of Education, and the How X Works learning library. Choose a reading route, work the examples, then use the transfer checks to test whether the method survives a changed context.
Choose a reading route
Build the foundations · chapters 1–2
Apply the core method · chapters 3–5
Test evidence and alternatives · chapters 6–7
Handle limits and disagreement · chapters 8–9
Transfer and govern · chapters 10
The whole problem at a glance
| Object | Meaning | Check |
|---|---|---|
| Capacity | Set function assigning value to coalitions | Monotone, normalized and decision-grounded |
| Choquet integral | Ordered aggregation using capacity increments | Scale, ordering and ties handled correctly |
| Interaction | Coalition value exceeds or falls below additive expectation | Interpret complementarity or redundancy |
| Decision attitude | Non-additivity reflects ambiguity or preference structure | Do not confuse belief with value |
A capacity may represent criterion importance, interaction, ambiguous belief or another set function. Its interpretation must be declared; the same mathematics does not make these meanings interchangeable.
CHAPTER 1 OF 10 · Build the foundations
1. Recognise when additivity is the wrong assumption
Core idea. An additive score assumes that each criterion contributes independently and that its trade-off is stable across the levels of other criteria. When two sources duplicate evidence, adding both double-counts. When two capabilities are jointly necessary, adding them separately understates their coalition.
Method. Draw a criterion-dependence map before choosing a model. Ask whether the value of a pair equals, exceeds or falls below the sum of separate contributions. Compare a weighted sum with bottleneck and interaction models using real decision cases.
Worked example. A programme is judged on access and ongoing support. Either alone has limited effect: accessible enrolment without support leads to attrition, while support inaccessible to the target group reaches few people. Their combination is complementary and may deserve more coalition value than an additive score gives.
Changed-context transfer. Move to two intelligence sources reporting from one upstream witness. The pair is redundant rather than complementary, so their coalition value should be less than the sum of separate nominal weights. Same non-additivity, different interpretation.
Limits. Observed interaction can come from poor scaling, omitted criteria or data leakage rather than genuine coalition value. Diagnose these alternatives before adopting a flexible capacity.
Misconception to avoid. Whenever criteria correlate, a Choquet integral is required. Correlation is a data property; interaction in a preference or belief model is a decision interpretation. Related signals may still warrant an additive rule after proper preprocessing.
Practical evidence check
- Take one current scorecard and identify a pair whose joint value is plausibly super-additive and a pair plausibly sub-additive; explain evidence for each.
- Return to chapter contents
CHAPTER 2 OF 10 · Build the foundations
2. Define a capacity on sets of criteria
Core idea. A normalized capacity assigns zero to the empty set, one to the full set and nondecreasing values to nested coalitions. Unlike an additive probability or weight system, the value of a union need not equal the sum of disjoint parts.
Method. List every coalition for a small criterion set, then elicit or estimate its value under one declared interpretation. Check monotonicity mechanically: adding a criterion cannot reduce capacity unless the chosen model explicitly leaves the capacity framework. Record uncertainty rather than forcing one precise table.
Worked example. With three criteria there are eight subsets. The decision team values access plus support at 0.75 even though their singleton values sum to 0.55, expressing complementarity. It values two overlapping achievement measures below their singleton sum, expressing redundancy.
Changed-context transfer. For ambiguous events, the set elements are states rather than criteria and the capacity represents non-additive belief. Keep that use separate from multi-criteria importance because stakeholders answer different elicitation questions.
Limits. The number of coalitions grows exponentially. Full capacities become impractical quickly, and arbitrary values can hide a preferred ranking. Restricted forms, interaction orders or robust sets may be more defensible.
Misconception to avoid. A capacity value is the probability that a coalition occurs. In multi-criteria aggregation it can encode importance or interaction, and in ambiguity models it may encode non-additive belief. Label the interpretation.
Practical evidence check
- Check normalization and every subset monotonicity relation in a proposed three-criterion capacity, then ask whether each non-additive value has a written rationale.
- Return to chapter contents
CHAPTER 3 OF 10 · Apply the core method
3. Compute the discrete Choquet integral by ordering scores
Core idea. For nonnegative criterion scores, the discrete Choquet integral sorts them and accumulates score increments multiplied by capacities of the sets meeting each threshold. This makes the contribution of a criterion depend on which higher-scoring criteria accompany it.
Method. Put criteria on a justified common scale, sort from low to high, form nested upper sets and multiply each score increase by the capacity of its upper set. State the convention for ties and negative values. Recompute with a reference implementation.
Worked example. Scores are 20 for support, 80 for access and 80 for achievement. The first twenty points apply to the full coalition; the next sixty apply to the set of access and achievement. Capacity values determine how much that strong pair contributes after the common floor.
Changed-context transfer. If criteria use euros, test scores and ordinal labels, direct ordering and subtraction are not meaningful. First build value functions onto a common decision scale or choose a method designed for heterogeneous scales.
Limits. Small scaling changes can alter order and therefore which coalition capacities apply. The model is not invariant to arbitrary nonlinear transformations, so scale construction is part of the judgement.
Misconception to avoid. The Choquet integral is just a weighted average with changing weights. It can be written through marginal increments, but the underlying set function imposes structured coalition values and must satisfy capacity properties.
Practical evidence check
- Compute one three-criterion case by hand, then permute a tied pair and verify that the implementation gives the same result under the declared convention.
- Return to chapter contents
CHAPTER 4 OF 10 · Apply the core method
4. Interpret complementarity, redundancy and importance
Core idea. Interaction indices summarize whether criteria work better together or overlap, while importance indices summarize average marginal contribution across coalitions. They aid explanation but do not replace the full capacity or causal evidence.
Method. Calculate Shapley-style importance and pairwise interaction where appropriate, then compare them with the elicited stories and observed decisions. Use plain language: complementary, redundant or near-additive. Inspect higher-order interactions when pair summaries fail.
Worked example. Two safety sensors share a failure mode, so their pair capacity is below their summed singleton importance. A mechanical inspection and an independent load test cover different mechanisms, so their coalition may be complementary. The explanation follows evidence pathways rather than labels alone.
Changed-context transfer. In policy appraisal, equity and accessibility may reinforce each other because nominal benefits cannot reach excluded groups. The interaction is normative as well as empirical, so affected stakeholders should help define it.
Limits. Indices average across coalitions and can conceal local reversals. Large interaction estimates from sparse data may be unstable. Present uncertainty and example profiles.
Misconception to avoid. A positive interaction index proves causal synergy. It describes the fitted or elicited capacity. Causal claims need design and evidence outside the aggregation formula.
Practical evidence check
- Choose one positive and one negative interaction, trace each to evidence or policy rationale and show a profile where the interaction changes the ranking.
- Return to chapter contents
CHAPTER 5 OF 10 · Apply the core method
5. Elicit capacities without tuning the winner
Core idea. Capacity elicitation can use direct judgements, rankings, trade-offs, observed choices or optimization. The danger is circularity: adjusting parameters until a preferred option wins, then presenting the result as neutral mathematics.
Method. Separate elicitation cases from evaluation cases, use structured questions about coalitions, preserve disagreement, constrain monotonicity and validate on held-out or changed-context profiles. Compare with simpler models and publish the feasible parameter range.
Worked example. A committee rates coalitions before seeing the final applicant list. Later, the capacity ranks applicants and reveals one surprising result. The team investigates scale and rationale rather than quietly modifying interaction values after learning who wins.
Changed-context transfer. In machine learning, fit a low-order capacity with cross-validation and regularization. In public allocation, empirical fit alone is insufficient; legitimacy, reasons and affected-party participation also matter.
Limits. People struggle to judge many abstract coalitions. Anchoring, fatigue and strategic answers can dominate. Use few meaningful parameters, concrete profiles and uncertainty intervals.
Misconception to avoid. More elicitation questions always improve the capacity. Beyond a point they add noise and opportunity for inconsistency. Ask only questions linked to consequential distinctions.
Practical evidence check
- Hide option names, elicit on representative profiles, then reveal the options and document every post-reveal change with an independent reason.
- Return to chapter contents
Worked three-criterion scholarship example
| Profile | Scores | Interpretation |
|---|---|---|
| A | Academic 80, access 80, support 20 | Strong record but weak delivery support |
| B | Academic 70, access 65, support 65 | Balanced and implementable |
| Capacity feature | Access and support are complementary | Their coalition gains extra weight |
| Choquet result | B can outrank A | Interaction changes aggregation |
| Sensitivity | Reduce coalition premium | Check whether ranking survives |
The numbers are illustrative. Criteria need a common defensible scale, and a capacity should come from policy, evidence or elicitation rather than being tuned to obtain a preferred winner.
CHAPTER 6 OF 10 · Test evidence and alternatives
6. Distinguish criterion aggregation from ambiguity attitude
Core idea. Choquet expected utility can also integrate utility with respect to a non-additive belief capacity. This models departures from additive subjective probability, including attitudes to ambiguity under stated axioms. That use is conceptually different from aggregating interacting criteria.
Method. Name the domain of the capacity: criteria, states or events. If states, define acts, outcomes and utility, and explain how the capacity represents belief or decision attitude. Do not reuse a multi-criteria table as if it were an event belief model.
Worked example. A chooser values an act under a capacity that gives less total support to disjoint uncertain events than additive probability would. The resulting Choquet expectation reflects ambiguity sensitivity, not redundancy between programme criteria.
Changed-context transfer. Transfer to insurance choice. Utility curvature handles outcome risk attitude, while non-additive event capacity handles ambiguity attitude. Keeping layers separate prevents one parameter from explaining everything.
Limits. Behavioural interpretations depend on axioms and can be contested. A fitted capacity may mix belief, preference, framing and noise. Avoid psychological diagnosis from one choice.
Misconception to avoid. Non-additive expectation means probabilities are wrong. It is one model for decisions where additive subjective probability and standard expected utility do not fit the declared preference structure.
Practical evidence check
- Take one example and label separately the states, outcomes, utility values and capacity; reject any explanation that calls all four “weights.”
- Return to chapter contents
CHAPTER 7 OF 10 · Test evidence and alternatives
7. Compare models and test robustness
Core idea. A flexible Choquet model can fit choices well yet produce unstable recommendations. Robust judgement asks whether rankings survive plausible capacities, scales and interaction restrictions and whether a simpler model reaches the same decision.
Method. Define a feasible set from elicitation uncertainty, monotonicity and policy constraints. Sample or optimize over it to find necessary and possible rankings, regret and reversal conditions. Compare weighted sum, Choquet and bottleneck rules.
Worked example. Option B outranks A for most capacities that value access-support complementarity, but A wins when that interaction falls below a documented threshold. The recommendation states the reversal condition and seeks evidence about programme support rather than claiming one exact score.
Changed-context transfer. In supplier selection, a robust set may leave two suppliers incomparable. The next move is not arbitrary averaging; identify the criterion or interaction information that would resolve the decision and assess whether it is worth collecting.
Limits. Robust sets can be too wide to guide action or falsely narrow if constraints reflect group conformity. Include dissenting capacities and changed-context stress tests.
Misconception to avoid. An ambiguous ranking means the method failed. It may honestly reveal that available preference information does not justify a unique choice, protecting against false precision.
Practical evidence check
- Remove each interaction and widen each elicited parameter in turn; report the smallest change that reverses the preferred option.
- Return to chapter contents
CHAPTER 8 OF 10 · Handle limits and disagreement
8. Handle scale, ties and negative values carefully
Core idea. Choquet aggregation depends on ordered values and their differences. Arbitrary scale mixing, inconsistent direction and unexamined negative values can make a calculation meaningless even when the capacity is valid.
Method. Construct commensurate value functions with clear zero and one anchors, reverse cost criteria intentionally, and document treatment of ties and values below reference. For bipolar gains and losses, choose an appropriate extension rather than forcing the nonnegative formula.
Worked example. A cost criterion is entered as dollars while quality is a zero-to-one score. Sorting raw numbers lets units, not judgement, control the coalition path. Converting both to stakeholder-validated value functions changes the ranking for defensible reasons.
Changed-context transfer. For performance above and below a target, a bipolar model may distinguish attractive and repulsive contributions. The neutral reference must be substantive, not chosen to benefit one option.
Limits. Value functions contain trade-offs and can be disputed. Ordinal survey scales may not support subtraction. Preserve raw measures and sensitivity around mappings.
Misconception to avoid. Normalizing every column to zero and one solves commensurability. Mechanical normalization sets endpoints but does not establish that equal intervals carry equal value or that extreme observations are legitimate anchors.
Practical evidence check
- Ask a stakeholder what a ten-point change means at low and high levels for every criterion; revise any value scale that lacks a consistent interpretation.
- Return to chapter contents
Model comparison
| Model | Strength | Warning |
|---|---|---|
| Weighted sum | Simple, additive and transparent | Cannot represent criterion interaction |
| Choquet integral | Models complementarity and redundancy | Many parameters and scale assumptions |
| Min or bottleneck rule | Highlights weakest essential criterion | Can ignore compensating strengths |
| Robust capacity set | Shows ranking across plausible models | May leave options incomparable |
A more flexible model is not automatically better. Compare conclusions, explanation quality and stability against simpler alternatives.
CHAPTER 9 OF 10 · Handle limits and disagreement
9. Prevent overfitting, opacity and parameter explosion
Core idea. A general capacity over many criteria has exponentially many parameters. Without structure it can memorize observed rankings, become impossible to explain and produce unsupported interactions. Restriction is a feature when it reflects what can be known.
Method. Use low-order or sparse capacities, regularization, hierarchical criteria or predefined interactions. Reserve a holdout set, test monotonicity and stability, and publish the parameters and examples needed to explain decisions.
Worked example. A ten-criterion model fits twenty historical choices perfectly with hundreds of effective degrees of freedom. On new profiles rankings swing wildly. A two-additive capacity with five justified interactions fits slightly worse but remains stable and understandable.
Changed-context transfer. In a small public programme, direct rule-based aggregation may be more legitimate than a statistically fitted capacity. In a repeated industrial process, validated estimation can support a richer model.
Limits. Sparsity can omit a real higher-order bottleneck, and explanation summaries can oversimplify. Retain challenge routes and performance monitoring.
Misconception to avoid. The model with the best in-sample fit is the most accurate expression of judgement. It may be the best expression of noise, strategic choices or an obsolete decision culture.
Practical evidence check
- Compare train and holdout rankings, parameter confidence and explanation length for a full and restricted model; choose using all three, not fit alone.
- Return to chapter contents
CHAPTER 10 OF 10 · Transfer and govern
10. Govern use, explanation and revision
Core idea. A Choquet score can affect real allocation, access or safety. Governance must connect capacity design, stakeholder authority, data quality, recommendation, appeal and later revision. The formula is one component of a decision institution.
Method. Publish the decision purpose, criteria, scales, capacity interpretation, interaction rationales, uncertainty, reviewer roles and reversal triggers. Give affected people an understandable explanation and a route to correct source data or challenge policy. Version every revision.
Worked example. A scholarship applicant sees that access and support were treated as complementary, which measures were used and how to correct an incorrect support assessment. An appeal updates the raw value and all derived rankings, while policy disagreement goes to a separate review body.
Changed-context transfer. Move to safety investment. Some constraints become non-compensatory: an option failing a legal minimum cannot be rescued by a high Choquet score. The method sits inside, not above, the authority framework.
Limits. Public explanation may need to protect personal or proprietary data, but secrecy increases the need for independent audit. Monitoring must detect group disparities and gaming.
Misconception to avoid. A transparent capacity table guarantees a fair decision. Fairness also depends on criteria, evidence, participation, constraints, implementation and remedy.
Practical evidence check
- Trace one recommendation from raw data through value scales and capacity contributions to human decision and appeal; verify that a correction propagates through the full route.
- Return to chapter contents
Build and check a small capacity
Start with three criteria. Define the empty set, singletons, pairs and full set, then record a plain-language reason and evidence for every non-additive pair.
Check normalization and monotonicity automatically. Calculate Shapley importance and pair interaction, then inspect whether summaries match stakeholder meaning.
Evaluate representative, adversarial and tied profiles by hand and software. Preserve ordering, coalition sets and marginal contributions so the result is reproducible.
Widen uncertain values into a feasible set and report necessary, possible and reversible rankings instead of hiding elicitation disagreement.
Ask a second reviewer to reconstruct the capacity from the written rationales. Differences expose hidden assumptions about whether a coalition value describes preference, evidence overlap, implementation dependence or ambiguity.
Audit criterion scales before aggregation
List raw measures, units, direction, data quality, affected groups and the decision meaning of the value function. Mark ordinal fields that do not support arithmetic differences.
Test anchors and intervals with concrete trade-off questions. Compare mechanical normalization with stakeholder-validated mappings and show ranking changes.
Inspect missing data, outliers and proxy effects before applying the capacity. Interaction cannot repair a criterion that is invalid or discriminatory.
Store raw data and transformation versions. An appeal must be able to correct the source and recompute every downstream result.
Stress the anchors with unusually good and poor cases, then add an option outside the historical range. Verify that rescaling cannot silently rewrite earlier decisions unless the policy explicitly uses a population-relative standard.
Run a robust ranking analysis
Define constraints from elicitation, monotonicity, known interaction signs and policy. Include dissent and plausible scale variation.
Find best and worst score for each option, necessary and possible preference, maximum regret and smallest parameter change that reverses the recommendation.
Compare with weighted sum and bottleneck rules. Where all reasonable models agree, the decision is stronger; where they differ, explain the structural reason.
Identify the next information worth collecting. Do not collect more data if the unresolved issue is a value disagreement requiring legitimate deliberation.
Report consequential reversal regions even when random sampling visits them rarely. Sampling frequency can hide a small but plausible parameter corner whose substantive rationale matters to safety, rights or a protected group.
Design a usable explanation and appeal
Show the applicant or decision owner the criteria, values, important coalitions, non-compensatory constraints and human authority in plain language.
Let people correct facts separately from challenging the capacity or policy. Give each route an owner, deadline and power to change the outcome.
After correction, recompute rankings and dependent records. Record restoration or compensation where a wrong decision caused harm.
Aggregate appeals and reversals to find invalid criteria, unstable interactions or disproportionate effects, then revise the model through the same governed process used to create it.
Ask a second reviewer to explain the result from stored inputs without seeing the first notice. Differences expose hidden tie rules, scale conventions, interaction summaries or human overrides that the public explanation should make clear.
Questions readers ask
Is a capacity a probability distribution?
- Not necessarily. It is a monotone set function. In some models it represents non-additive belief; in multi-criteria work it often represents coalition importance.
Why not use an ordinary weighted sum?
- Use it when additivity is defensible. A Choquet integral is useful when complementarity or redundancy materially changes value.
How many capacity values are needed?
- A full capacity grows exponentially with criteria. Practical models often restrict interaction order, use sparsity or work with robust sets.
Can a Choquet integral combine any scales?
- No. Ordering and score differences must have a defensible common meaning, or value functions and another method are required.
Does positive interaction prove synergy?
- No. It describes the model. Causal synergy needs separate evidence.
What if no option wins for every plausible capacity?
- Report incomparability, reversal conditions and the information or deliberation needed to resolve it.
Practise, verify and continue
Choose three criteria with one plausible complementarity and one redundancy. Build a normalized monotone capacity, compute two alternatives by hand, compare a weighted sum, calculate sensitivity to the interaction values and write a plain-language explanation that an affected person could challenge.
Move the model from programme selection to evidence aggregation or safety investment. State whether the capacity now concerns criteria, events or beliefs; rebuild the scale and governance instead of reusing the numbers.
Related eduKate reading
Imprecise probabilities, credal sets and robust choice
Possibility theory, necessity measures and maxitive uncertainty
Random sets, belief functions and decision bounds
Probability boxes and robust decisions
Stochastic dominance and partial preferences
Authoritative sources and scope
David Schmeidler, Subjective Probability and Expected Utility without Additivity
Schmeidler 1989, Econometrica article DOI record
Grabisch and Roubens, Application of the Choquet Integral in Multicriteria Decision Making
Michel Grabisch, Review of Methods for Capacity Identification in Choquet Integral Models
Angilella, Corrente and Greco, robust acceptability analysis for Choquet preference models
This article is an educational decision-analysis guide. The numerical examples are illustrative, not recommendations about any real applicant, programme or investment. A Choquet integral is appropriate only when the capacity interpretation, criterion scales and interaction structure are defensible. It does not prove causality, legitimacy or fairness. High-stakes use requires stakeholder participation, validation, uncertainty, simple-model comparison, non-compensatory legal and safety constraints, independent review, accessible explanation and effective correction. Do not infer a person’s psychology from one fitted capacity or hide a value dispute inside technical parameters.