THE GOLD STANDARD OF JUDGEMENT · SPECIALIST STUDY
Make priority visible before aggregation: define which higher-level requirements constrain lower-level gains, then test whether ties, near-ties or band boundaries reverse the decision
Ordinary weighted averaging allows a sufficiently large score on a lower-priority criterion to compensate for failure on a more important one. Prioritised aggregation changes that logic: satisfaction at higher priority controls how much influence lower levels may exert, while prioritised OWA can still express an attitude toward ordered values within a level. The gold standard separates strict lexicographic rules from graded priority, defines bands and gates before seeing outcomes, works examples by hand, discloses tie policy, and stress-tests discontinuities and rank reversals.
This guide belongs to the Judgement subject guide, the Gold Standard of Education, and the How X Works learning library. Choose a reading route, work the examples, then use the transfer checks to test whether the method survives a changed context.
Choose a reading route
Build the foundations · chapters 1–2
Apply the core method · chapters 3–5
Test evidence and alternatives · chapters 6–7
Handle limits and disagreement · chapters 8–9
Transfer and govern · chapters 10
The whole problem at a glance
| Rule family | Compensation permitted | Key question |
|---|---|---|
| Weighted mean | Across all criteria | Are weights meaningful? |
| OWA | Across ordered positions | What attitude to high or low values? |
| Prioritised aggregation | Lower levels gated by higher satisfaction | How strong is precedence? |
| Lexicographic rule | None across decisive priority | When are values tied? |
A very large finite weight can imitate but does not necessarily equal lexicographic precedence.
CHAPTER 1 OF 11 · Build the foundations
1. Distinguish importance weights from priority precedence
Core idea. Importance weights trade criteria on one compensatory scale; priority states that performance at one level limits whether lower levels may influence the outcome.
Method. Write a verbal rule first, identify prohibited compensation, then choose weighted, OWA, prioritised or lexicographic mathematics that actually implements it.
Worked example. A hospital procurement rule says minimum safety cannot be offset by lower price; a weighted mean with a large safety weight still allows compensation in some ranges.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Real policies may mix hard thresholds, graded priority and legal exceptions.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 1 is complete only when the claimed result survives that review.
Misconception to avoid. A higher numerical weight creates strict priority. Finite weights usually preserve some compensation.
Practical evidence check
- Find values where a low-priority gain overturns the high-priority criterion under the proposed formula.
- Return to chapter contents
CHAPTER 2 OF 11 · Build the foundations
2. Define priority levels and within-level criteria before scoring
Core idea. A priority model requires an ordered partition or relation over criteria, plus an aggregation rule inside each level.
Method. Name each level, justify membership and order, state whether levels are strict or partial, and define within-level OWA or other aggregation with scale and missingness rules.
Worked example. Safety and legal compliance form level one; continuity and accessibility form level two; cost and convenience form level three.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Criteria can serve several purposes and stakeholder priorities may conflict or be incomparable.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 2 is complete only when the claimed result survives that review.
Misconception to avoid. The mathematics will discover priority from data. Priority is a normative input that must be authorised.
Practical evidence check
- Ask two independent reviewers to classify criteria and explain every disagreement.
- Return to chapter contents
CHAPTER 3 OF 11 · Apply the core method
3. Understand the attenuation mechanism in prioritised aggregation
Core idea. In common prioritised constructions, satisfaction at preceding levels produces a cumulative factor that attenuates the effective influence of later levels.
Method. Write each level score and cumulative gate explicitly, calculate contributions in sequence, and show how a low high-priority value reduces later-level voice.
Worked example. If the first-level score is 0.6, second-level evidence may enter with only 0.6 of its otherwise available influence; a further level can be attenuated again.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Different published operators use different t-norms, normalisations and OWA weight generation, so formulas are not interchangeable.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 3 is complete only when the claimed result survives that review.
Misconception to avoid. Priority means sorting criteria by their scores. Priority order is assigned to criteria or levels, not inferred from current magnitude.
Practical evidence check
- Compute the same case with product, minimum and no attenuation and compare interpretations.
- Return to chapter contents
CHAPTER 4 OF 11 · Apply the core method
4. Use OWA inside a priority level with a declared attitude
Core idea. Within one priority band, OWA weights ordered positions rather than named criteria, allowing optimistic, pessimistic or balance-seeking treatment of the band’s values.
Method. Sort only the values belonging to that band, apply a published OWA vector, report orness and attribution, and keep source importance separate.
Worked example. A safety band uses a pessimistic OWA that emphasises its weakest component before the band score gates lower priorities.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. OWA assumes commensurable inputs and can shift named-criterion influence whenever ranks change.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 4 is complete only when the claimed result survives that review.
Misconception to avoid. OWA weights attach to safety, legality and reliability names. They attach to ordered positions unless another hybrid rule is declared.
Practical evidence check
- Swap two named criteria while preserving their values and inspect attribution.
- Return to chapter contents
CHAPTER 5 OF 11 · Apply the core method
5. Separate graded priority from strict lexicographic choice
Core idea. Graded prioritised operators let lower levels contribute when higher levels are sufficiently satisfied; lexicographic rules compare the highest priority first and consult lower levels only on a tie.
Method. State which logic applies, define equality or indifference tolerances, and never describe a smooth weighted approximation as exact lexicographic order.
Worked example. Option A leads the safety score by 0.001. Strict lexicographic choice selects A, while a tolerance-based rule treats safety as tied and compares continuity.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Measurement noise makes exact equality rare and strict precedence can amplify tiny errors.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 5 is complete only when the claimed result survives that review.
Misconception to avoid. Lexicographic means assigning descending large weights. Approximation can fail outside assumed bounds.
Practical evidence check
- Search the admissible range for a counterexample where the large-weight score violates the intended lexicographic order.
- Return to chapter contents
Priority-band example
| Option | Safety band | Service band | Overall interpretation |
|---|---|---|---|
| A | 0.90, 0.85 | 0.60, 0.70 | Higher band satisfied; service contributes |
| B | 0.55, 0.95 | 0.95, 0.95 | Weak safety can suppress service advantage |
| C | 0.90, 0.85 | 0.80, 0.50 | Tie at safety; service rule decides |
| D | 0.90, 0.85 | 0.60, 0.70 | Exact tie requires declared policy |
The precise aggregation depends on the published prioritised-OWA formulation.
CHAPTER 6 OF 11 · Test evidence and alternatives
6. Design tie and near-tie policy before observing winners
Core idea. Exact or practical ties can arise within OWA sorting, between priority bands or at lexicographic comparison, and arbitrary input order can decide outcomes silently.
Method. Define numerical tolerance from measurement resolution, use symmetric averaging or substantively justified secondary evidence, preserve tied sets, and test permutation invariance.
Worked example. Two options differ by 0.0001 on a survey-derived safety score whose uncertainty is 0.03; the policy treats them as tied rather than claiming false precision.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. A wide tolerance can hide real differences and secondary criteria may reintroduce disputed priority.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 6 is complete only when the claimed result survives that review.
Misconception to avoid. Computer precision provides an objective tie rule. Decimal differences can be meaningless relative to evidence error.
Practical evidence check
- Randomly permute tied criteria and options and require identical decisions or a documented lottery.
- Return to chapter contents
CHAPTER 7 OF 11 · Test evidence and alternatives
7. Audit discontinuities at gates and band boundaries
Core idea. Thresholds and multiplicative gates can create sharp changes when a criterion crosses a boundary, even if evidence changes only slightly.
Method. Plot the outcome and every contribution across boundary ranges, use uncertainty intervals, identify cliffs and consider smoother rules only when they preserve the policy meaning.
Worked example. Raising compliance from 0.699 to 0.701 activates a lower band under a hard 0.70 gate, causing a much larger score jump than the evidence warrants.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Some legal or safety rules legitimately require a hard threshold; smoothing them can undermine authority.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 7 is complete only when the claimed result survives that review.
Misconception to avoid. Continuity is always fairer. Sometimes the boundary is the decision rule, but it must be visible and reviewable.
Practical evidence check
- Perturb every gate by the measurement uncertainty and count winner changes.
- Return to chapter contents
CHAPTER 8 OF 11 · Handle limits and disagreement
8. Check admissible scales and normalisation
Core idea. Prioritised and OWA formulas often assume bounded comparable satisfaction values; arbitrary min-max scaling can change gates, ranks and compensation.
Method. Define anchors from substantive meaning, document direction and transformations, test permissible scale changes, and avoid option-set-dependent normalisation when decisions must be stable.
Worked example. Normalising cost against only current bids changes every score when an extreme new bid arrives, altering which priority gate appears satisfied.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Some qualitative criteria resist precise cardinal mapping and may require ordinal or rule-based treatment.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 8 is complete only when the claimed result survives that review.
Misconception to avoid. All values in [0,1] are comparable. Shared range does not create shared meaning.
Practical evidence check
- Apply a permissible transformation to one criterion and test whether the decision interpretation survives.
- Return to chapter contents
Priority stability certificate
| Test | Perturbation | Failure signal |
|---|---|---|
| Band boundary | Move one criterion slightly | Large unexplained jump |
| Tie rule | Permute equal inputs | Order-dependent result |
| Scale | Permissible transformation | Priority meaning changes |
| Option set | Add irrelevant alternative | Selected option reverses |
A stable total can still hide changed contribution paths or gate activation.
CHAPTER 9 OF 11 · Handle limits and disagreement
9. Test option-set, rank and attribution stability
Core idea. Adding, removing or slightly changing alternatives can alter within-band ranks, normalisation and tie sets even when an option’s raw evidence is unchanged.
Method. Run leave-one-option-out, dominated-option and perturbation tests; publish band scores, gate factors and named contributions for every decision.
Worked example. A clearly inferior bid changes min-max anchors, allowing a preferred bid to cross a gate and reverse the ranking between two unchanged bids.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Some context-dependent comparison is intentional; stability tests diagnose rather than automatically prohibit it.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 9 is complete only when the claimed result survives that review.
Misconception to avoid. Irrelevant alternatives cannot affect a score-only formula. Data-dependent scaling and rank rules can make them relevant.
Practical evidence check
- Add a dominated alternative and trace every changed intermediate value.
- Return to chapter contents
CHAPTER 10 OF 11 · Transfer and govern
10. Compare priority models instead of hiding model choice
Core idea. Weighted means, constrained optimisation, outranking, lexicographic rules and prioritised OWA encode different notions of compensation, veto and uncertainty.
Method. Specify a small defensible model set, compare decisions and explanations, use boundary cases to expose disagreements, and connect model choice to the authorised policy.
Worked example. A weighted model selects cheap option B, strict lexicographic selects safe option A and graded priority selects A only above a gate; the disagreement becomes the decision issue.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Model plurality cannot replace responsibility for choosing the governing rule.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 10 is complete only when the claimed result survives that review.
Misconception to avoid. Choose the formula that gives the intuitive winner. That is outcome fitting, not policy design.
Practical evidence check
- Have stakeholders choose among blind worked cases before revealing which formula produced each.
- Return to chapter contents
CHAPTER 11 OF 11 · Transfer and govern
11. Govern priority as a contestable normative rule
Core idea. Priority ordering, band membership, gates, OWA attitude, ties, scaling and exceptions embed values that can distribute benefits and burdens.
Method. Version every rule, publish hand calculations and counterexamples, log contribution paths, monitor drift, allow affected people to challenge evidence and rule application, and retire invalid models.
Worked example. A legal standard changes the highest-priority band; lineage identifies decisions that must be replayed and explains why lower priorities can now contribute differently.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Transparent arithmetic cannot create lawful authority, representative values or a fair remedy.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 11 is complete only when the claimed result survives that review.
Misconception to avoid. Publishing the final score is enough. Accountability requires the precedence chain and boundary behaviour.
Practical evidence check
- Reproduce one decision from raw criteria, then challenge its band order and tie tolerance independently.
- Return to chapter contents
Build the specialist decision record
For “The Gold Standard of Judgement | Prioritised OWA, Lexicographic Bands, Non-Compensatory Precedence, Tie-Breaking and Stability”, begin with one bounded decision rather than an abstract claim. Name the affected people, data or institutions; the time window; the authority for acting; and the conclusion the method is allowed to support.
Preserve source material before interpretation. Record definitions, exclusions, transformations and the person responsible for each judgement. A reviewer should be able to separate an observed fact from an inferred result, a policy choice and a presentation shortcut.
Work one ordinary case in full. Show the input, every material rule or calculation, the output and the action that follows. Then write what the same evidence would mean under the simplest credible alternative method.
Keep an explicit unresolved state. Missing, protected, conflicting or unevaluated information must not become zero, false, consent or absence merely because a form expects one value.
Finish the record with a claim ledger. For every sentence that could drive action, identify its supporting source, inferential step, confidence or legal status, intended reader and expiry condition. This keeps descriptive evidence, formal consequence, professional judgement and institutional authority from collapsing into one impressive-looking conclusion.
Run the boundary-case audit
Select a case near the method’s domain edge: a missing input, a disputed definition, an endpoint, a sparse group, a resource limit or a legal deadline. Boundary cases reveal the contract that routine examples can conceal.
Change one assumption at a time and retain every output. Mark what stays stable, what reverses and what becomes undefined. If several assumptions move together, report that dependence instead of attributing the change to one convenient cause.
Search deliberately for a counterexample to the strongest claim. A method that works on the published illustration can still fail under a permitted input, an alternative grouping, a changed unit or an inaccessible evidence pathway.
Translate the counterexample into plain language: who would receive a different decision, what evidence would be hidden, and which safeguard or narrower claim repairs the problem.
Include an adversarial but plausible case. Let a missing group, ambiguous identifier, extreme weight, conflicting rule or urgent deadline interact with normal system incentives. The aim is not theatrical failure; it is to learn whether ordinary pressure makes the method conceal uncertainty, shift burden or reward a misleading shortcut.
Test independent reproduction
Give a reviewer the bounded inputs, public method contract and permitted access, but not the desired answer. Ask them to reproduce the result and identify every place where judgement, local knowledge or unavailable evidence was required.
Compare the reviewer’s intermediate states, not only the final number or label. Matching outputs can hide different denominators, inference paths, missing-data treatments or authorities that will diverge on the next case.
Where reproduction fails, classify the cause: ambiguous definition, undocumented transformation, incompatible implementation, inaccessible source, calculation error or legitimate alternative interpretation. Each class needs a different remedy.
Retain the disagreement and resolution. A corrected record should show what changed, why it changed and which prior outputs or decisions must be revisited.
Run the reproduction on a changed but structurally similar case after the first discrepancy is repaired. A method is not transferable merely because its authors can explain the original example. The new case tests whether the public contract carries the reasoning or whether hidden expertise is still doing essential work.
Design correction, challenge and remedy
Assign a visible route for correcting inputs, challenging assumptions and reviewing the application of the method. Technical accuracy without a usable remedy leaves the affected person carrying an error they cannot contest.
Separate urgent live-case repair from later system learning. A current decision may need pause, reassignment, fresh evidence or a human review before the organisation finishes its longer investigation.
Protect confidential and privileged material during review, but do not use confidentiality as a reason for unrecorded authority. Layer public explanation, protected oversight and individual reasons according to lawful access.
Verify that correction reaches cached results, derived records, dashboards and future decisions. Closing a complaint is not enough if the old conclusion continues to travel.
Measure remedy from the affected person’s position. A revised database row is useful, but it is incomplete when a hearing, service allocation, published claim or downstream model still relies on the old result. Record notification, restored opportunity, compensation or procedural repair where the governing framework provides it.
Set review and retirement triggers
Review after a change in definitions, population, law, source access, measurement scale, software, vocabulary or incentives. A stable formula can become invalid when the world or the meaning of its inputs changes.
Monitor leading signals as well as final outcomes: coverage gaps, timeouts, overrides, complaints, disagreement, missingness and boundary frequency. These often reveal failure before a headline metric moves.
Predefine thresholds for recalibration, suspension and retirement. Do not force a method to answer after its assumptions fail merely because replacing it is inconvenient.
At retirement, preserve the versioned contract, evidence, known limitations and migration decision. State which historic results remain interpretable and which require warning, restatement or withdrawal.
Schedule one prospective review before confidence fades or controversy forces it. Assign named owners for evidence, implementation and rights impact, and invite at least one reviewer outside the design team. The review should be able to narrow the claim, change the method or stop use without requiring proof of catastrophic failure.
Questions readers ask
Is priority the same as a larger weight?
- No. Priority can restrict or eliminate compensation from lower levels.
What is prioritised OWA?
- A family of rules that combines priority-level influence with ordered-weight aggregation.
When is lexicographic choice appropriate?
- When higher-priority differences must decide before any lower level is considered, subject to a declared tie rule.
Why define a tie tolerance?
- Because evidence resolution and uncertainty may make tiny numerical differences practically meaningless.
Can hard gates be used?
- Yes when substantively authorised, but their discontinuities and uncertainty must be visible.
What should an audit show?
- Band scores, gate factors, ordered positions, named contributions, ties and sensitivity.
Practise, verify and continue
Choose one consequential case. Reproduce the complete method, work one ordinary and one boundary case, test the strongest rival explanation, and state the evidence that would reverse or suspend the result. Build an evidence pack containing original inputs, definitions, calculations or rule steps, alternatives rejected, uncertainty, reviewer comments and the final action. Give the pack to a reader who did not design the method. Ask that reader to reproduce the result, identify one missing assumption and construct a case that would make the method return unresolved. Revise the public explanation only after the evidence pack and the operational process agree.
Move the method into an education or public-service setting. Keep the reasoning structure while replacing domain authority, population, privacy, evidence and correction routes. Start with a low-stakes historical example, then compare the specialist method with a transparent baseline. Check whether the same people, records and decisions remain inside scope; whether the scale or vocabulary still means the same thing; and whether a person affected by error can understand and challenge it. Pilot under human review, record reversals and burden, and stop the transfer if the formal resemblance hides a different ethical or legal problem.
Related eduKate reading
Ordered weighted averaging, orness and compensation
Induced OWA and rank attribution
Sugeno integrals and non-compensatory choice
Authoritative sources and scope
Yager, Prioritized Aggregation Operators
Yager, Prioritized OWA Aggregation
Mitchell and Schaefer, Multiple Priorities in an Induced OWA Operator
Yager, On Ordered Weighted Averaging Aggregation Operators
Merigó and Gil-Lafuente, The Induced Generalized OWA Operator
This article is mathematical and decision-design education. Examples are original teaching cases. Prioritised aggregation encodes normative precedence, compensation, scale and tie choices. It does not establish lawful authority or a morally correct priority order. Use human review, publish boundary behaviour and contribution paths, protect sensitive evidence, and provide challenge, correction and retirement mechanisms.