eduKateSG · Judgement · The Gold Standard of Education
Turn verbal differences of attractiveness into a checked cardinal value scale
Choose the explanation you need, or follow all twelve chapters in order.
Full chapter index · Independent practice · Judgement subject guide
MACBETH—Measuring Attractiveness by a Categorical Based Evaluation Technique—helps a decision maker build value scores from pairwise verbal judgements. Instead of asking for exact numerical differences, it asks whether one option is more attractive than another and whether that difference is very weak, weak, moderate, strong, very strong or extreme.
The numerical scale is not a fixed translation table for those words. It is constructed so that all stated difference categories can be represented consistently. This guide makes that constraint system visible, works a small example, and tests how the result changes when a judgement is revised or left imprecise.
Continue from The Gold Standard of Judgement within The Gold Standard of Education: learning that transfers into life and the How X Works learning library.
Full chapter index
See the mechanism · Chapters 1–2
Build the worked model · Chapters 3–5
Test what changes · Chapters 6–8
Check evidence and limits · Chapters 9–10
Practise and continue · Chapters 11–12
Name the decision owner, options, criterion, evaluation levels, date and context. For a single criterion, ask how attractive each performance level is for the decision—not how probable, costly or morally worthy it is unless that is the criterion’s stated meaning.
Keep factual performance descriptions separate from value judgements. “Response time is 12 minutes” is evidence; “12 minutes is moderately more attractive than 20 minutes” is a preference statement. Both can be wrong in different ways.
MACBETH can build a scale for option performance and can also support criterion weighting through swing comparisons. Do not mix these tasks silently. This article first builds one within-criterion scale.
Evidence check: for Define the decision and one value question, preserve the observation before the interpretation. Record decision owner, options, criterion and context. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
For an ordered pair (x,y), first establish whether x is more attractive than y, indifferent, or incomparable under the current question. If x is preferred, classify the positive difference as very weak, weak, moderate, strong, very strong or extreme.
The words order differences; they do not impose equal numerical spacing. A strong difference must exceed a moderate difference, but the method does not begin by declaring “strong equals four points.” The interval boundaries are inferred with the whole judgement set.
Allow a union of adjacent categories when the evaluator is unsure, such as weak-to-moderate. Forced precision creates brittle models. Record the wording and any explanation, because two people can use the same adjective differently.
Evidence check: for Understand the semantic categories, preserve the observation before the interpretation. Record preference direction and semantic categories. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Place performance levels in a matrix and enter one judgement for each relevant ordered pair. Start with clear contrasts and adjacent levels rather than demanding every cell immediately. The matrix should preserve indifference and blank or uncertain cells distinctly.
Order the levels provisionally from most to least attractive. If the ordinal order itself is uncertain, resolve that before interpreting difference categories. A cardinal scale cannot repair a confused direction.
Use descriptors that are mutually understandable. For service reliability, levels might be 99.9, 99.0, 97.0 and 90.0 percent; for a qualitative criterion they might be complete, partial and absent safeguards with observable definitions. Labels without operational meaning invite inconsistent judgements.
Evidence check: for Build the judgement matrix, preserve the observation before the interpretation. Record matrix cells, uncertainty and descriptors. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Suppose four response-time levels are Excellent, Good, Fair and Poor. The evaluator judges Excellent–Good weak, Good–Fair moderate, Fair–Poor strong, Excellent–Fair strong, Good–Poor very strong, and Excellent–Poor extreme.
One compatible anchored scale might assign Poor=0, Fair=25, Good=60 and Excellent=80. The adjacent differences are 25, 35 and 20; those raw values alone do not mirror the category labels because all pairwise inequalities and chosen anchors determine the feasible scale. A different compatible scale may also exist.
The point of the exercise is not to find the numbers hidden inside the words. It is to construct a numerical representation consistent with the pattern of verbal difference judgements, then discuss whether that representation is useful.
| Level | Value | Selected pair judgement | Interpretation |
|---|---|---|---|
| Excellent | 80 | Excellent–Good: weak | Representative score |
| Good | 60 | Good–Fair: moderate | Not a fixed word conversion |
| Fair | 25 | Fair–Poor: strong | All pairs constrain fit |
| Poor | 0 | Anchor | Reference, not natural zero |
Evidence check: for Construct a fictional scale, preserve the observation before the interpretation. Record every verbal judgement, anchor and value. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Let v(x) denote the value of level x. A judgement that the difference x–y belongs to a higher semantic category than z–w requires v(x)−v(y) to exceed v(z)−v(w) by a positive separation under the formulation. Indifference requires equal value.
Category membership can be represented by common interval bounds whose order is constrained. Anchors such as worst=0 and best=100 fix location and scale; they do not create preference information.
Feasibility is the first test. If no scale and category intervals satisfy all statements, the judgement set is semantically inconsistent. Solver tolerance should be small relative to the anchored range and reported rather than hidden.
Evidence check: for See the linear constraints, preserve the observation before the interpretation. Record linear inequality and category-order meaning. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
An inconsistency may arise when two differences are assigned categories whose required order contradicts other comparisons. It may also reveal that a performance description was misunderstood or that the decision context changed.
Identify a small conflicting set and show possible repairs without selecting one automatically. Ask which judgement the evaluator wishes to revisit and why. The goal is not to make the person obey a machine; it is to expose a pattern that cannot presently be represented together.
Keep an audit trail of original and revised judgements. Repeatedly editing cells until the software turns green can erase the reasoning. A consistent but poorly considered matrix is not high-quality evidence.
Evidence check: for Diagnose inconsistency constructively, preserve the observation before the interpretation. Record small conflicting set and revision audit. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Not every pair needs a precise category. Missing or interval-valued judgements leave a larger feasible set of scales. One displayed scale is therefore a representative, not the uniquely true value function.
Explore ranges for each level or sample alternative compatible scales. If a decision changes across them, the model needs another informative judgement or a cautious conclusion. If the ranking remains stable, the incompleteness may be harmless for the present choice.
Choose new questions that separate consequential possibilities. Explain why the pair matters. Elicitation should reduce decision-relevant ambiguity, not maximise the number of filled cells.
When several scales remain feasible, show a narrow range only if it is calculated from the complete constraint set. Do not vary one score while freezing all others, because the category intervals and remaining values may have to move together. A valid lower or upper bound comes from a new optimisation over the same admissible model, with the target value minimised or maximised. This preserves the joint structure that gives the scale its meaning.
Evidence check: for Handle incompleteness and non-uniqueness, preserve the observation before the interpretation. Record incomplete judgements and compatible scales. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
A cardinal value scale is unique only up to a positive affine transformation when the represented difference structure is fixed. Rescaling 0–100 to 0–1 changes the labels, not the comparisons of value differences.
Anchors must have clear meanings. “Worst” may mean the least attractive level currently considered, an absolute unacceptable level, or a neutral reference. Those are different. Document which one is used before weighting or comparing criteria.
Do not interpret 80 as twice as attractive as 40. An interval scale supports comparisons of differences: the 20-point gain from 60 to 80 can be compared with another 20-point gain on the same scale. Ratios to an arbitrary zero lack that meaning.
Evidence check: for Anchor and rescale without changing meaning, preserve the observation before the interpretation. Record affine scaling and anchor interpretation. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
For each criterion, build a value scale over its performance levels. Then construct weights by comparing value swings from a defined lower reference to a defined upper reference, not by asking which criterion is “important” in the abstract.
An additive model takes V(a)=Σ wj vj(a) under preferential-independence and compensation assumptions. Check interactions and veto-like concerns before aggregation. A safety minimum may need a constraint rather than a low score compensated by convenience.
Weights are scale factors tied to the chosen swing ranges. Changing a criterion’s worst or best reference changes the meaning of its weight. Never copy weights into a model with different descriptors.
| Element | Question | Needed record | Common mistake |
|---|---|---|---|
| Within-criterion scale | How attractive is each level? | Descriptors and pair judgements | Treating scores as facts |
| Swing weight | How valuable is worst-to-best change? | Both reference levels | Abstract importance rating |
| Aggregation | Can gains compensate losses? | Independence and constraints | Automatic additivity |
| Sensitivity | What changes the choice? | Ranges and alternatives | One decimal ranking |
Evidence check: for Extend carefully to multiple criteria, preserve the observation before the interpretation. Record swing weights and compensation assumptions. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 10 OF 12 · Check evidence and limits
10. State limits and common misconceptions
MACBETH does not turn ordinary language into objective measurements. It structures one decision maker’s or group’s judgements and derives compatible cardinal representations. Facilitation, framing and descriptor quality still matter.
The seven categories are ordered but not equally spaced. A consistent matrix does not prove stable preferences, and a displayed score is not a probability, forecast or moral truth. Group agreement can also hide power differences; record who spoke and how dissent was handled.
Sensitivity analysis is necessary. Test uncertain categories, anchors, descriptor ranges, weights and aggregation rules. If minor revisions reverse the recommendation, report a fragile decision rather than extra decimals.
Evidence check: for State limits and common misconceptions, preserve the observation before the interpretation. Record language, power and consistency limits. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Return to the fictional four-level example. The evaluator revises Fair–Poor from strong to weak-to-moderate after seeing that “Poor” still meets a basic standard. Re-run feasibility and the value scale; do not manually lower Fair’s score while keeping every other number fixed.
Now leave Good–Fair unspecified. If the recommendation remains the same across all compatible scales, the blank may not require another question. If two options exchange order, ask a targeted comparison whose answer can resolve that decision-relevant ambiguity.
Evidence check: for Practise a changed judgement, preserve the observation before the interpretation. Record the revised and blank judgement cases. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Archive the decision question, owner, option set, criterion descriptors, evaluation data, matrix wording, uncertain cells, category definitions, original and revised judgements, inconsistency explanations, anchors, feasible scale ranges, chosen representative scale, weights, sensitivity runs and decision note. Frequently asked questions: Are semantic categories fixed numbers? No. Does consistency prove correctness? No. Can uncertain categories be retained? Yes. Are weights abstract importance scores? No; they scale defined value swings.
Continue with UTA preference disaggregation and marginal value functions, robust ordinal regression, and PROMETHEE flows and rank reversal. Consult the original paper on MACBETH and cardinal value functions, the official M-MACBETH user guide, and the project’s concise explanation of qualitative difference judgements and value scores. State the exact formulation and software version because implementations can differ.
Evidence check: for Build the evidence route and continue, preserve the observation before the interpretation. Record the narrowest compatible-scale claim supported. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.