THE GOLD STANDARD OF JUDGEMENT · SPECIALIST GUIDE
Use SMAA to expose which rankings survive incomplete weights instead of hiding uncertainty behind one score
A multi-criteria ranking can look decisive because one set of weights produces one ordered list. Yet decision-makers may not know the weights precisely, stakeholders may disagree, and performance estimates may be uncertain. Stochastic Multicriteria Acceptability Analysis, or SMAA, explores the admissible space instead of forcing a single point estimate too early.
This guide develops rank acceptability indices, central weight vectors and robustness checks through a transparent additive-value example. It separates uncertainty models from truth, counts simulated ranks carefully, tests scaling and dependence, and keeps deliberation and authority visible beside computation.
SMAA is a family of methods, not one universal algorithm. The worked numbers use a simplified weighted-sum teaching model. Real decisions need a justified preference model, valid data, appropriate distributions, reproducible computation and accountable decision-makers.
Full chapter index · Independent practice · Judgement subject guide
This specialist guide belongs to The Gold Standard of Judgement, the Gold Standard of Education and the How X Works learning library.
Full chapter index
Frame the uncertainty · Chapters 1–3
Work the acceptability · Chapters 4–6
Stress-test the result · Chapters 7–9
Govern the judgement · Chapters 10–11
Continue audibly · Chapters 12–12
CHAPTER 1 OF 12 · FRAME THE UNCERTAINTY
1. Start with the decision and the information that is genuinely missing
SMAA is useful when alternatives must be compared across several criteria but preference weights, performance values or both are incomplete or uncertain. The method samples from a declared set or distribution of admissible inputs, applies a chosen aggregation model and records how often each alternative reaches each rank.
The first task is not simulation. It is to define the alternatives, criteria, direction of preference, decision horizon and stakeholders. Then distinguish unknown from disputed. A weight may be unknown because no elicitation occurred, or broad because several stakeholders hold legitimate different priorities. Those situations may deserve different models and communication.
State what the analysis is for. Screening alternatives, selecting one project, identifying robust options and structuring a negotiation are different jobs. A rank acceptability table can support all of them, but the decision rule and consequences should not be smuggled into the sampling code.
Evidence check
Record the source and status of every performance value and weight statement: measured, forecast, elicited, bounded, ordinal, disputed or absent. Keep exclusions visible.
Changed-context transfer
If the task changes from choosing one alternative to building a portfolio, rank acceptability alone may not represent interaction, budget and diversification constraints. Rebuild the model.
Limit and misconception check
Missing weights do not mean every weight vector is equally plausible. The admissible space must reflect defensible preference information or an explicitly neutral exploratory choice.
Practical decision record
Write the decision sentence, alternatives, criteria, missing information and intended use of the robustness output before selecting distributions.
CHAPTER 2 OF 12 · FRAME THE UNCERTAINTY
2. Treat distributions as models of uncertainty, not frequencies of truth
SMAA represents incomplete information through probability distributions or sampling rules. A uniform distribution over feasible weights can express the exploratory choice to give equal sampling density to that parameterisation. It does not prove that stakeholder preferences occur uniformly in the real world.
Performance uncertainty can be represented with empirical samples, fitted distributions, intervals or scenarios, depending on evidence. Bounds should respect physical and logical constraints. Correlated criteria require joint sampling; drawing them independently can create impossible combinations and misleading rank frequencies.
Separate aleatory variation from epistemic uncertainty when it changes interpretation. Repeated operational variation and lack of knowledge about a fixed parameter are not the same, even if both are sampled. The model may still combine them for decision support, but the report should say what each distribution means.
Evidence check
For every distribution, document data, parameter estimation, bounds, dependence, truncation and the reason the chosen form fits the uncertainty claim.
Changed-context transfer
If stakeholder weights are available as an ordinal statement such as safety exceeds cost, restrict samples to that order rather than drawing from the unrestricted simplex.
Limit and misconception check
A simulated percentage is conditional on the distribution model. It is not an objective probability that an alternative is truly best.
Practical decision record
Create a distribution register linking each uncertain input to its interpretation, evidence, sampling rule and sensitivity alternative.
Contents · Previous chapter · Next chapter
CHAPTER 3 OF 12 · FRAME THE UNCERTAINTY
3. Choose and scale the value model before sampling
The aggregation model translates criterion performance and weights into preference. In the simplified additive model, each criterion value is placed on a comparable scale, multiplied by a non-negative weight and summed. Weights normally sum to one. This model assumes preferential conditions that may not hold when criteria interact or thresholds and vetoes matter.
Scaling is substantive. Min–max normalisation depends on the observed alternatives and can change when an extreme option is added. Value functions tied to meaningful performance anchors can be more stable, but their shapes require justification. A cost criterion must be transformed so that higher value consistently means better before aggregation.
SMAA does not repair a poor value model. It can reveal sensitivity within the model, yet repeated simulation of incomparable or double-counted criteria produces a precise description of the wrong construct. Inspect criteria definitions and overlap first.
Evidence check
Publish raw units, direction, value functions, anchor points, weight interpretation and any thresholds or interactions omitted by the additive model.
Changed-context transfer
Add an extreme but infeasible alternative to a min–max-scaled dataset. If the existing ranks change only because the scale changed, diagnose a normalisation artefact.
Limit and misconception check
Weights depend on scale. A weight of 0.4 has no stable meaning if the criterion value range or value function is silently changed.
Practical decision record
Save the raw matrix and the transformed value matrix separately, with formulas and versioned anchors.
Contents · Previous chapter · Next chapter
CHAPTER 4 OF 12 · WORK THE ACCEPTABILITY
4. Work a small performance matrix before running thousands of draws
Consider three fictional resilience projects assessed on two value-scaled criteria: service performance and affordability. Values run from zero to one, with higher better. Project A emphasises performance, Project C emphasises affordability and Project B is balanced. The numbers are constructed to expose weight sensitivity.
Let the performance weight be w and the affordability weight be one minus w. The additive score for A is 0.9w plus 0.4(1−w); B is 0.7 on both criteria; C is 0.4w plus 0.95(1−w). Before simulation, solve or inspect where their score lines cross. This reveals why different weight regions favour different projects.
At w below roughly one third, C is attractive; around the middle, B can lead; at higher w, A leads. The exact boundaries follow the supplied values. A simulation should reproduce that structure rather than substitute for understanding it.
| Alternative | Service performance value | Affordability value | Interpretive profile |
|---|---|---|---|
| Project A | 0.90 | 0.40 | High performance, lower affordability |
| Project B | 0.70 | 0.70 | Balanced |
| Project C | 0.40 | 0.95 | High affordability, lower performance |
The matrix makes trade-offs explicit. No project dominates both criteria, so weights matter. If one alternative were at least as good on every criterion and better on one, the admissible value model would normally preserve that dominance.
Check the arithmetic with a few hand-calculated weights before trusting code. At w = 0.5, the scores are 0.65 for A, 0.70 for B and 0.675 for C, so B ranks first.
Evidence check
Verify that each value is derived from a meaningful criterion scale and that the alternatives are feasible under the same scenario and time horizon.
Changed-context transfer
Change B’s affordability from 0.70 to 0.62. Recalculate crossing points and predict how its first-rank region shrinks before resampling.
Limit and misconception check
Two criteria make the geometry visible but understate the complexity of a real high-dimensional preference space and uncertain performance matrix.
Practical decision record
Keep the hand calculations, crossing-point logic and test cases as unit tests for the simulation implementation.
Contents · Previous chapter · Next chapter
CHAPTER 5 OF 12 · WORK THE ACCEPTABILITY
5. Calculate rank acceptability indices as conditional frequencies
A rank acceptability index describes the share of admissible samples in which an alternative attains a particular rank under the chosen model. For each draw, sample weights and uncertain performances as specified, calculate values, rank alternatives and increment the appropriate cells. The row across ranks should sum to one for each alternative, subject to tie treatment and numerical tolerance.
Use five equally weighted teaching draws for the performance matrix: service weights 0.1, 0.3, 0.5, 0.7 and 0.9. Project C ranks first in the first two draws, B in the middle draw and A in the last two. The corresponding first-rank acceptabilities are forty per cent for A, twenty per cent for B and forty per cent for C.
These coarse values illustrate the count. A real Monte Carlo analysis uses enough draws to estimate indices to the required precision and examines convergence. The percentages describe the sampled input space; they do not certify a winner.
| Service weight w | A score | B score | C score | First rank |
|---|---|---|---|---|
| 0.1 | 0.45 | 0.70 | 0.895 | C |
| 0.3 | 0.55 | 0.70 | 0.785 | C |
| 0.5 | 0.65 | 0.70 | 0.675 | B |
| 0.7 | 0.75 | 0.70 | 0.565 | A |
| 0.9 | 0.85 | 0.70 | 0.455 | A |
The full rank table would also count second and third places. That matters because an alternative rarely first but almost always second may be more dependable than one alternating between first and last.
Ties require a declared convention. Random tie breaking, shared ranks and tolerance-based grouping can change acceptability entries when scores are close.
Evidence check
Check row sums, column totals implied by one alternative at each rank, tie handling, seed control and repeated runs with larger samples.
Changed-context transfer
Replace the five fixed draws with a non-uniform preference distribution concentrated near w = 0.5. B’s first-rank acceptability should rise because the sampling question changed.
Limit and misconception check
Rank acceptability is not expected utility, probability of factual superiority or stakeholder approval. It is conditional on the model and sampled information.
Practical decision record
Save the sample count, random seed, weight distribution, tie rule and acceptability matrix with sufficient precision for independent reproduction.
Contents · Previous chapter · Next chapter
CHAPTER 6 OF 12 · WORK THE ACCEPTABILITY
6. Interpret central weight vectors as conditional preference profiles
For an alternative, the central weight vector summarises the sampled weights associated with that alternative achieving first rank, commonly as their conditional average. It answers a diagnostic question: what preference profile tends to make this option most preferred within the sampled model?
In the five-draw example, A wins at service weights 0.7 and 0.9, so its simple conditional mean service weight is 0.8. C wins at 0.1 and 0.3, giving 0.2. B wins only at 0.5. These are teaching summaries of winner regions, not recommended weights and not estimates of what people believe.
A central vector can help explain an option to stakeholders and support a confidence calculation in SMAA variants. Yet an average can fall in a low-density or non-representative part of a non-convex winner region. Inspect distributions and feasible geometry, not only the centroid.
Evidence check
Report how the central vector is calculated, the number and spread of conditional samples, and whether the winning region is connected or multimodal.
Changed-context transfer
If A wins in two separated weight regions because performance is uncertain, its average weight may lie where A seldom wins. Plot or summarise the conditional distribution.
Limit and misconception check
Central weights are descriptive consequences of the model. They should not be presented as elicited stakeholder values or normative compromises.
Practical decision record
Pair every central vector with its first-rank acceptability, conditional spread and a plain-language description of the preference region.
Contents · Previous chapter · Next chapter
CHAPTER 7 OF 12 · STRESS-TEST THE RESULT
7. Check Monte Carlo error, convergence and rare rank regions
Simulation adds numerical uncertainty to the substantive uncertainty model. An estimated acceptability of one per cent may represent a real narrow region or random noise from too few draws. Track estimates as the sample size grows and repeat with independent seeds. Standard errors or intervals can help judge precision.
Rare but important regions deserve targeted investigation. Stratified, quasi-random or importance-sampling approaches may improve coverage when implemented correctly. The objective is not to inflate an option’s acceptability, but to ensure that consequential corners of the feasible space are not missed.
Validate code with problems that have known answers: a dominated alternative, identical alternatives, two-criterion crossing lines and fixed inputs. Reproducibility requires software versions, seeds and data, but deterministic reproducibility does not prove the model is substantively valid.
Evidence check
Plot acceptability estimates against sample count, compare seeds, test row sums and known cases, and record numerical tolerances.
Changed-context transfer
Double the number of alternatives while keeping the same draws. Lower acceptabilities and more close ranks may require greater sampling precision and clearer tie handling.
Limit and misconception check
A stable Monte Carlo estimate can still be stably wrong if the distributions, value functions or alternatives are wrong.
Practical decision record
Set a precision target before simulation and record the stopping rule, validation tests and any targeted sampling used.
Contents · Previous chapter · Next chapter
CHAPTER 8 OF 12 · STRESS-TEST THE RESULT
8. Preserve dependence, constraints and incomplete preference information
Criteria estimates often move together. Cost and delivery time can be correlated; environmental benefits may share a climate scenario; weights must sum to one and may satisfy order or ratio constraints. Independent sampling destroys these relationships and can populate impossible states.
Use a joint model that preserves evidence-based dependence. For weights, sample from the feasible simplex subject to elicited constraints. Rejection sampling may become inefficient in a narrow region, so specialised parameterisations or samplers may be needed. Check that the realised sample actually matches the intended constraints.
Incomplete preference information is valuable. Statements such as safety is at least as important as cost can restrict the space without demanding exact weights. SMAA can show how much ranking uncertainty remains after incorporating that information.
Evidence check
Report correlations, copulas or scenario links, weight constraints, acceptance rates and diagnostics comparing sampled and intended distributions.
Changed-context transfer
Add the rule that service weight must be at least affordability weight. In the worked example, C’s low-service winner region is excluded; recalculate rather than retaining the old percentages.
Limit and misconception check
Constraints can encode bias as easily as knowledge. Every restriction needs an owner, rationale and sensitivity check.
Practical decision record
Maintain a constraint register with the statement, source, mathematical form, affected samples and alternative interpretation.
Contents · Previous chapter · Next chapter
CHAPTER 9 OF 12 · STRESS-TEST THE RESULT
9. Compare robust rank regions without inventing one winner rule
Decision-makers may value different robustness properties. High first-rank acceptability highlights frequent winners. Low probability of poor ranks highlights downside protection. A balanced expected rank can conceal a bimodal pattern. Pairwise winning probabilities answer yet another question. No single summary dominates every decision purpose.
Define the decision rule before seeing the result where possible. A safety-critical choice may screen out alternatives with unacceptable lower-tail performance before comparing acceptability. A reversible pilot may favour learning value. A consensus process may focus on options acceptable across stakeholder regions.
Use dominance and feasibility first. SMAA should not give an infeasible or ethically excluded option a second life simply because it ranks well under some sampled weights. Constraints and rights-based floors belong outside or within the model explicitly.
Evidence check
Present the full rank acceptability distribution, not only first rank. Include feasibility, threshold breaches, pairwise results and performance uncertainty beside preference uncertainty.
Changed-context transfer
Change the objective from choosing one project to selecting a safe shortlist. An option with moderate first-rank and very low worst-rank frequency may become more useful.
Limit and misconception check
Robustness is always relative to a specified uncertainty set and model. It is not immunity to surprises outside them.
Practical decision record
State the selection rule, unacceptable outcomes, reversibility and which robustness summaries are decision-relevant.
Contents · Previous chapter · Next chapter
CHAPTER 10 OF 12 · GOVERN THE JUDGEMENT
10. Use sensitivity analysis to challenge distributions and value functions
A credible study varies modelling choices, not just random draws within one chosen model. Compare weight distributions, performance ranges, dependence, value-function shapes, normalisation anchors and excluded scenarios. If the recommendation changes, identify which assumption drives the change.
Scenario sensitivity and probabilistic sensitivity answer complementary questions. Named scenarios preserve coherent narratives such as high demand and delayed delivery. Probabilistic sampling explores variation within and across declared assumptions. Mixing them without labels can hide the source of uncertainty.
The purpose is learning, not searching for a specification that confirms a preferred option. Pre-register major alternatives when stakes are high and preserve all material results.
Evidence check
Create a sensitivity map that links each modelling choice to changes in acceptability, central weights, threshold breaches and shortlist membership.
Changed-context transfer
Replace min–max scaling with fixed meaningful anchors. If an option’s robustness depends on the current competitor set, explain the rank-reversal pathway.
Limit and misconception check
Testing many models can overwhelm readers. Focus on assumptions with a plausible mechanism and decision consequence, while preserving a technical appendix.
Practical decision record
For each sensitivity run, record the changed assumption, rationale, expected direction, result and decision impact.
Contents · Previous chapter · Next chapter
CHAPTER 11 OF 12 · GOVERN THE JUDGEMENT
11. Connect computation with stakeholder deliberation and authority
SMAA can show which priorities favour each alternative and where evidence uncertainty matters. That makes it a useful conversation structure. It does not decide whose preferences count, how conflicting interests are weighted or which institution has authority.
Stakeholders should be able to inspect criteria, ranges, constraints and outputs in accessible language. Participation can reveal missing alternatives, unequal burdens and criteria that averages hide. Technical experts can correct model errors; affected people contribute lived consequences and values; authorised bodies remain accountable for the decision.
Document reasons for choosing outside the highest acceptability. A lower-ranked option may satisfy a rights floor, distributional obligation or implementation constraint absent from the value model. The explanation should improve the model or clearly state the extra-model reason.
Evidence check
Keep participation records, model revisions, dissent, declared interests, authority, reasons and the route for review or appeal.
Changed-context transfer
If two stakeholder groups have distinct admissible weight regions, report their results separately before pooling. A combined distribution can erase disagreement.
Limit and misconception check
An apparently neutral distribution is still a governance choice. Simulation cannot substitute for legitimate participation and reason-giving.
Practical decision record
Create a decision record linking every adopted criterion or constraint to evidence, stakeholder input or lawful authority.
Contents · Previous chapter · Next chapter
CHAPTER 12 OF 12 · CONTINUE AUDIBLY
12. Continue through a reproducible SMAA judgement pathway
A defensible pathway defines the decision, screens feasibility, builds raw and value-scaled matrices, represents performance and preference uncertainty, preserves dependence, validates the sampler, calculates rank acceptability and central profiles, and stress-tests modelling choices.
The result should answer three questions: which alternatives are robust across the admissible space, which preference profiles favour each option, and which uncertainties could change the decision. The output is useful when it narrows disagreement and identifies better evidence, not when it decorates a predetermined answer.
Version data, code, seeds, distributions, stakeholder constraints and decision rules. Set review triggers for new evidence, changed alternatives or revised priorities. Re-running the model should reproduce the prior result before producing a new one.
Evidence check
The final audit should include alternatives, criteria, raw data, value functions, distributions, constraints, code tests, seeds, convergence, outputs, sensitivities, decision rule and authority.
Changed-context transfer
Apply the pathway to uncertain project costs, uncertain clinical benefits and incomplete stakeholder weights. Preserve the steps but rebuild the model and governance.
Limit and misconception check
This guide teaches SMAA reasoning through an additive example. Other SMAA variants and preference models require their own assumptions and validation.
Practical decision record
Produce a one-page robustness summary with the complete model archive linked behind it and a dated trigger for reconsideration.
Contents · Previous chapter · Return to the Judgement guide
Read a robustness dashboard without collapsing it into a league table
A useful dashboard places the full rank acceptability distribution beside each alternative’s central weight profile, threshold failures and raw performance ranges. Keep the same alternative order across panels. A reader should be able to see whether an option is frequently first, consistently near the top, or polarised between very good and very poor ranks. Showing only the largest first-rank percentage discards that structure.
Add a model-change panel. One column can show the baseline uncertainty model; the others can show meaningful alternatives such as narrower performance evidence, correlated costs and delivery times, a stakeholder weight constraint, or fixed value anchors. Report changes in shortlist membership and rank distributions, not only whether the first name changes. Stability of the decision can be broader than stability of rank one.
Use an evidence-priority panel to connect analysis with the next useful study. If Project A loses mainly when its service estimate is low, better service evidence may have high value. If Project C wins only under weight profiles that no stakeholder endorses, further technical measurement may not resolve the preference issue. If Project B remains near the top across data and weight changes, the decision may be ready for feasibility and implementation review.
Keep the denominator visible on every percentage. A forty per cent first-rank acceptability under unrestricted weights is different from forty per cent after applying stakeholder constraints, and both differ from forty per cent across uncertain performance with fixed weights. Label the sample space, model version and run date so screenshots cannot detach a number from its meaning.
Finally, preserve the alternatives that were screened out and the reason. Feasibility, rights, safety or legal constraints may remove an option before SMAA. A transparent dashboard shows that boundary instead of implying that the simulation considered every imaginable course of action. This makes the result easier to audit and harder to misuse as an automatic decision.
Questions readers ask
Does first-rank acceptability mean the probability an option is truly best?
No. It is the share of samples where the option ranks first under the chosen uncertainty and preference model.
Can SMAA work when no exact weights are known?
Yes, if the admissible weight space or distribution is defined transparently. That choice remains an assumption, not discovered stakeholder preference.
What is a central weight vector?
It summarises weights associated with an alternative attaining first rank. It describes a favourable preference region; it is not automatically the recommended weight set.
Why show the full rank distribution?
First place alone can hide an option that alternates between best and worst, or another that is consistently near the top.
Practise, verify and continue
Independent practice: use the three-project matrix and draw service weights 0.2, 0.4, 0.6 and 0.8. Calculate all scores, ranks and first-rank frequencies. Then impose service weight at least affordability weight and explain why the analysis question changes.
Next, make Project B’s affordability uncertain between 0.60 and 0.80. Design a sampling rule, state whether it is independent of service performance, and identify the output that would reveal whether B is a stable compromise or a fragile middle option.
Related eduKate reading
TOPSIS ideal distances, normalisation and rank reversal
VIKOR compromise solutions, regret and acceptable advantage
EDAS average solution, positive distance and negative distance
MABAC border approximation areas, distance signs and stability
MARCOS ideal and anti-ideal alternatives, utility functions and stability
Authoritative sources and scope
Lahdelma and Salminen, SMAA-2: Stochastic Multicriteria Acceptability Analysis for Group Decision Making
Aalto University research record for the SMAA-2 paper
Tervonen and Figueira, survey on stochastic multicriteria acceptability analysis methods
Lahdelma, Miettinen and Salminen on ordinal criteria in SMAA
The cited papers establish the SMAA family, rank acceptability and central-weight concepts and extensions for incomplete information. The project values and five-draw calculation are original fictional teaching examples. Real use requires a justified value model, uncertainty distributions, reproducible software and accountable decision authority.