The Gold Standard · Experimental Design
Make the decision path visible
A decision may depend on weights for underrepresented groups, stakeholder priorities or uncertain nonresponse adjustments. Sometimes the evidence does not justify one trusted scheme. Picking a single vector and adding decimal places turns unresolved representation into false precision.
The gold standard defines a transparent set of admissible weights, tests every action across it and uses regret, reversibility and value of information to decide what can responsibly happen next.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · BUILD THE SET
1. State why no single weight scheme is trusted
The uncertainty may come from low response, sparse groups, competing calibration targets, uncertain preference intensity or legitimate disagreement about whose interests receive what role.
Name the source. A set that mixes statistical uncertainty with value disagreement can still be useful, but the two need separate explanation and governance.
CHAPTER 2 OF 12 · BUILD THE SET
2. Define the target and admissible components
Write the population, estimand, actions, outcomes and weight components. Decide which factors may vary and which are fixed by sampling design, law or a rights constraint.
Do not vary a weight merely because it is inconvenient. Each admissible range needs evidence, a calibration condition or a legitimate value rationale.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · BUILD THE SET
3. Construct a feasible uncertainty set
Possible sets use lower and upper bounds, calibration tolerances, distance from a baseline, subgroup-share ranges or several named schemes. Enforce non-negativity, normalisation and population constraints.
Show the geometry in plain language. A broad rectangle can include impossible combinations; a constrained set is usually more honest.
| Input | Possible treatment |
|---|---|
| Base weights | Usually fixed by design |
| Nonresponse | Model range |
| Calibration | Tolerance constraints |
| Preference | Named or bounded schemes |
CHAPTER 4 OF 12 · COMPARE ACTIONS
4. Calculate action loss across the set
For each admissible weight scheme, combine the outcome evidence and loss rule for every action. Preserve outcome uncertainty as a separate layer or a joint set where appropriate.
Record the best action at each point and the range of loss for each action. A single average across schemes can hide an action that performs badly in one plausible region.
CHAPTER 5 OF 12 · COMPARE ACTIONS
5. Define regret carefully
Regret is the loss of an action minus the loss of the best action under the same admissible scheme. It asks how much is given up if that scheme were the relevant one.
Worst-case loss and worst-case regret answer different questions. Minimax loss is cautious about absolute harm; minimax regret is cautious about choosing the wrong alternative relative to what could have been achieved.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · COMPARE ACTIONS
6. Worked example: two plans and three plausible schemes
Plan A has outcome scores 72, 68 and 55 across three groups; Plan B has 67, 70 and 63. The admissible third-group share ranges from 0.15 to 0.30, while the first two shares retain a 5:3 ratio.
Plan A is better near the lower end; Plan B is better once the third group receives enough weight. Calculating regret over the set may favour B because its worst missed opportunity is smaller. Changed-context check: if Plan A is reversible and B commits the full budget, option value may justify an A pilot even when B minimises static regret.
| Measure | Question |
|---|---|
| Loss range | How bad can this action be? |
| Regret | What opportunity is missed? |
| Dominance | Is one action never worse? |
| Value of information | Would new evidence change choice? |
CHAPTER 7 OF 12 · CHOOSE ROBUSTLY
7. Check dominance before using a compromise rule
If one action has no greater loss across the entire set and is better somewhere, it dominates. Remove dominated actions before debating aggregation.
When frontiers cross, the decision genuinely depends on assumptions or values. Do not manufacture dominance by excluding the region that favours the less convenient action.
CHAPTER 8 OF 12 · CHOOSE ROBUSTLY
8. Use minimax regret as one governed rule
Select the action with the smallest maximum regret only when that caution matches the decision context. Publish the scheme that creates each action’s maximum regret.
Minimax regret is not neutral. It values protection from the largest missed opportunity and may ignore probability information or absolute catastrophe unless those enter the loss model.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · CHOOSE ROBUSTLY
9. Add reversibility and value of information
A reversible pilot, staged rollout or monitoring trigger can reduce the cost of choosing under weight uncertainty. Estimate whether new responses, better calibration data or deliberation could change the preferred action.
Collect information where it can change the decision, not merely where it is easy. A sparse influential group often has high decision value.
| Finding | Interpretation |
|---|---|
| One action dominates | Set-wide preference |
| Frontiers cross | Assumption-dependent |
| Near-equal regret | Consider reversibility |
| Sparse group controls | Gather targeted evidence |
CHAPTER 10 OF 12 · GOVERN UNCERTAINTY
10. Deliberate about the set, not only the answer
Stakeholders should be able to challenge the admissible ranges, constraints, outcome definitions and loss rule. Preserve dissent and show which changes alter the decision.
Formal authority remains explicit. Consultation improves evidence and legitimacy but does not automatically settle nontradeable rights or statutory duties.
CHAPTER 11 OF 12 · GOVERN UNCERTAINTY
11. Report conditional conclusions
State which action is preferred across the whole set, across most named schemes or only within one region. Show the decision boundary and the worst-regret scenario.
Avoid saying “robust” without naming the set. Robustness is always relative to the assumptions that were allowed to vary.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · GOVERN UNCERTAINTY
12. Update the set when evidence or values change
Define triggers such as new response data, changed population controls, repeat elicitation, observed burden or legal change. Version the set, code and decision.
The gold standard can reach a decision without pretending one weight scheme is true. It preserves the conditions under which the choice should be revisited.
| Record | Purpose |
|---|---|
| Set definition | Bound assumptions |
| Loss rule | Expose values |
| Worst scenario | Explain protection |
| Revision trigger | Keep decision reviewable |
Frequently asked questions
Is an uncertainty set just a confidence interval for weights?
Not necessarily. It can combine statistical ranges, calibration constraints and several legitimate value schemes, which should remain distinguishable.
Is minimax regret always the safest choice?
No. It protects against the largest missed opportunity, not automatically the largest absolute harm or rights violation.
Should every imaginable weight be included?
No. Include schemes supported by design evidence, feasible calibration or legitimate governance arguments.
Can a decision be made without one trusted scheme?
Yes, by identifying dominance, minimising governed regret, staging reversible action or collecting information that resolves the relevant uncertainty.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Preference-weight sensitivity maps and decision-flip thresholds
- Preference-weight trimming and effective sample size
- Preference-model nonresponse and representation weights
- Loss-function elicitation and decision legitimacy