The Gold Standard · Experimental Design
Make the decision path visible
A statistical frontier does not choose itself. Once several defensible actions remain, a decision rule needs some account of what is lost when each outcome occurs. Writing that account as a loss function can improve clarity, but the numbers are not discovered in the data alone. They express values, responsibilities and distributional choices.
Different stakeholders may reasonably disagree. A participant may fear a rare severe harm more than an average gain. An operator may see workload and reversibility. A funder may emphasise population benefit. The design problem is not to force these perspectives into one convenient weight. It is to elicit them carefully, show where conclusions depend on them and define who has authority to decide.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · FRAME THE DECISION
1. Separate the evidence model from the value model
The evidence model describes what may happen under each action and how uncertain those consequences are. The value model describes how outcomes are judged. Mixing them makes it difficult to see whether disagreement concerns facts or preferences.
Write the decision table first: available actions, relevant outcomes, affected groups, time horizon and reversibility. Then attach probabilities or ranges from the evidence. Only after that should losses or utilities be elicited.
A loss function should not be tuned until it selects the desired answer. Pre-specify the elicitation and report how alternative plausible losses change the decision.
CHAPTER 2 OF 12 · FRAME THE DECISION
2. Identify stakeholders and decision authority
People can be affected, consulted, responsible, accountable or legally authorised. These roles are not interchangeable. A broad consultation can improve evidence and legitimacy without transferring formal authority.
Map direct participants, families, practitioners, operators, communities, funders and oversight bodies as relevant to the decision. Explain why each perspective is included and where representation remains incomplete.
Do not treat the loudest or easiest-to-recruit respondents as the population. Recruitment and nonresponse can shape the apparent preference distribution.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · FRAME THE DECISION
3. Choose outcomes before weights
Weights are meaningless if the outcome set omits what matters. Include benefits, harms, burden, delay, reversibility, opportunity cost and distribution where they are decision-relevant.
Avoid double counting. If anxiety, treatment burden and quality of life already overlap in a composite, adding each again can amplify one concern. Define scales and time horizons so stakeholders compare the same objects.
Keep process values visible. Trust, procedural fairness and right to appeal may not fit comfortably into a single outcome score, yet they can constrain acceptable actions.
| Element | Question |
|---|---|
| Actions | What can actually be chosen? |
| Outcomes | Who experiences what and when? |
| Evidence | What is known and uncertain? |
| Authority | Who is accountable for the choice? |
CHAPTER 4 OF 12 · ELICIT VALUES
4. Use understandable elicitation tasks
Common tasks include ranking, pairwise comparison, swing weighting, threshold questions and discrete choices. Select a task that respondents can understand and that matches the decision structure.
Use concrete scenarios with absolute risks and consistent denominators. Test whether framing, order and reference points change responses. Provide a practice item and an option to say the trade-off is unacceptable or unclear.
Elicitation is measurement. Record instrument wording, mode, interviewer support, missing responses and revisions.
CHAPTER 5 OF 12 · ELICIT VALUES
5. Check scale, consistency and uncertainty
Preference answers are not exact constants. Repeat selected tasks, vary equivalent presentations and estimate uncertainty or intervals where possible. Apparent inconsistency may reflect learning, ambiguity or genuine context dependence.
Normalising weights to sum to one does not make different people's scales interpersonally comparable. State what transformations are allowed and what conclusions are invariant to them.
Do not discard respondents simply because their answers are difficult to model. Investigate comprehension and preserve a category for unresolved values.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · ELICIT VALUES
6. Work a threshold-disagreement case
Consider a programme choosing between a standard monitoring plan and an intensive plan. The intensive plan is expected to prevent 12 additional serious failures per 10,000 cases but creates 180 additional burdensome reviews and costs capacity that could support another service.
Group A treats one prevented serious failure as equivalent to 25 burdensome reviews; Group B uses 8; operators add a capacity loss equivalent to 40 reviews. Under Group A's loss, the intensive plan is preferred. Under Group B's loss plus the capacity term, the standard plan is preferred.
Changed-context transfer check: if reviews are easily reversible and supported, their loss may fall. If a review can suspend access for weeks, its loss rises. The evidence about event counts did not change; the action changed because consequences and values did. Report both layers.
| Risk | Design response |
|---|---|
| Framing effect | Use equivalent presentations |
| Scale confusion | Practice tasks and plain units |
| Nonresponse | Report who is missing |
| False precision | Use intervals and repeats |
CHAPTER 7 OF 12 · PRESERVE DISAGREEMENT
7. Represent heterogeneity instead of averaging it away
Show the distribution of elicited thresholds and weights, not only the mean. Clusters may correspond to different experiences, roles or exposure to harm. Investigate them without stereotyping groups.
A representative-agent loss function can select an action that few actual stakeholders prefer. Compare decisions under subgroup-specific losses and identify who bears regret under the aggregate choice.
When sample sizes are small, use intervals and qualitative evidence. Precision should not be manufactured by pooling incompatible perspectives.
CHAPTER 8 OF 12 · PRESERVE DISAGREEMENT
8. Choose aggregation rules explicitly
Possible rules include equal-person weighting, population weighting, role-based deliberation, rights constraints, minimax regret and negotiated thresholds. Each embodies a governance choice.
Do not let a mathematical average conceal vetoes or statutory duties. Some harms may be treated as constraints rather than tradeable losses. Likewise, a decision authority may be required to consider population benefit while explaining disproportionate burdens.
Document who selected the aggregation rule and why. Test alternatives that a reasonable critic would propose.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · PRESERVE DISAGREEMENT
9. Run sensitivity and regret analysis
Evaluate each action across a grid or distribution of plausible loss functions. A robust action performs acceptably across disagreement; a fragile action wins only under a narrow weighting.
Regret compares each action with the best action under each loss specification. Minimax regret can be useful when no shared probability over values exists, but it is not neutral: it prioritises protection from the worst missed opportunity.
Display decision boundaries so stakeholders can see which value change would alter the choice. This is often more informative than debating a single decimal weight.
| Rule | What it protects |
|---|---|
| Equal-person average | Symmetric voice in sample |
| Population weighting | Population distribution |
| Rights constraint | Nontradeable minimum |
| Minimax regret | Worst missed opportunity |
CHAPTER 10 OF 12 · EARN LEGITIMACY
10. Design legitimate deliberation
Legitimacy requires more than a survey. Participants need understandable evidence, time to question assumptions, disclosure of conflicts and a record of how input affected the decision.
Separate elicitation from deliberation when helpful. Private tasks can reveal individual values; facilitated discussion can expose reasons and revise misunderstandings. Preserve both pre- and post-deliberation results rather than reporting only the convenient stage.
Compensate participation and provide accessible formats. Otherwise the procedure may systematically exclude people carrying the greatest burden.
CHAPTER 11 OF 12 · EARN LEGITIMACY
11. Pre-specify revision and appeal
Values and circumstances change. Define when the loss model will be revisited: new evidence, a changed population, observed burden, legal change or persistent disagreement outside the original range.
An appeal route should distinguish factual error, misrepresentation of preference and disagreement with the authorised rule. Each needs a different response.
Version the outcome definitions, elicitation instrument, stakeholder sample, aggregation rule and decision threshold. A revised decision should not erase the earlier basis.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · EARN LEGITIMACY
12. Know when one loss function is the wrong endpoint
Do not force one scalar loss when rights are nontradeable, stakeholder groups lack legitimate representation, consequences are poorly understood or aggregation would hide a fundamental conflict.
Alternatives include reporting a set of admissible actions, using constraints, running separate subgroup decisions, collecting more evidence or referring the choice to an accountable deliberative body.
A legitimate decision can remain contested. The gold standard is not unanimous agreement; it is a transparent relationship among evidence, values, authority, uncertainty and avenues for revision.
| Record | Purpose |
|---|---|
| Stakeholder map | Show inclusion and gaps |
| Instrument version | Reproduce elicitation |
| Sensitivity map | Expose value dependence |
| Revision trigger | Keep the decision reviewable |
Frequently asked questions
Is a loss function objective once numbers are assigned?
No. Calculation can be consistent, but outcome choice, weights, constraints and aggregation remain value-laden decisions.
Should stakeholder weights simply be averaged?
Not automatically. Averaging assumes comparability and an aggregation rule that must itself be justified.
What if stakeholders disagree strongly?
Show the disagreement, test decision robustness and use legitimate authority or deliberation rather than manufacturing consensus.
Can a decision still be legitimate if some people oppose it?
Yes, when the process is representative, transparent, accountable, reasoned and open to review. Legitimacy is not the same as unanimity.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Frontier selection under explicit loss functions
- Cross-criterion relaxation frontiers
- Discordant relaxations and decision robustness
- Near-minimum relaxations and stability