The Gold Standard · Experimental Design
Make the decision path visible
A loss model records how outcomes are valued for a decision, but those values are not frozen. People learn from experience, new burdens appear, treatment or service options change and the affected population may shift. A model that was carefully elicited can therefore become a poor account of current priorities.
Repeat elicitation is not a ritual survey on a calendar. It is a measurement and governance problem: distinguish genuine preference change from framing, sample turnover and instrument drift; decide whose current views carry authority; and update the decision without rewriting why the earlier choice was made.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · DEFINE PREFERENCE DRIFT
1. State what may have changed
Identify the outcome definitions, scales, stakeholder population, elicitation instrument, aggregation rule and decision authority in the current loss model. Preference drift can enter through any of them.
Separate preference change from evidence change. If the probability of harm rises but stakeholders value that harm the same way, update the evidence model. If the consequence is valued differently after lived experience, the value model may need revision.
Do not call every changed choice a changed preference. The available options, information, reference point or constraints may have changed while underlying values remained stable.
CHAPTER 2 OF 12 · DEFINE PREFERENCE DRIFT
2. Name plausible mechanisms
Preferences can move through learning, adaptation, fatigue, changing responsibilities, new alternatives, experienced harms or public deliberation. Population composition can also shift as different people become affected.
Instrument drift is another mechanism: altered wording, order, visual display, interviewer support or survey mode can move answers without a real value change.
Write competing explanations before collecting the repeat wave. This makes the design testable and reduces the temptation to label any difference as meaningful drift.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · DEFINE PREFERENCE DRIFT
3. Pre-specify triggers and authority
Triggers may include a new outcome, sustained burden outside the original range, changed population, legal or policy change, weak decision robustness or a scheduled review after enough experience accumulates.
Clarify who may revise the loss model and who must be consulted. Repeat elicitation can inform authority; it does not automatically transfer it to the newest sample.
Define interim policy. A scheduled update may keep the current rule in force, while evidence of serious unrepresented harm may justify temporary constraints.
| Observed change | Possible explanation |
|---|---|
| Same people, same task | Learning or lived experience |
| New sample only | Population composition |
| New wording or mode | Instrument effect |
| New options or evidence | Decision context |
CHAPTER 4 OF 12 · REPEAT ELICITATION WELL
4. Keep a bridge between elicitation waves
Retain a core set of unchanged tasks, definitions and presentations so differences can be compared. Add new items in a separate module and randomise order where learning or fatigue is plausible.
Use the same units, denominators and time horizons for bridge items. If an old question was misleading, do not preserve it merely for continuity; instead run a documented bridge that compares old and improved versions.
Archive instruments, scripts, visual aids, translations, mode and interviewer training. Reproducibility requires more than a spreadsheet of final weights.
CHAPTER 5 OF 12 · REPEAT ELICITATION WELL
5. Separate panel change from preference change
Where feasible, recontact some original participants and recruit a fresh sample from the current affected population. The longitudinal panel estimates within-person movement; the repeated cross-section captures population change.
Attrition is informative. People who experienced the greatest burden may be less willing or able to return. Compare retained, lost and newly recruited participants using ethical, privacy-preserving measures.
Weighting can address measured differences but cannot recover unobserved perspectives automatically. Report sensitivity to plausible nonresponse patterns.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · REPEAT ELICITATION WELL
6. Work a repeat-elicitation case
A monitoring programme originally elicited losses from participants, operators and community representatives. The intensive option was chosen because respondents treated one prevented severe failure as equivalent to 25 burdensome reviews.
Two years later, repeat elicitation keeps half the original panel and adds a current cross-section. The median threshold falls to 12 reviews after people experience long suspensions during review. Bridge questions show little wording effect, and the change is strongest among directly reviewed participants. Under the revised distribution, the intensive plan is no longer robust unless review duration is reduced.
Changed-context transfer check: if the review process has since become same-day and reversible, the observed historical burden should not be copied into the new loss model. The update must value the current consequence while retaining the old evidence as an explanation of why preferences changed.
| Component | Purpose |
|---|---|
| Bridge items | Compare waves |
| Longitudinal panel | Estimate within-person change |
| Fresh cross-section | Represent current population |
| New module | Capture emerging outcomes |
CHAPTER 7 OF 12 · UPDATE THE DECISION
7. Model change with uncertainty
Estimate within-person change, population-composition change and instrument effects separately where the design permits. Use intervals and distributions rather than declaring a precise new weight.
Test measurement invariance: do respondents interpret outcomes and scales similarly across waves? A numerical difference is difficult to interpret when the construct changed.
Report stable as well as changed elements. A model can update one burden while retaining values for severe harms and reversibility.
CHAPTER 8 OF 12 · UPDATE THE DECISION
8. Preserve heterogeneity and minority burdens
Show preference distributions and subgroup patterns supported by the design. A stable average can hide opposing movements, while a moving average can arise from sample turnover.
Do not give every subgroup its own rule automatically. Examine whether different exposure, rights or operating contexts justify constraints, tailored decisions or representation in deliberation.
Protect small groups from identification and false precision. Qualitative evidence may be more truthful than a narrow estimate from a handful of respondents.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · UPDATE THE DECISION
9. Update the loss model transparently
Version the outcomes, weights or ranges, constraints, aggregation rule, stakeholder sample and effective date. Preserve the prior model and the decisions made under it.
Run old and proposed loss models against the same current evidence. Show which actions change, which value assumptions drive the change and whether the choice is robust across uncertainty.
Apply updates prospectively unless retrospective review is justified. If earlier decisions caused continuing harm under the revised model, specify a separate remediation policy rather than pretending the old rule never existed.
| Check | Question |
|---|---|
| Measurement | Are constructs comparable? |
| Representation | Whose values are missing? |
| Robustness | Does the action change across plausible losses? |
| Authority | Who may adopt the revision? |
CHAPTER 10 OF 12 · GOVERN FUTURE REVISIONS
10. Use deliberation to interpret change
Bring evidence of preference movement, lived consequences and sampling limits to an accountable forum. Participants should be able to question whether the elicitation tasks represent the real decision.
Distinguish learning from pressure. Preferences may reasonably change after discussion, but record pre- and post-deliberation results and how facilitation handled power differences.
Explain the final update in terms of evidence, values and authority. A vote total or fitted coefficient alone does not make the revision legitimate.
CHAPTER 11 OF 12 · GOVERN FUTURE REVISIONS
11. Monitor the consequences of the updated rule
Pre-specify indicators such as burden, severe events, distribution across groups, appeals and capacity displacement. Compare observed outcomes with the scenarios used during elicitation.
Treat monitoring as evidence, not automatic preference. More appeals may reflect easier access; fewer may reflect exclusion. Combine counts with direct feedback and process checks.
Set the next review trigger based on decision risk and rate of contextual change. Constant re-elicitation can create fatigue and unstable policy, while indefinite delay preserves outdated assumptions.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · GOVERN FUTURE REVISIONS
12. Know when one updated loss model is insufficient
Do not collapse to one scalar model when groups face qualitatively different rights, the elicitation lacks representation, outcomes are poorly understood or reasonable aggregation rules select incompatible actions.
Use a set of admissible models, rights constraints, subgroup pathways, minimax regret or additional evidence where appropriate. The aim is not to force consensus but to keep disagreement visible and governable.
The final record should show what changed, what remained stable, why the revised model has authority, which decisions it changes and how future challenge will be heard.
| Field | Preserve |
|---|---|
| Outcome definitions | Old and new meanings |
| Elicitation | Instrument, sample and mode |
| Aggregation | Weights, constraints and rationale |
| Decision | Effective date and affected actions |
Frequently asked questions
How often should preferences be re-elicited?
Use risk-based triggers and enough time for meaningful experience. A fixed calendar can help governance but does not replace substantive triggers.
Does a different sample mean preferences drifted?
Not necessarily. Separate population composition, nonresponse and instrument changes from within-person preference movement.
Should the latest wave always replace the old model?
No. Evaluate comparability, representation, authority and decision consequences; preserve both versions and explain the update.
What if groups move in opposite directions?
Report the heterogeneity and test governance options or constraints. An average may conceal the central disagreement.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Loss-function elicitation and legitimacy
- Frontier selection under explicit loss functions
- Cross-criterion relaxation frontiers
- Discordant relaxations and decision robustness