The Gold Standard · Experimental Design
Make the decision path visible
Preference weights repair unequal representation only under assumptions about who is missing, how represented cases stand for them and which target totals matter. Trimming can prevent a few records from dominating, but it also changes the estimand. A single final weighted answer hides those choices.
The gold standard makes them visible with sensitivity maps, influence bounds and decision-flip thresholds. The aim is not to search until a preferred answer appears. It is to show how far the conclusion travels across defensible weighting schemes.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · FRAME THE WEIGHTS
1. State the target population and estimand
Weights only make sense relative to a target: current participants, all eligible people, a future service population or a stakeholder composition. Write the estimand before constructing weights.
A weighted mean, weighted risk difference and weighted loss can answer different questions. State the action selected and the threshold that turns the estimate into a decision.
CHAPTER 2 OF 12 · FRAME THE WEIGHTS
2. Decompose the weight
Record base selection weight, nonresponse adjustment, calibration factor, preference intensity and any trimming or normalisation. Multiplying them into one number hides where influence entered.
Define which components are evidence-based and which express values. A response-propensity adjustment and a stakeholder importance weight should not be defended as though they were the same kind of quantity.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · FRAME THE WEIGHTS
3. Check distribution and effective information
Inspect minimum, maximum, percentiles, concentration, sum of squares and effective sample size. A large nominal sample can carry the information of a much smaller balanced sample when weights are concentrated.
Examine these diagnostics by outcome and subgroup. A modest global maximum can still dominate a rare outcome or one side of a close comparison.
| Component | Question |
|---|---|
| Base | Who could be selected? |
| Nonresponse | Who is represented? |
| Calibration | Which totals are matched? |
| Preference | Whose values count how much? |
CHAPTER 4 OF 12 · MEASURE INFLUENCE
4. Measure unit and group influence
Recompute the estimate after deleting or capping one unit, cluster or subgroup. For smooth estimators, analytic influence approximations can speed the scan; verify the largest cases with exact recomputation.
Influence is not evidence that a respondent is wrong. It shows that the conclusion depends heavily on how that case represents others. Investigate the weight construction and target coverage before excluding anyone.
CHAPTER 5 OF 12 · MEASURE INFLUENCE
5. Define plausible perturbations before seeing results
Specify ranges for response adjustments, calibration margins, preference intensities and trim caps using design evidence and stakeholder governance. Include alternative models that a reasonable critic would accept.
Predefining the grid prevents sensitivity analysis from becoming a search for a desired answer. Keep infeasible combinations out and explain why.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · MEASURE INFLUENCE
6. Worked example: a funding choice flips
Plan A has outcomes 72, 68 and 55 for three stakeholder groups; Plan B has 67, 70 and 63. Baseline group weights of 0.50, 0.30 and 0.20 give scores of 67.4 for A and 67.7 for B, so B narrowly wins.
Let the third group weight be x while the first two retain their 5:3 ratio. Solving the weighted score difference shows the decision flips near x = 0.184. The baseline is only 0.016 above that boundary. Changed-context check: if the third group is sparsely observed and its score interval is wide, mapping only its weight is insufficient; vary both weight and outcome uncertainty.
| Measure | Meaning |
|---|---|
| Maximum weight | Largest representation |
| Effective sample size | Information after concentration |
| Delete-one change | Unit leverage |
| Delete-group change | Group dependence |
CHAPTER 7 OF 12 · MAP SENSITIVITY
7. Build a one- and two-way sensitivity map
Start with a curve showing the decision metric across one weight or trim cap. Then map two influential assumptions jointly, colouring regions where each action is preferred and marking the baseline.
Do not imply equal plausibility across the whole rectangle. Overlay the empirically or normatively defensible region and label areas excluded by calibration constraints.
CHAPTER 8 OF 12 · MAP SENSITIVITY
8. Find decision-flip thresholds
For each assumption, solve or search for the nearest value at which the chosen action changes, a constraint is violated or the sign crosses zero. Report distance in understandable units.
A far threshold supports robustness only for the dimensions tested. Interactions may create a closer diagonal path, so compute joint or worst-case thresholds when assumptions can move together.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · MAP SENSITIVITY
9. Use influence bounds and uncertainty sets
When exact weights are disputed, define a set: bounds on individual weights, calibration tolerance, total variation or divergence from a baseline scheme. Optimise the decision metric over that set to obtain best- and worst-case values.
Bounds can be conservative and depend on the chosen geometry. State why the set reflects plausible representation error, not simply a mathematically convenient shape.
| Layer | Show |
|---|---|
| Baseline | Chosen specification |
| Plausible region | Defensible assumptions |
| Flip boundary | Where action changes |
| Uncertainty | Outcome range |
CHAPTER 10 OF 12 · DECIDE AND REPORT
10. Separate statistical uncertainty from weight uncertainty
Sampling variance assumes a weighting procedure; sensitivity analysis varies that procedure or its inputs. Present both. A narrow confidence interval under one weight scheme does not make the scheme certain.
Use replicate weights or design-consistent methods where appropriate, then repeat across meaningful specifications. Avoid treating dozens of sensitivity runs as independent evidence.
CHAPTER 11 OF 12 · DECIDE AND REPORT
11. Respond to a fragile decision
If the decision flips under a small plausible change, options include gathering data from the influential group, improving nonresponse modelling, choosing a reversible pilot, applying minimax regret or deferring the irreversible action.
Do not average across arbitrary specifications merely to create stability. The response should address the source of fragility or govern it explicitly.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · DECIDE AND REPORT
12. Publish the map, not only the selected point
Report target population, estimand, weight components, diagnostics, perturbation ranges, influence cases, flip thresholds and final authority. Provide code or a reproducible table where confidentiality permits.
The clearest conclusion may be conditional: Plan B is preferred across most calibrated schemes, but Plan A becomes preferred if the underrepresented group receives more than a stated share. Conditional honesty is more useful than false certainty.
| Finding | Next move |
|---|---|
| Far boundary | Proceed with monitoring |
| Near boundary | Gather targeted evidence |
| Sparse influential group | Improve representation |
| Irreversible split | Use robust or regret rule |
Frequently asked questions
Is the largest weight always a problem?
No. It may correctly represent rare undercoverage. The question is whether the construction is defensible and whether the decision is overly dependent on it.
Does trimming make an estimate more accurate?
It can reduce variance and extreme influence, but may introduce bias or change the target. Compare schemes and disclose the trade-off.
What is a decision-flip threshold?
It is the nearest assumption value or combination at which the preferred action changes or a decision criterion crosses its boundary.
Should sensitivity analysis include every imaginable weight?
No. Use ranges grounded in design evidence, calibration feasibility and legitimate value choices, while testing reasonable alternatives.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Preference-weight trimming and effective sample size
- Preference-model nonresponse and representation weights
- Stakeholder preference drift and repeat elicitation
- Loss-function elicitation and stakeholder heterogeneity