The Gold Standard · Experimental Design
Make the decision path visible
A preference study may weight each respondent so the sample better represents the population or decision constituency. That correction matters when response differs across groups. Yet very large weights can make an apparently large sample behave like a much smaller one, with one or two answers carrying an unstable share of the decision.
Trimming can reduce variance, but it also changes representation and can reintroduce bias. The gold-standard question is not how to make weights look tidy. It is whether the decision remains legitimate and robust across defensible weighting choices while sparse groups are described honestly rather than averaged out of sight.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · UNDERSTAND THE WEIGHTS
1. Define the population and estimand
A weight only has meaning relative to a target population, time and quantity being estimated. State whether the study represents all affected people, service users, eligible participants or a deliberative panel.
Define the preference estimand: a mean rating, choice share, threshold distribution or expected loss. The same weights can behave differently for smooth averages and rare subgroup decisions.
Do not use population weights to answer a governance question that assigns roles or rights rather than proportional representation. Some voices may require constraints or separate reporting instead.
CHAPTER 2 OF 12 · UNDERSTAND THE WEIGHTS
2. Decompose how each weight arose
Record base selection weights, nonresponse adjustment, calibration or raking factors and any final trimming. A final number alone cannot reveal which representation problem it was meant to solve.
Inspect weight distributions overall and within important groups. A moderate overall range can hide one sparse group whose entire representation rests on a single respondent.
Check whether adjustment variables predict both response and preferences. Calibrating to convenient demographics does not guarantee correction of preference nonresponse.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · UNDERSTAND THE WEIGHTS
3. Measure concentration and effective sample size
Report the nominal sample beside a weight-based effective sample size or design effect appropriate to the estimator. The calculation is a warning about information concentration, not a new count of people actually interviewed.
Inspect maximum weight share, sums by group and leave-one-out influence. Two weighting schemes with the same effective size can concentrate risk in different respondents.
Use replicate-weight or design-aware variance methods where the sample design requires them. A simple weighted spreadsheet standard error can be misleading.
| Component | Purpose |
|---|---|
| Base weight | Reflect selection probability |
| Nonresponse adjustment | Address differential response |
| Calibration | Align known totals |
| Trim and redistribution | Bound influence and restore totals |
CHAPTER 4 OF 12 · DESIGN TRIMMING
4. Choose trimming objectives before results
Possible objectives are limiting maximum influence, reducing mean-squared error, stabilising subgroup estimates or meeting operational decision tolerances. State the objective before seeing which trim favours a preferred policy.
Define whether caps are absolute, percentile-based or relative to group medians. A common cap is easy to explain but may treat legitimately unequal selection probabilities as errors.
Pre-specify exceptions and sensitivity ranges. Quietly changing the cap until the conclusion looks stable is outcome-driven analysis.
CHAPTER 5 OF 12 · DESIGN TRIMMING
5. Reallocate or recalibrate trimmed mass
Capping weights reduces the represented total unless the removed mass is redistributed or weights are recalibrated. State the method and the control totals retained.
Redistribute within appropriate classes when evidence supports exchangeability. Spreading a sparse group's mass across unlike respondents can erase the very difference the original weight tried to restore.
After recalibration, check new extremes and balance. A second adjustment can recreate the concentration trimming was intended to reduce.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · DESIGN TRIMMING
6. Work a sparse-representation case
Suppose 400 people answer a preference survey. Twenty respondents from a low-response group receive weights averaging 8, and one receives 28. The weighted effective sample size falls to 176, and the policy choice changes when that one response is omitted.
The team tests caps of 12, 16 and 20, recalibrates within population groups and reports the untrimmed analysis. The preferred action is stable at caps 12 and 16 but reverses untrimmed and at 20, so the result is labelled weight-sensitive rather than definitive.
Changed-context transfer check: if the highly weighted respondent represents a legally protected perspective that cannot be traded away, trimming is not the right governance remedy. The study should report that perspective separately and use an explicit constraint or deliberative rule.
| Measure | Question |
|---|---|
| Maximum share | Can one response dominate? |
| Effective sample size | How much information remains? |
| Group respondent count | Who is sparse? |
| Leave-one-out change | Is the decision fragile? |
CHAPTER 7 OF 12 · TEST THE DECISION
7. Distinguish the weighted estimands
Trimming changes the estimator, even if the variable name stays the same. Report whether the result targets the original population distribution, a stabilised approximation or a bounded-influence decision rule.
Compare weighted population summaries with subgroup-specific results. A stable overall mean can coexist with unstable decisions for a sparse community.
Keep weights out of outcome definitions. Representation adjustment and preference scoring are separate layers and should remain auditable.
CHAPTER 8 OF 12 · TEST THE DECISION
8. Carry uncertainty from weighting choices
Variance should reflect sampling and the chosen adjustment procedure where possible. Sensitivity across plausible nonresponse models and trim rules addresses model uncertainty that one standard error does not.
Use intervals and decision boundaries, not false precision in final weights. A respondent weight of 7.843 is not known to three decimals simply because software prints it.
When control totals are estimated or dated differently from the preference survey, include that mismatch in the limitations.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · TEST THE DECISION
9. Stress-test the final decision
Run the decision under unweighted, base-weighted, fully adjusted and several pre-specified trimmed schemes. Compare actions, losses and which groups bear regret.
Use leave-one-out or bounded-influence checks for extreme respondents, but do not delete inconvenient people. The purpose is to reveal dependence, not manufacture stability.
A robust decision remains acceptable across credible specifications. A fragile decision needs more data, narrower scope or an accountable governance choice rather than a single winning model.
| Specification | Role |
|---|---|
| Unweighted | Show raw respondent pattern |
| Base weighted | Correct selection |
| Fully adjusted | Address response and controls |
| Trimmed variants | Test bounded influence |
CHAPTER 10 OF 12 · PROTECT REPRESENTATION
10. Keep sparse groups visible
Report respondent counts, weighted population shares and uncertainty for important groups. Large weighted totals must not disguise that direct evidence came from very few people.
Protect confidentiality when groups are small. Use ranges, aggregation or controlled access without pretending the evidence is more complete than it is.
Combine quantitative weights with qualitative reasons where appropriate. Sparse numbers may still reveal harms or constraints that an average preference score cannot represent.
CHAPTER 11 OF 12 · PROTECT REPRESENTATION
11. Repair the next study upstream
Extreme weights often signal recruitment, frame or accessibility problems. Improve contact modes, language support, oversampling, follow-up and response monitoring before relying on larger corrections.
Set live triggers for low response and weight concentration during data collection. Early action can recover representation while responses are still possible.
Reserve budget for the hard-to-reach groups named in the design. A representative target without an operational inclusion plan is only an aspiration.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · PROTECT REPRESENTATION
12. Know when weights cannot carry the decision
Do not make a population claim when key groups have no respondents, adjustment variables do not explain missing preferences or the decision changes across every defensible trimming rule.
Alternatives include collecting more evidence, reporting a set of actions, separating subgroup decisions or using explicit rights and deliberative constraints.
The final assurance record should connect the target population, response pattern, weight construction, effective information, trimming, sensitivity and decision authority. Representation is not repaired by one number alone.
| Record | Why it matters |
|---|---|
| Target population | Defines representation |
| Trim rule | Makes judgement reproducible |
| Effective information | Prevents nominal-sample illusion |
| Robustness result | Shows dependence on weights |
Frequently asked questions
Does a large nominal sample guarantee stable weighted results?
No. Unequal weights can concentrate information so the effective sample size is much smaller than the respondent count.
Is trimming always better than keeping extreme weights?
No. It trades variance reduction against bias and representation; the rule must be justified and tested.
Should trimmed weight be discarded?
No. Reallocation or recalibration is usually needed, with the method and retained totals reported.
When is the decision too fragile to use?
When reasonable weight construction or trimming choices change the action materially and the evidence cannot resolve which specification is defensible.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Preference-model nonresponse
- Stakeholder-preference drift
- Loss-function elicitation disagreement
- Frontier selection under explicit loss functions