The Gold Standard · How X Works
Randomisation protects assignment, not every later comparison
An experiment can begin with comparable groups and end with a biased observed sample. When outcomes are missing after assignment, the reason and pattern matter. This guide shows how to preserve the target question, expose selection and test how much conclusions depend on unverifiable assumptions.
Choose a reading route
- Start from first principles — define the unit before interpreting a number.
- Follow the worked example — see the method under controlled conditions.
- Test transfer — change the context and check what still holds.
- Audit evidence — inspect records, assumptions and failure points.
- Use the practical checklist — apply the idea to new work.
Chapter contents
Foundations: Chapters 1–3
Method: Chapters 4–6
Transfer and limits: Chapters 7–9
Connections and use: Chapters 10–12
1. Assignment and observation are different events
Random assignment controls how units enter treatment groups. It does not guarantee that every assigned unit receives treatment, remains in the study or supplies an outcome. Attrition is loss from observation or follow-up; a missing outcome is the absence of a required measurement. They overlap, but they are not identical.
A participant can discontinue an intervention yet still provide an outcome. Another can remain enrolled but miss the primary measurement. The design and analysis must keep assignment, treatment received, follow-up and outcome observation as separate variables.
First principle
Preserve the assigned groups and account for every randomised unit before analysing only those with observed outcomes.
2. How post-randomisation selection appears
Complete-case analysis conditions on being observed after assignment. If treatment, prognosis or their consequences affect observation, the observed groups may no longer be comparable. The randomisation happened upstream; selection happened later.
For example, a demanding programme may cause struggling participants to skip the final test. Comparing only completers can make the programme look unusually effective because the lowest potential outcomes are missing from the treatment group. The reverse pattern is also possible.
| Stage | Record | Why it matters |
|---|---|---|
| Assignment | Original group | Defines the randomised comparison |
| Exposure | What was received | Describes adherence and crossover |
| Follow-up | Contact status and reason | Shows attrition process |
| Outcome | Observed, missing, timing | Defines the analysis data |
← Previous · Contents · Next →
3. Start with the estimand
Before choosing an imputation method, state the treatment effect being estimated: population, treatment conditions, outcome, summary and how post-assignment events are handled. International Council for Harmonisation guidance calls this target an estimand.
Treatment discontinuation, rescue treatment or switching can be intercurrent events that alter interpretation. Loss to follow-up and missing measurements are limitations in the data needed for the estimand. Conflating them can make the analysis answer a different question from the protocol.
- Name the target population.
- Keep the assigned treatment condition visible.
- Specify the outcome and time point.
- Define strategies for intercurrent events.
- Identify which required data may be missing.
← Previous · Contents · Next →
4. Worked example: bounded outcomes
A fictional trial randomises 50 units to treatment and 50 to control. The outcome ranges from 0 to 20. Treatment has 40 observed outcomes with mean 12; control has 48 observed outcomes with mean 10. The complete-case difference is +2 points.
Treatment’s observed total is 40 × 12 = 480. If the ten missing treatment outcomes could each be anywhere from 0 to 20, the full-group mean ranges from 480 ÷ 50 = 9.6 to (480 + 200) ÷ 50 = 13.6. Control’s observed total is also 480; with two missing outcomes, its full-group mean ranges from 9.6 to 10.4.
| Quantity | Lower bound | Upper bound |
|---|---|---|
| Treatment mean | 9.6 | 13.6 |
| Control mean | 9.6 | 10.4 |
| Treatment − control | −0.8 | +4.0 |
The observed +2 is compatible with a wide range under minimal assumptions. Bounds are often broad, but they make the information limit visible.
← Previous · Contents · Next →
5. Describe missingness before modelling it
Report missing counts and percentages by assigned group, time point and reason. Plot follow-up over time. Compare measured baseline variables for observed and missing units without treating a non-significant test as proof that missingness is harmless.
Reasons are evidence, not a complete statistical mechanism. “Moved away” may relate to treatment burden or outcome. “Unknown” should remain unknown. Avoid rewriting messy histories into reassuring categories.
| Description | Useful question | Caution |
|---|---|---|
| Differential rate | Do groups lose different proportions? | Equal rates can still be selective |
| Reason pattern | Are causes plausibly outcome-related? | Reasons may be incomplete |
| Timing | When does loss occur? | Late loss can still bias the endpoint |
← Previous · Contents · Next →
6. Analysis needs assumptions
Methods such as likelihood models, multiple imputation and weighting depend on assumptions about the unseen outcomes and observation process. The choice must align with the estimand, design and available predictors. No algorithm can recover information that the study never collected without assumptions.
Pre-specify a main analysis and sensitivity analyses that alter the most consequential missing-data assumptions while targeting the same estimand. Show the direction and size of changes, not just whether a threshold is crossed.
A practical sequence
Prevent missingness, continue outcome collection after intervention changes where ethical and feasible, document the process, analyse according to a stated target, then stress-test the assumptions.
← Previous · Contents · Next →
7. Transfer test: a tutoring programme
Move the design to a school programme. Eighty students are randomised to new tutoring or usual support. Some assigned students never attend; others attend but miss the final assessment. Excluding non-attenders estimates neither a clean randomised effect nor necessarily the effect of attendance, because attendance is selected after assignment.
For the assignment effect, retain students in their assigned groups and pursue final outcomes regardless of attendance when feasible. If the question concerns actual participation, use an analysis designed for that causal question rather than relabelling a completer comparison.
Transfer passes when the target effect remains explicit as behaviour changes after randomisation.
← Previous · Contents · Next →
8. Misconceptions and limits
- “Randomisation prevents attrition bias.” It protects assignment, not selective observation.
- “Intention-to-treat means inventing outcomes.” It defines comparison by assignment; missing outcomes still require a principled method.
- “Similar attrition percentages mean no bias.” Selection can differ even when rates match.
- “Imputation fills in the truth.” It produces values under a model and its assumptions.
- “Per-protocol is the treatment effect among compliers.” Simple restriction usually creates post-randomisation selection.
Extreme bounds can be uninformative, and more plausible sensitivity models still require judgement. The honest conclusion may be that the study cannot distinguish benefit from no benefit under reasonable departures from the main assumption.
← Previous · Contents · Next →
9. Evidence audit
Build a participant or unit flow from eligibility through assignment, intervention, follow-up and analysis. Reconcile every count. Check whether outcome collection stopped after treatment discontinuation, whether reasons were recorded before outcomes were known, and whether the analysis population matches the estimand.
| Audit question | Evidence | Warning sign |
|---|---|---|
| Are all assigned units traceable? | Flow diagram and ID log | Denominators change without explanation |
| Was follow-up independent of group? | Contact protocol | Extra effort only for one arm |
| Are assumptions explicit? | Analysis plan | Method named without rationale |
| Was robustness tested? | Sensitivity analyses | Only complete cases reported |
A sensitivity analysis is useful when it changes a key assumption while holding the scientific question steady.
← Previous · Contents · Next →
10. Questions readers often ask
Is attrition the same as non-adherence?
No. Non-adherence concerns treatment received; attrition concerns observation or participation in follow-up. A non-adherent unit can still provide valuable outcomes.
Can baseline balance be checked only among completers?
That describes selection but does not restore randomisation. The primary assigned-group balance comes from all randomised units.
When is complete-case analysis reasonable?
Only under assumptions that make selection ignorable for the target analysis. Those assumptions should be argued and tested where possible, not presumed.
← Previous · Contents · Next →
11. Reading routes and ecosystem connections
Start with the Gold Standard of Experimental Design. Ground the logic in the Scientific Method, develop estimation through Biostatistics, and use How Research Bias Works to locate selection among other threats.
The companion guide on treatment fidelity, protocol deviations and the effect being estimated separates delivery from the target effect. Continue through learning that transfers into life, The Gold Standard and the How X Works hub.
← Previous · Contents · Next →
12. Sources and practical checklist
Authoritative guidance includes the International Council for Harmonisation’s E9(R1) addendum on estimands and sensitivity analysis, its E9(R1) training materials, and E8(R1) General Considerations for Clinical Studies. The principles extend beyond clinical trials: define the question, preserve assignment and make assumptions visible.
- Define the estimand before analysis.
- Track all assigned units.
- Separate adherence, follow-up and outcome observation.
- Report missingness by group, time and reason.
- Prevent loss where feasible.
- Use a justified main method.
- Stress-test key assumptions and state the information limit.