The Gold Standard of Experimental Design | Tipping-Point Analysis, Reference-Based Imputation and Sensitivity Thresholds

1. Start with the estimand and the primary missing-data assumption

Back to contents

Define the treatment condition, population, outcome, intercurrent-event strategy and summary measure before choosing an imputation model. State which values are truly missing and what the primary analysis assumes about them, such as missing at random conditional on observed history. A sensitivity analysis should address the same estimand; answering a different treatment question does not test robustness.

2. Understand what a tipping point means

Back to contents

Introduce a sensitivity parameter that shifts or otherwise changes the distribution of missing outcomes relative to the primary assumption. Refit the analysis over a prespecified grid and locate the value at which the confidence interval, p-value or decision criterion crosses its threshold. The tipping point is not an estimate of truth. It is the boundary between conclusions under a family of hypothetical assumptions.

3. Use reference-based scenarios as explicit stories

Back to contents

In longitudinal trials, jump-to-reference can assume that after discontinuing experimental treatment a participant follows the reference arm distribution; copy-reference applies the reference trajectory more broadly; copy-increments-in-reference uses reference-arm changes after discontinuation. Each scenario encodes a different statement about maintained treatment benefit. Name that statement rather than treating the method as a neutral software option.

4. Preserve imputation and sampling uncertainty

Back to contents

Generate multiple completed datasets under each scenario, run the planned analysis on each and combine estimates with a justified variance procedure. Do not replace every missing value with one deterministic number. Reference-based methods have particular information and variance properties, so implementation should be checked against simulations or validated software. The apparent smoothness of a tipping plot does not guarantee calibrated uncertainty.

5. Design a sensitivity grid that clinicians can interpret

Back to contents

Choose shifts, arm-specific assumptions and time patterns before unblinding where possible. Cover both favourable and unfavourable departures that are plausible for the clinical context, then extend far enough to find a decision boundary if one exists. Express a shift in outcome units as well as standard deviations and compare it with observed changes, measurement error and clinically important differences.

6. Worked example: pain scores after treatment discontinuation

Back to contents

A twelve-week trial finds a mean pain benefit of 1.2 points, but more participants discontinue the experimental arm. The primary model assumes missing at random. Jump-to-reference imputation reduces the benefit to 0.7 points and remains significant. A delta grid then worsens missing experimental outcomes by additional 0.1-point steps; the statistical conclusion tips at minus 0.8 and the clinical-importance criterion tips at minus 0.3. Reporting both boundaries shows that clinical robustness is weaker than significance robustness.

7. Judge plausibility with external and internal evidence

Back to contents

Compare the tipping scenario with outcomes observed before discontinuation, reasons for missingness, retrieved follow-up, reference-arm trajectories and subject-matter knowledge. Ask whether the required shift is common, extreme or impossible in the outcome scale. This appraisal should be multidisciplinary and recorded before the result is described as robust. A distant numerical threshold is reassuring only if the path to it is clinically meaningful.

8. Changed-context transfer check: a school intervention

Back to contents

A tutoring trial loses follow-up test scores when students change schools. The transferable method defines the same learning estimand, imputes under a primary model and varies missing scores until the conclusion changes. The changed context rules out copying a medical control trajectory blindly. A reference-based scenario must match school transitions, grade progression and test scaling, and cluster structure must remain in the imputation and analysis.

9. Know the limits and common misconceptions

Back to contents

No sensitivity analysis proves the missing-data mechanism. A wide grid can look thorough while omitting the most relevant time pattern or arm asymmetry. Reference-based imputation is not automatically conservative, and different scenarios can be more or less severe for different estimands. A result that never tips within the scale may be structurally robust, but coding errors, model misspecification and missing covariates can still matter.

10. Audit the full sensitivity path

Back to contents

Publish the estimand, missingness by arm and time, reasons, retrieved outcomes, primary imputation model, reference scenario, delta definition, grid, random seeds, number of imputations, analysis code, variance method and every decision boundary. Plot estimates and intervals across the grid, not only the first unfavourable result. Reproduce the primary estimate at the zero or reference setting before interpreting departures.

11. Frequently asked questions

Back to contents

Does a tipping point reveal the true missing values? No. Is jump-to-reference always worst case? No. Should sensitivity analyses target the same estimand? Yes. Can statistical and clinical tipping points differ? Yes, because their decision thresholds differ. How wide should the grid be? Wide enough to include plausible departures and expose the decision boundary, with units readers can interpret.

12. The gold standard turns hidden assumptions into visible decisions

Back to contents

A credible tipping-point analysis begins with a clear estimand, gives each reference-based scenario a substantive meaning and propagates uncertainty through the planned analysis. It reports where statistical and clinical conclusions change and asks whether those assumptions are plausible. The goal is not to manufacture a robust-looking graph; it is to show exactly what must be believed for the claim to hold.