The Gold Standard of Experimental Design | Sensitivity Amplification, Confounder Scenarios and Substantive Gamma Calibration

1. Start with the matched design and the causal contrast

Back to contents

Sensitivity analysis does not repair a poorly defined observational study. State the treatment, outcome, matched-set construction, eligible population, follow-up period and causal contrast before introducing hidden bias. Show balance on measured pretreatment covariates and identify residual differences. In a matched pair, two people can look similar on recorded covariates while differing on an unmeasured factor that affects both treatment and outcome. The sensitivity model asks how strong that hidden assignment difference could be and whether inference would survive it. Gamma concerns treatment odds within matched sets under a specified model; it is not the probability that the study is biased, the size of an omitted variable or a universal quality score. The test statistic and outcome scale remain part of the design.

2. Interpret Gamma narrowly before amplifying it

Back to contents

At Gamma equal to one, the sensitivity model corresponds to no hidden difference in treatment odds within a matched set. Larger values allow a greater odds ratio between otherwise matched units because of an unobserved covariate. A result remaining significant to a larger Gamma is less sensitive under that model, but the number has no context-free label such as strong or weak. Treatment prevalence, matching quality, effect size, outcome distribution and test statistic can change the threshold. Report the upper and lower sensitivity bounds or the relevant sensitivity value, the null hypothesis and the inference criterion. Do not say that an unmeasured confounder must increase both treatment and outcome by Gamma; that conflates the assignment bound with a two-path confounding story.

3. Use amplification to separate the two confounding paths

Back to contents

Amplification expresses a one-parameter hidden-bias result as combinations of an unmeasured factor’s association with treatment assignment and its association with the outcome. Many combinations can correspond to the same Gamma threshold. A factor strongly related to treatment but barely related to outcome may create limited bias; the reverse can also be true. The exact mapping depends on the chosen amplification framework, outcome and model. Plot or tabulate several combinations rather than advertising one dramatic pair. Label the scales, such as odds ratios or outcome shifts, and keep direction visible. Amplification does not identify the omitted variable. It turns an abstract robustness boundary into a family of scenarios that subject-matter experts can assess.

4. Construct confounder scenarios from the data-generating story

Back to contents

Name plausible omitted factors before seeing which scenario would overturn the result. For each one, specify how it could vary within matched sets, affect treatment uptake, affect the outcome and interact with measured covariates. Consider binary, ordinal and continuous forms rather than forcing everything into presence or absence. A scenario should respect timing: a variable measured after treatment cannot be used as a pretreatment confounder without a different causal model. Also consider protective or opposing confounding, not only the direction that explains away the effect. Several moderate factors may matter jointly. The purpose is not to prove that no confounder exists; it is to connect the sensitivity frontier with credible mechanisms and observable implications.

5. Calibrate against observed covariates carefully

Back to contents

Observed covariates provide familiar benchmarks: how strongly did age, prior severity, school type or baseline achievement relate to treatment and outcome before matching? Re-estimating the study while withholding one measured covariate at a time can show how an omitted variable of known meaning would affect balance and inference. Model-based calibration can compare sensitivity parameters with fitted associations. These benchmarks are informative, not upper bounds. An unmeasured factor may be stronger than every recorded variable, and measured covariates may be noisy proxies. Correlated benchmarks cannot simply be added. Report the scale and model used, whether associations are marginal or conditional and how matching changed them. Calibration makes Gamma interpretable only to the extent that the observed covariates illuminate the plausible hidden ones.

6. Worked example: a matched job-training study

Back to contents

A fictional study matches 180 programme participants to 180 nonparticipants on age, prior earnings, education and employment history. The rank-based analysis finds higher twelve-month earnings and remains statistically distinguishable from the null until a sensitivity value of Gamma 1.6 under the prespecified model. The team does not conclude that bias is unlikely. It amplifies the boundary into several treatment-and-outcome association pairs and compares them with prior motivation, which was measured imperfectly by application records. Leaving out prior earnings creates a benchmark smaller than the boundary; leaving out employment stability creates one closer to it. A scenario in which motivation strongly affects enrolment and shifts earnings within pairs could still overturn the result. The report presents the curve and mechanism rather than declaring Gamma 1.6 robust by convention. The numbers are invented teaching data.

7. Match calibration to the outcome and test statistic

Back to contents

A confounder scenario that threatens a binary employment outcome may not threaten a continuous earnings outcome in the same way. Sensitivity values can also differ across sign tests, signed-rank tests, M-estimates or effect-ratio statistics because the tests weight pair differences differently. Prespecify the primary analysis and treat alternative robust statistics as design sensitivity checks, not a search for the largest Gamma. If several outcomes are tested, multiplicity and shared assignment bias need joint attention. Report whether calibration concerns statistical significance, effect attenuation, confidence-interval inclusion or a policy-relevant effect threshold. A study can remain statistically significant while the plausible effect shrinks below substantive importance.

8. Changed-context transfer check: tutoring and school selection

Back to contents

Suppose students who receive an intensive tutoring programme are matched to similar students using prior scores, attendance and demographics. Unmeasured family organisation or teacher encouragement could affect both programme uptake and later attainment. The transferable method is to state the matched contrast, compute the sensitivity frontier, amplify it into selection and outcome associations and benchmark against observed predictors. The changed context introduces school-level clustering, interference among classmates and possible measurement error in prior scores. A Gamma analysis for individual matched pairs does not automatically cover school-level hidden bias or spillovers. The scenario must align with the actual assignment process and outcome scale, and conclusions should remain about the matched population rather than every student.

9. Know the limits and common misconceptions

Back to contents

Gamma is not the strength of the treatment effect, the probability of confounding or proof that an omitted variable exists. A large threshold does not establish causality, especially when selection bias, outcome measurement, interference, attrition or model misspecification lies outside the sensitivity model. Amplification is not unique; different parameterisations can describe the same boundary. Benchmarking against observed covariates cannot prove that hidden factors are weaker. Poor overlap and bad matches should be fixed in design, not celebrated through sensitivity calculations. Statistical significance is not substantive importance. The analysis is conditional on the null, test statistic, assignment model and treatment definition, and results should be read alongside negative controls and other design evidence.

10. Audit the robustness argument from match to mechanism

Back to contents

Archive the causal diagram or assignment story, eligibility rules, matching variables and algorithm, distance and caliper settings, discarded units, balance diagnostics, outcome definitions, test statistic, sensitivity grid, software version, amplification formula, calibration models, benchmark covariates and all scenario assumptions. Preserve both favourable and unfavourable directions and any multiplicity adjustment. Show which matched sets dominate the test and whether the result changes under alternative defensible statistics or match specifications. Record why each candidate confounder is temporally and substantively plausible. An independent reviewer should be able to reproduce the Gamma frontier and see exactly how it was translated into the scientific claim.

11. Frequently asked questions

Back to contents

What does Gamma equal to one mean? No hidden difference in treatment odds within matched sets under the model. Is Gamma an odds ratio for the outcome? No; amplification is needed to describe treatment and outcome paths. Can observed covariates calibrate the threshold? Yes, as benchmarks, not guaranteed maxima. Should the biggest sensitivity value be reported? Report the prespecified analysis and explain alternatives rather than selecting the most favourable statistic. Does robustness to Gamma prove causality? No; it addresses one class of hidden assignment bias. Can several weak confounders combine? Yes, so scenarios should consider joint mechanisms rather than only one named binary factor.

12. The gold standard turns a sensitivity number into a scientific argument

Back to contents

A Gamma threshold is valuable because it makes an untestable assumption inspectable, not because it certifies an observational result. The gold standard first defines the matched comparison, then amplifies the hidden-bias boundary into treatment and outcome relationships, builds credible scenarios and calibrates them against measured evidence without treating benchmarks as ceilings. The final conclusion says which confounding mechanisms would change which claim. Robustness becomes a transparent argument about the study’s context rather than a single number detached from it.