The Gold Standard of Experimental Design | Sensitivity Curves, Pointwise Bands and Familywise Robustness Claims

1. Define the curve before drawing it

Back to contents

A sensitivity curve maps an assumption parameter to an inferential quantity. In a matched observational study, Gamma may bound how much treatment odds can differ within a matched set because of hidden bias. The vertical axis might be a worst-case p-value, effect bound, confidence limit or sensitivity value. These are not interchangeable.

State the estimand, test, direction and model. A curve for the sharp null of no individual effect answers a different question from a confidence interval for an average effect under heterogeneous effects. A one-sided robustness claim differs from a two-sided effect claim. The graph should not conceal those distinctions.

Define the domain where the parameter has a defensible interpretation. Extending Gamma far beyond substantively plausible values may be useful as a diagnostic, but it should not be presented as observed confounding. Sensitivity parameters index assumptions; they are not estimated exposure effects unless a separate calibration justifies that language.

Practice check: Write the horizontal and vertical axes as full sentences, including the estimand and worst-case operation. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Switch from a p-value curve to a lower confidence-bound curve and identify which verbal claims must change. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Keep the sensitivity model, estimand, hypotheses, sidedness, parameter meaning, domain and software implementation. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

2. Distinguish pointwise coverage from simultaneous coverage

Back to contents

A 95% pointwise interval at Gamma=1.2 is designed to cover the target at that fixed Gamma under its assumptions. Repeating the calculation at twenty Gamma values does not create a band with 95% probability of covering the entire underlying curve. Some point on the grid can miss even when every marginal interval has nominal coverage.

The distinction matters when the conclusion uses words such as “for all Gamma up to 1.5” or when the most favourable part of the curve is selected after inspection. That is a simultaneous or familywise statement. Pointwise intervals remain appropriate for prespecified individual values and can describe local uncertainty, but the caption must not promise uniform protection.

Coverage is also conditional on the sensitivity model. A simultaneous band handles multiplicity across the chosen domain; it does not validate the hidden-bias model, matching quality or outcome measurement. Stronger statistical calibration cannot repair a weak scientific estimand.

Practice check: Take a pointwise plot and underline every sentence that quietly refers to the whole curve. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Reduce the display to one prespecified Gamma and explain why the multiplicity problem changes. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Label intervals or bands explicitly, state the coverage target and preserve the family of Gamma values covered by the claim. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

3. Treat the grid as a family of linked inferences

Back to contents

Sensitivity results at nearby Gamma values are highly dependent because they use the same outcomes, matches and optimization structure. This dependence means a Bonferroni correction may be conservative, but dependence does not make multiplicity disappear. The calibration should reflect the joint process where possible.

Define the family before looking at the curve: Gamma values, outcomes, estimands, subgroups and directions. A family may be the full curve for one primary outcome, or it may include several outcomes if the substantive claim joins them. Splitting one scientific claim into several graphs does not necessarily create independent families.

The familywise error rate concerns the chance of at least one false rejection in the family. A uniform band instead targets joint coverage of the curve. These goals are related but not identical in every procedure. Report which quantity is controlled and under what assumptions.

Practice check: List every inferential choice that contributes to the headline robustness statement, not only Gamma points. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Add three secondary outcomes and decide whether they belong to the same family as the primary curve. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Archive the prespecified family, dependence model, error criterion, alpha allocation and any post hoc additions. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

4. Choose the Gamma domain and resolution deliberately

Back to contents

A dense grid looks more informative but can encourage false precision. The useful Gamma range should be connected to design knowledge, calibration with observed covariates or a policy-relevant robustness target. Include Gamma=1 as the no-hidden-bias reference and predefine the largest value for the primary claim.

Grid spacing affects how threshold crossings are reported. If significance changes between 1.35 and 1.40, the sensitivity value is not exactly 1.40. Use an algorithm or interpolation justified by monotonicity and numerical behaviour, and report tolerance. Do not refine the grid only near a favourable boundary after seeing the result without labelling that exploration.

A simultaneous claim over a continuum may require methods stronger than correction on a finite grid. If only the grid is covered, say so. Monotonicity can help translate grid results, but it must be demonstrated for the actual statistic and procedure.

Practice check: Justify the maximum Gamma and grid spacing from substantive and numerical considerations. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Replace a monotone worst-case p-value with a nonmonotone estimate curve and reassess interpolation. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Keep the domain rationale, grid, adaptive refinements, convergence tolerances, monotonicity checks and exact threshold calculation. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

5. Construct a band for the claim you intend to make

Back to contents

A simultaneous band can be built by calibrating a joint critical value for the supremum of an appropriately standardised process, using analytic, permutation, bootstrap or other valid methods. The correct construction depends on the estimator, matching design and sensitivity model. Reusing pointwise critical values is not a simultaneous method.

Studentisation can matter because variability changes across Gamma. A constant-width band may be inefficient or misleading. Resampling must reproduce the dependence created by matched sets and any nuisance estimation. Small samples and non-smooth worst-case optimization can make asymptotic or bootstrap approximations fragile.

Validate with simulation under designs resembling the study. Check joint coverage across the prespecified grid, not only average pointwise coverage. If a justified simultaneous method is unavailable, show pointwise intervals honestly and narrow the claim rather than adding the word “band”.

Practice check: Match the resampling unit and statistic to the randomisation or sampling structure of the design. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Move from matched pairs to variable-ratio matched sets with few clusters and inspect which approximation changes. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Retain algorithms, critical-value construction, studentisation, resampling seeds, convergence, simulations and empirical coverage. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

6. Worked example: a curve that crosses the null

Back to contents

A matched study estimates a beneficial effect. At Gamma values from 1.0 to 1.6, the pointwise 95% lower confidence limit stays above zero through 1.40 and touches zero at 1.45. The authors want to state that the effect remains positive for every Gamma up to 1.40.

That sentence is stronger than twenty separate pointwise statements. A simultaneous 95% lower band over Gamma 1.0–1.40 is calibrated using a design-respecting resampling procedure. Its lower edge crosses zero at 1.32. The pointwise curve remains useful, but the familywise robustness claim is limited to Gamma 1.30 on the chosen grid.

The result does not mean hidden bias of Gamma 1.31 exists or overturns the study. It means the chosen simultaneous procedure no longer guarantees a positive lower bound across the whole domain. A calibration exercise then compares that magnitude with observed covariate imbalances, while acknowledging that unobserved covariates can behave differently.

Practice check: Write separate captions for the pointwise display and the simultaneous band, then compare the strongest permitted claim. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Change the headline to a statement at one policy-chosen Gamma rather than across a range. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Preserve the grid, pointwise results, joint critical value, band, crossing tolerance, calibration and exact wording of the conclusion. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

7. Handle multiple outcomes and subgroups explicitly

Back to contents

A sensitivity curve for each of five outcomes creates another layer of multiplicity. Controlling each curve simultaneously over Gamma does not automatically control the chance of any false claim across outcomes. The same issue arises with subgroups, alternative matching specifications and several estimands.

Predefine a primary family and separate confirmatory from exploratory displays. Hierarchical or gatekeeping procedures may preserve power when the scientific logic has an order. Joint methods can use dependence among outcomes, but the procedure must maintain its error guarantee under the sensitivity model.

Do not choose the most robust-looking curve after inspection and report its pointwise threshold as if selected in advance. Exploratory curves can generate hypotheses and diagnose heterogeneity. Their value does not require confirmatory language.

Practice check: Count the total opportunities for a favourable robustness claim across outcomes, subgroups and specifications. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Declare one primary outcome and use the others as mechanism checks; redesign the family accordingly. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Keep outcome hierarchy, subgroup definitions, specification set, familywise method, exploratory labels and all curves. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

8. Changed-context transfer: breakdown frontiers and tuning curves

Back to contents

The same logic appears outside Rosenbaum Gamma. A breakdown frontier maps combinations of assumption relaxations under which a conclusion holds. A regularisation path maps results over tuning parameters. An equivalence conclusion may be checked across plausible margins. Pointwise uncertainty at each coordinate does not automatically support a statement about the entire frontier or path.

Identify the object: a curve, surface or selected threshold. Define whether coverage is required at one prespecified point, across a finite grid or uniformly over a continuum. Geometry and dependence determine the method; a two-dimensional frontier usually requires more than copying a one-dimensional correction.

The transferable principle is claim matching. Uncertainty should be calibrated for the selection and scope used in the conclusion. If the reader is invited to scan the picture and choose a favourable region, the analysis must either account for that search or call it exploratory.

Practice check: Rewrite a frontier claim as a precise coverage event over the plotted domain. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Move from a one-dimensional Gamma curve to a two-parameter bias surface and identify the expanded family. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Archive the domain, grid or continuum target, selection rule, geometry, joint method and exploratory interactions. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

9. Know the limits and common misconceptions

Back to contents

A simultaneous band is not automatically wider at every point than every alternative, but it usually pays for joint protection. Pointwise intervals are not wrong; they answer a narrower question. A familywise correction does not prove the sensitivity model is realistic or convert Gamma into a measured confounder.

A robustness threshold is not an effect size and should not be compared across studies without matching design, outcome and test details. A high threshold can coexist with bias from measurement, selection or interference that the model does not represent. Sensitivity analysis probes specified departures, not every threat to validity.

Finally, “insensitive up to Gamma” is incomplete without the outcome, hypothesis, alpha, sidedness, domain and whether the claim is pointwise or simultaneous. The concise wording can appear in the headline, but the full definition must be available beside the figure and in the methods.

Practice check: Find every use of “robust” and replace it with the exact assumption and inferential event. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Compare two studies with the same Gamma threshold but different matching ratios and outcomes. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: State model exclusions, design limitations, calibration caveats and the precise family protected by the procedure. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

10. Audit the curve from design to claim

Back to contents

The audit begins with the matched design, outcome definition, estimand and sensitivity model. Preserve match diagnostics, set sizes, missing-data rules and any outcome transformation. Then retain code and outputs for every Gamma, including optimisation convergence and worst-case configurations where meaningful.

For uncertainty, document whether intervals are pointwise or simultaneous, how the critical value was obtained and what family it covers. Reproduce the crossing rule and exact claim from the saved result rather than reading a rounded graph. Test nearby grids, plausible nuisance specifications and small-sample performance.

The publication should connect figure, caption, methods and abstract. A caption that says pointwise while the abstract says “throughout the range” is a broken evidence chain. Automated checks can flag such mismatches, but substantive review is still needed to ensure the statistical family matches the scientific family.

Practice check: Trace the headline robustness sentence back to the exact rows of output and the coverage procedure. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Repeat after discovering an unplanned subgroup curve was added during peer review. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Archive design objects, data versions, code, grid, raw results, critical values, simulations, figures, captions and final claims. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

A curve audit should include a claim ledger. For each sentence in the abstract, results and caption, record the mathematical event it requires: one fixed Gamma, all points on a finite grid, an entire interval, one outcome, or a family of outcomes and subgroups. Link that event to the procedure that controls it. Empty cells expose language that has outrun the calculation. The ledger also prevents a simultaneous band for one curve from being used to support a familywise claim across several selected curves.

Simulation should mimic the design rather than a generic normal curve. Preserve matched-set sizes, outcome distribution, treatment prevalence, clustering and nuisance estimation. Generate settings at the boundary of the null and under plausible effect heterogeneity. For each replicate, compute the whole pipeline and ask whether the true curve lies inside the full band across the protected domain. Report Monte Carlo uncertainty in the coverage estimate. Pointwise coverage averaged over Gamma is not a substitute for joint coverage.

Numerical auditing matters because sensitivity bounds often involve optimisation. Save convergence status, tolerances, active constraints and any fallback algorithm for every Gamma. A smooth plotted line can hide a failed solver or a switch between local solutions. Re-run boundary points with tighter tolerances and an independent implementation when feasible. If the curve is expected to be monotone, flag violations for investigation rather than smoothing them away automatically. The audit trail should retain both raw and corrected displays.

Separate two uncertainties that are often blended. Sampling uncertainty concerns what would vary under repeated sampling or assignment within the design. Sensitivity uncertainty concerns conclusions under a family of unobserved-bias assumptions. A band over Gamma may express both in a defined way, but Gamma itself is not thereby estimated. Calibration with observed covariates supplies substantive scale, not a confidence interval for the hidden confounder. Captions should keep these statements distinct.

When a threshold is the main summary, treat its selection as an estimand. Define whether it is the first Gamma where a pointwise p-value exceeds alpha, the first crossing of a simultaneous lower band, or another rule. Specify grid and interpolation, then quantify uncertainty when a valid method exists. Reporting only a rounded crossing can create apparent differences between studies that come from grid spacing or random noise. The full curve and band should remain available beside the threshold.

A layered display can be honest and readable. Show the estimated sensitivity curve, a light pointwise envelope for local precision and a clearly differentiated simultaneous band for the prespecified family. Add vertical markers for substantively calibrated Gamma values and label exploratory regions. The legend should say exactly which object supports which claim. Accessibility requires that line style and text, not colour alone, distinguish the layers.

Finally, freeze the analysis before reading the most persuasive curve. If reviewers request a larger domain or additional outcome, rerun the familywise procedure for the expanded family or label the addition exploratory. Do not preserve the old critical value while expanding the scientific claim. A transparent update may widen the band and weaken the headline; that is the cost of asking a larger question, not a defect in the method.

11. Frequently asked questions

Back to contents

Can I connect pointwise intervals and call them a band? Visually yes, inferentially no; the label must state pointwise unless joint coverage was calibrated. Is Bonferroni always required? No. Dependence-aware simultaneous methods may be more efficient, but some valid familywise method is needed for the family claim.

Does a dense grid automatically require a huge penalty? Dependence matters; nearby points are not independent. Does simultaneous coverage over Gamma handle multiple outcomes? Only if outcomes are included in the protected family. Can I report the sensitivity value from pointwise tests? Yes, with the correct definition and uncertainty; do not convert it into a uniform claim.

What if software supplies only pointwise intervals? Use them for prespecified local questions, label exploratory curve reading and avoid global language. Why show both? Pointwise intervals reveal local precision; a simultaneous band supports a whole-domain statement. Displaying both can teach the reader how the inferential target changes.

Practice check: Write a short answer for the figure caption and a longer answer for the statistical methods. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Explain the distinction to a decision-maker choosing one Gamma rather than scanning a range. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Maintain the approved terminology, procedure limitations, protected family and examples of claims that are and are not supported. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.

12. The gold standard calibrates uncertainty to the whole claim

Back to contents

A sensitivity curve makes assumptions visible by showing how inference changes over a prespecified domain. Its usefulness depends on a clear estimand, interpretable parameter, reproducible grid and honest distinction between pointwise and simultaneous uncertainty.

When the conclusion concerns every value in a range, the analysis treats the range as a family. It uses a joint band or familywise procedure suited to dependence and design, checks finite-sample behaviour and includes other outcomes or selections that belong to the scientific claim. Exploratory displays remain valuable when labelled as such.

The final test is linguistic and mathematical: does the coverage event match the sentence? “At Gamma=1.3” is local. “For all Gamma through 1.3” is simultaneous. When those are kept distinct, sensitivity analysis becomes a disciplined account of model dependence rather than a decorative curve that promises more robustness than it measured.

Practice check: Pair every planned sentence with its exact inferential event and delete any sentence without a matching calculation. Do the check before settling the conclusion, because a neat answer produced after the decision is not evidence that the rule guided it. State what would count as a different answer and who is responsible for acting on it.

Changed-context drill: Imagine the curve crosses the null one grid point earlier and ask whether the reporting rule stays unchanged. Keep the principle, then change only the parts that genuinely depend on the setting. If the method cannot say what changes and what stays fixed, it is a slogan rather than a transferable standard.

Evidence record: Publish the model, domain, family, procedure, validation, full curve, supported claim and explicit limits. Preserve enough detail for a careful reader to reconstruct the reasoning without relying on the memory or authority of the person who made the decision. Name uncertainty directly; a visible limit is more useful than silent precision.