The Gold Standard of Experimental Design | Pre-Specified Gamma Grids, Threshold Interpolation and Reporting Resolution

1. Define the sensitivity question before defining the grid

Back to contents

Gamma is a model-based bound on how much two subjects with the same observed covariates may differ in their odds of treatment because of an unobserved covariate. Gamma equals one under the matched randomisation reference model. Values above one allow increasing hidden-bias departures. The analysis asks how an inference behaves under this family of departures; it does not estimate the probability that such bias exists.

State the outcome, effect or test statistic, alternative, tail, significance level, matched-set structure and inferential bound before choosing grid points. A sensitivity threshold for an upper-bound P-value is not interchangeable with a threshold for a confidence interval excluding zero, a sensitivity value, a design sensitivity or an amplification into treatment and outcome associations. Each quantity can cross at a different Gamma.

Write the reporting claim in advance. For example: “We will report the smallest Gamma at which the upper-bound one-sided P-value is at least 0.025, using the specified grid and an exact recomputation at the crossing where available.” This sentence identifies the event, direction and error criterion. Without it, a dense numerical output invites retrospective selection of the most favourable threshold.

Write a machine-readable threshold specification before running the analysis. It should contain the Gamma vector or generation rule, outcome identifier, test statistic, alternative, alpha, bound direction, crossing operator, interpolation setting, numerical tolerance and maximum extension. Store it beside the code. This turns prose into a testable contract: an audit script can confirm that the output rows match the planned grid and that the reported threshold follows the stated operator.

2. Choose the lower and upper Gamma limits for a reason

Back to contents

A grid usually begins at Gamma equals one because this reproduces the no-hidden-bias reference analysis. Verify that the result at one agrees with the primary matched analysis, allowing for any stated difference in statistic or tail. A mismatch can expose coding, missing-data or set-definition errors before the sensitivity curve is interpreted.

The upper limit should be large enough to answer the substantive robustness question, not simply copied from a default. Use prior studies, an amplification exercise, negative controls or subject-matter knowledge to identify a meaningful range. If plausibly important unmeasured confounding would correspond to much larger departures than the chosen maximum, reporting “robust throughout the grid” is weak evidence. Conversely, extending Gamma to implausibly large values can create a dramatic-looking curve without improving interpretation.

Prespecify what happens if no crossing occurs. Report that the criterion was not reached up to the tested maximum, using an open statement such as “greater than 3.00 on the prespecified grid.” Do not report infinity unless the method proves an infinite threshold. The absence of a crossing within a finite range is a censored numerical result, not a declaration that hidden bias cannot overturn the conclusion.

Calibrate the upper range without pretending that amplification removes model dependence. An amplification can express Gamma through combinations of an unobserved covariate’s association with treatment and outcome, but many combinations yield the same bound. Use scientifically plausible examples to judge whether the tested range is informative. Report the assumptions and avoid translating Gamma into a single imagined confounder. The exercise interprets the scale; it does not identify what was missed.

3. Set the increment from the resolution the claim needs

Back to contents

A coarse grid is efficient but brackets the threshold imprecisely. If Gamma is evaluated at 1.40 and 1.50 and the P-value bound crosses the criterion between them, the directly supported result is an interval: the crossing lies above 1.40 and at or below 1.50, subject to monotonicity and numerical accuracy. Reporting 1.47 requires an additional calculation or a declared interpolation rule.

Choose the increment so that the maximum rounding uncertainty is acceptable for the intended claim. A paper comparing thresholds to two decimal places generally needs calculations fine enough to support two decimals near each crossing. Uniform spacing is simple, but an adaptive design can be more efficient: evaluate a coarse prespecified grid, locate the first bracket and then recompute a fixed fine subgrid inside it. The refinement algorithm must be declared in advance and applied consistently across outcomes.

Resolution should follow decision relevance, not visual smoothness. Differences of 0.01 in Gamma may look precise while having little substantive meaning, especially when thresholds vary with test choice or sampling. Retain the full evaluated grid so readers can see the bound curve, and distinguish computational resolution from uncertainty about the study and confounding model.

Choose refinement rules that cannot favour one outcome. For example: evaluate 1.00 to 3.00 by 0.10; if the first crossing occurs, evaluate the enclosing interval by 0.01; if no crossing occurs, stop at 3.00. Apply the same rule to every prespecified outcome. Allowing each curve to receive extra refinement until it produces an attractive decimal creates unequal reporting precision and an undocumented analytic choice.

4. Define the crossing rule at the boundary

Back to contents

Thresholds depend on whether equality counts as crossing. If the criterion is loss of significance at alpha, define the threshold as the infimum Gamma for which the upper-bound P-value is greater than or equal to alpha, or use the strict alternative and state it. Floating-point output displayed as 0.050 may lie slightly above or below 0.05, so comparisons should use the stored value and documented numerical tolerance.

For confidence intervals, define whether crossing occurs when the interval first includes the null, touches the null or changes sign. For effect-bound criteria, define the clinically or practically relevant value. For several outcomes, specify whether each has its own threshold or whether a multiplicity-adjusted familywise rule governs the crossing. A threshold is meaningful only relative to its decision rule.

Ties and flat segments need explicit handling. Several Gamma values may produce the same rounded P-value even though the stored values differ. Report the first evaluated point satisfying the rule and the preceding non-crossing point. If the function is non-monotone because a procedure, approximation or optimisation changes regime, do not force a single threshold; display all crossings and investigate the computation.

Test boundary code with synthetic values. Supply a sequence containing a bound just below alpha, exactly equal in stored precision and just above. Confirm that greater-than-or-equal and greater-than rules return the intended row. Repeat with rounded display values that look identical. This small unit test prevents a common problem in which a written definition and an implementation disagree at equality, particularly when results are copied from formatted tables.

5. Interpolate only when the method supports it

Back to contents

Linear interpolation between two P-values assumes that the bound changes linearly with Gamma inside the bracket. That is a numerical convenience, not a property guaranteed by the sensitivity model. It may be adequate for a narrow interval on a smooth curve, but it should be labelled an interpolation and checked by recomputing near the proposed crossing. Interpolation on the log-Gamma scale or on a transformed P-value scale produces a different answer.

Prefer exact or method-consistent root finding when the software exposes a continuous objective and can recompute the bound at arbitrary Gamma. Specify the tolerance, maximum iterations and returned side of the crossing. If only a discrete function is available, report the bracket rather than inventing precision. A midpoint summary may be useful descriptively, but it does not replace the supported interval.

Never interpolate across a change in test statistic, approximation, integer optimisation solution or matched sample. The curve must refer to one fixed analysis. Keep the rows on both sides of the crossing, the candidate interpolated value and a direct recomputation at that value. If recomputation does not satisfy the rule, refine the grid and retain the discrepancy as a numerical diagnostic.

If interpolation is planned, prespecify the scale and verification. Given endpoints, calculate the candidate root on the chosen scale, recompute the sensitivity bound at that Gamma and iterate until the stored bound is within tolerance on the required side. Keep the original bracket even when a point value is reported. The bracket records what was directly learned from the grid and protects against an apparently precise root generated by an unsuitable local approximation.

6. Worked example: a matched employment programme

Back to contents

A study matches 320 programme participants to controls and tests whether earnings increased. The prespecified one-sided upper-bound P-value is evaluated from Gamma 1.00 to 2.00 in increments of 0.10, with alpha 0.025. The stored bounds are 0.0218 at 1.40 and 0.0296 at 1.50. The supported coarse-grid statement is that the significance threshold lies in the interval (1.40, 1.50], under the stated crossing rule.

The protocol permits refinement by increments of 0.01 inside the first bracket. Recalculation gives 0.0247 at 1.44 and 0.0255 at 1.45. The reported grid threshold is therefore 1.45, with the bracket (1.44, 1.45]. A linear interpolation of the two stored values would give roughly 1.444, but the article reports it only as a numerical illustration because the inferential procedure was not defined by linear interpolation.

A second outcome does not cross alpha by Gamma 2.00. It is reported as greater than 2.00 on the prespecified range, not as infinitely robust. The complete table also shows the estimate-bound criterion and the unadjusted primary P-value. This keeps the claim tied to the outcome, tail, alpha, grid and rule that generated it.

For the employment example, add a reporting-resolution check. A refined crossing in (1.44, 1.45] supports a grid threshold of 1.45 under the first-crossing rule. It does not support claiming that robustness increased by 0.01 compared with a second study whose threshold was generated by different scores or interpolation. Before comparing, harmonise outcomes, statistics, tails, alpha and grid protocol; otherwise describe each result within its own design.

7. Check monotonicity, precision and reproducibility

Back to contents

For many Rosenbaum sensitivity procedures, the relevant worst-case P-value bound should move monotonically in the conservative direction as Gamma increases. Check this property numerically. Small reversals can arise from optimisation tolerances, simulation error, approximation changes or rounding; larger reversals may indicate a coding error or that different samples were used across rows. Do not conceal reversals by plotting only a smoothed line.

Store more numerical precision than is displayed. The crossing comparison should use machine values, while the table can show a sensible number of decimals. Record software, package version, function, statistic, seed for any stochastic procedure and convergence information. An apparently tiny difference in a threshold can otherwise reflect only a version change or tolerance.

Reproduce the Gamma-equals-one result, verify a few rows independently where possible and rerun the crossing with tighter tolerances. If exact and approximate methods are both available, compare them near the boundary. The goal is not to maximise decimal agreement; it is to know whether the reported threshold would change at the promised reporting resolution.

Investigate any non-monotone row rather than deleting it. Re-run with tighter optimisation, fixed random seeds and stored full precision. Confirm that the same matched observations and weights are used at every Gamma. If the procedure legitimately produces non-monotone criteria, replace the one-number summary with a curve and all crossing intervals. A sensitivity threshold assumes an ordered robustness path; when that ordering fails, the summary must change.

8. Changed-context transfer check: matched sets and clustered outcomes

Back to contents

Move from matched pairs with a continuous outcome to variable-ratio matched sets in which several controls share a treated unit, and outcomes are clustered by school. The general reporting discipline transfers: define Gamma, the inferential bound, the grid, the crossing and the rounding. The computational procedure does not transfer automatically because the reference distribution and dependence structure have changed.

A function designed for matched pairs may not provide valid bounds for variable-size sets, and treating correlated pupils as independent can overstate evidence before sensitivity analysis begins. Select a procedure that supports the actual matched structure and outcome. If cluster-level assignment or interference is plausible, explain which units Gamma compares and what treatment-odds model is being bounded.

The changed context may also require a different practical range. An odds-ratio departure that seems modest at the pupil level may have a different interpretation at the school level. Report the analysis unit and, where useful, an amplification in scientifically meaningful confounder associations. Transfer succeeds when the transparent grid protocol remains but the inferential engine is rebuilt for the design.

In variable-ratio matching, document how Gamma operates within each matched set and whether software accommodates set size. Verify that the statistic and reference distribution correspond to the actual design. If outcome dependence is clustered, use a method that reflects that structure before computing sensitivity bounds. Fine Gamma steps cannot compensate for an invalid independence assumption. The numerical protocol is the final layer of a design-valid analysis, not a substitute for it.

9. Know what grid precision cannot repair

Back to contents

A finely resolved threshold does not show that the study is unbiased, identify the unmeasured confounder or give the probability that the conclusion is correct. Gamma bounds treatment assignment odds under a specific model. Two unobserved covariates, effect modification, outcome measurement error, selection after treatment and interference can fall outside the story told by one threshold.

The threshold is also outcome- and test-specific. Changing the score statistic, alternative, trimming rule or matched sample can change it. Searching among these choices after viewing thresholds turns a sensitivity analysis into another optimisation for significance. Prespecify primary choices and label exploratory variants. If several defensible analyses disagree, the disagreement is evidence about fragility rather than a reason to select the largest Gamma.

Computational resolution is not sampling uncertainty. A threshold reported as 1.45 may move substantially under resampling or a different eligible sample even if the numerical crossing is known to 0.001. Where the goal requires uncertainty, use a method designed to quantify it and report that separately. Do not attach extra decimals to suggest inferential certainty the design does not provide.

Create the publication sentence directly from archived output. Code should select the last non-crossing row, first crossing row, threshold label and non-crossing maximum, then render the exact outcome, criterion and range. Compare that generated sentence with the manuscript. This avoids transcription and rounding drift during revisions. If the prose is edited to make a stronger claim than the generated evidence supports, the mismatch becomes visible before publication.

Add a complete miniature table to the protocol before real results exist. Columns should include Gamma, upper-bound P-value, lower and upper effect bounds where relevant, crossing flag, convergence flag and method version. Populate it with synthetic values and generate the intended text. This forces decisions about missing values, equality, failed convergence and non-crossing outcomes while no substantive result is at stake.

Define how failed rows are handled. Skipping a non-converged Gamma and accepting the next row can widen or narrow the threshold silently. Retry under a documented numerical procedure, report the unresolved bracket and avoid interpolation across the failure. Numerical failure near a crossing is part of the uncertainty readers need to see.

A curve can communicate more than a crossing. Show how quickly the bound approaches the criterion, whether conclusions remain similar over a substantively calibrated range and whether effect bounds become uninformative before P-values cross. Mark the evaluated points so a smooth plotting function does not imply calculations that were never performed.

Use accessible labels: Gamma is the allowed treatment-odds departure, not a percentage of bias. Provide the crossing bracket in text so the result remains interpretable without the figure. The graph supports the audit; it should not be the only record of the numerical rule.

10. Audit the full curve, not only the winning threshold

Back to contents

Archive every evaluated Gamma and bound, the analysis data identifiers, matched-set definitions, exclusions, code and session information. A single threshold hides curve shape and makes it impossible to check the crossing. Plot the complete prespecified range with the criterion line and mark evaluated points so interpolation is visible. Include the last non-crossing and first crossing row in the main report.

Run a mechanical audit before publication. Confirm Gamma values are strictly increasing, the no-bias row matches the primary analysis, the sample is constant, bounds satisfy expected monotonicity and the threshold code implements the written equality rule. Verify that displayed rounding does not shift a row across alpha. Recompute every published threshold from the archived table rather than copying it by hand.

For multiple outcomes, retain a manifest connecting each curve to its outcome, tail, statistic, alpha and multiplicity rule. If an outcome never crosses, store the tested maximum and open-ended label. If it crosses at the first point above one, report the narrow supported interval rather than implying a stable value. The audit should make an independent reader able to regenerate the sentence in the abstract from the underlying grid.

When several significance levels are relevant, designate one for the primary threshold and display others as secondary curves or columns. Do not choose alpha after examining which value yields a memorable Gamma. If a one-sided alpha of 0.025 corresponds to the primary hypothesis, retain it consistently across the no-bias and sensitivity analyses.

Multiplicity adjustments can make thresholds interdependent. A familywise procedure may change which outcome crosses first as Gamma varies. Prespecify whether adjustment is recomputed at every Gamma or fixed from the reference analysis, and justify the choice. A collection of outcome-specific unadjusted thresholds does not establish familywise robustness.

For reproducibility, hash or version the analysis dataset and matched-set file used for the grid. A small change in exclusions can alter the threshold while leaving code unchanged. Store seeds for stochastic approximations and confirm that parallel execution does not change results at the reported resolution.

Run the full analysis in a clean environment from raw analytic input to publication table. Capture warnings. Package updates should trigger a comparison near every reported crossing, not only a rerun of the primary estimate. This is proportionate because the sensitivity sentence may depend on one boundary row.

11. Frequently asked questions

Back to contents

Must Gamma increments be equally spaced? No. A declared coarse-to-fine algorithm can be more efficient, and log spacing may suit a wide range. The reported resolution must match the local spacing or root-finding tolerance. Can I interpolate a threshold from a graph? Not reliably. Use stored numerical values, declare the scale and verify by direct recomputation when possible. Is the first non-significant Gamma always the sensitivity value? Terminology varies; define the quantity rather than relying on the label.

How many decimals should be reported? No more than the grid, tolerance and substantive interpretation support. With a 0.10 grid and no refinement, a two-decimal point threshold is usually unjustified; report the bracket. What if the P-value equals alpha? Follow the prespecified equality rule using unrounded values. What if software rounds Gamma in its output? Retain the internal values or rerun with an explicit vector.

Should the grid be extended after seeing no crossing? Only under a prespecified extension rule or as clearly labelled exploratory analysis. Otherwise the upper range becomes outcome-dependent. Can thresholds from different studies be ranked? Only cautiously: outcome, design, statistic, sample, confounding model and amplification may differ. A numerically larger Gamma is not automatically stronger evidence.

Report amplification as a set of examples, not a unique translation. For a reported Gamma, show several combinations of treatment-assignment and outcome associations that are compatible with the amplification formula, using definitions readers can verify. Compare those combinations with measured covariates only as a calibration aid; observed associations can be affected by adjustment, measurement error and study population.

Avoid statements that an unmeasured confounder would need to be “Gamma times stronger” than every observed covariate unless the comparison truly follows from the chosen model. The sensitivity analysis bounds odds of treatment within matched sets; it does not supply a universal strength scale.

If a threshold is used in an abstract or policy summary, include its conditions in the same sentence: outcome, inferential rule and tested range. “The result remained significant to Gamma 1.45” is incomplete if 1.45 is merely the first non-significant grid point. Prefer “the upper-bound one-sided P-value first reached 0.025 at Gamma 1.45 on the prespecified 0.01 refinement grid.”

The extra words prevent a point from being mistaken for an estimated confounder or a universal robustness score. Concision should remove decoration, not the definition that gives the number meaning.

12. The gold standard makes numerical resolution part of the design

Back to contents

A defensible sensitivity threshold is generated by a protocol, not discovered by scrolling through software output. The protocol fixes the inferential quantity, lower and upper Gamma limits, grid or refinement algorithm, crossing inequality, interpolation rule, numerical tolerance and treatment of non-crossing results. These choices belong in the analysis plan because they control the sentence ultimately reported.

The standard then preserves the curve behind the threshold. It verifies the no-bias row, checks monotonicity, records the crossing bracket, recomputes any interpolated value and limits decimals to supported resolution. It keeps computational precision separate from sampling uncertainty and substantive calibration. A threshold can be numerically exact within its model while the overall robustness claim remains scientifically uncertain.

The practical test is simple: could another analyst, using the same data and written protocol, produce the same crossing and the same wording without making an undocumented choice? If not, the number has more apparent precision than procedural meaning. Prespecification, complete grids and honest brackets turn Gamma from a decorative robustness score into a reproducible sensitivity analysis.

Distinguish a threshold computed from one observed sample from design sensitivity, which concerns large-sample behaviour under an assumed favourable model. A test with high design sensitivity may still have a modest finite-sample threshold, and the ranking of tests can vary with effect distribution. If test choice was motivated by design sensitivity, prespecify it before outcomes are inspected and still report the finite grid honestly.

This distinction also limits cross-study comparisons. Two studies can use the same Gamma grid but differ in matching quality, effect heterogeneity, sample size and statistic. Treat thresholds as components of a design-specific argument, not portable grades.

Finally, preserve exploratory extensions without letting them rewrite the primary protocol. If reviewers request a wider range, different statistic or finer root finder, add a labelled analysis and retain the original grid result. Explain why the new view was added and whether it changes interpretation.

A transparent extension is often useful. The problem is replacing the prespecified result with a more attractive one and erasing the path. Versioned grids let readers distinguish planned robustness evidence from later learning while benefiting from both.

Add a threshold ledger to the statistical analysis plan. Each row should name the outcome, analysis population, statistic, tail, alpha, Gamma range, coarse step, refinement rule, crossing operator, interpolation method, tolerance, multiplicity treatment and software implementation. Approve the ledger before unblinded outcome analysis where feasible. If a field changes later, retain the original entry, date the amendment and state whether results were known. This makes analytic flexibility visible without pretending that protocols never evolve.

Use interval language consistently. With evaluated points 1.44 and 1.45, a first crossing at 1.45 supports a bracket determined by the operator and monotonicity assumption. If the preceding point is non-crossing and the next is crossing under greater-than-or-equal, write the interval accordingly. Do not alternate between “robust to,” “fails at” and “threshold of” without defining whether endpoints are included. Small wording changes can reverse the status of the boundary.

Check reporting precision with a perturbation analysis. Tighten numerical tolerance, add one decimal to the local grid and, if applicable, change only the interpolation scale. If the published two-decimal threshold changes, reduce the reported precision or use a bracket. This is a numerical stability check, not an invitation to select the most favourable answer. Preserve all variants and designate the protocol result.

For confidence intervals, archive both endpoints across Gamma. The relevant crossing may be caused by one endpoint touching the null, but the widening pattern can reveal whether the bound is informative. If effect bounds cross a practical threshold earlier than the P-value crosses alpha, report both quantities under their distinct rules. Statistical and substantive robustness answer different questions.

Where packages offer convenience functions that return a sensitivity value directly, inspect their definition. Determine whether they use a discrete grid, continuous optimisation, finite-sample correction, asymptotic approximation, one- or two-sided level, and what happens at equality. A function name is not a specification. Validate it against a manually constructed small example or the full returned curve.

The completed report should enable three reproductions: reconstruct the matched analysis at Gamma one, regenerate every evaluated bound, and derive the publication sentence from the threshold rule. If any step depends on a spreadsheet cell, visual reading of a plot or undocumented rounding, the workflow is not complete. A reproducible Gamma number is not the whole robustness argument, but it is the minimum evidence for claiming a numerical threshold.

Retain the sign and direction of every bound. When an outcome is coded so that smaller values are beneficial, a copied upper-bound routine or plot label can reverse interpretation. Include a synthetic sign check in the test suite and label the favourable direction in the outcome manifest.

If weights, scores or matched sets are estimated from the same data, freeze them before traversing Gamma. Re-estimating the analysis at each grid point changes more than the hidden-bias parameter and breaks the meaning of the curve. Any alternative re-estimation should be a separate analysis with its own protocol.

Finally, treat prose review as part of verification. A statistician should confirm the algorithm, while a subject-matter reader should explain the reported threshold without calling Gamma the amount of confounding actually present. Agreement between those readings is evidence that numerical precision and substantive meaning have remained aligned.