The Gold Standard of Experimental Design | Partial Identification Sets, Sensitivity Parameters and Set-Valued Causal Claims

1. Define the causal estimand before the identified set

Back to contents

Partial identification begins with a target such as the average treatment effect, effect among treated units or a policy value. The data and assumptions then determine which values of that target are observationally compatible. Without a clear estimand, a bound can be mathematically correct and scientifically irrelevant.

State population, treatment versions, outcome time and intervention contrast. A set for a sharp effect under one treatment protocol does not answer a question about usual-care variation. If the target is conditional or subgroup-specific, define the support and aggregation.

Separate identification from estimation. The population identified set exists even with infinite data under the maintained assumptions. A sample estimate of its endpoints adds sampling variability. These are two uncertainties with different sources and remedies.

First-principles check: Write the counterfactual target in words and notation before deriving any bound. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Define an average treatment effect for one concrete experiment, including treatment versions and outcome time. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Change the target from the full population to treated participants only. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep estimand, population, treatment protocol, outcome, scale and assumptions. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

2. Understand why a set can be the correct answer

Back to contents

Point identification means the observed distribution and assumptions imply one value for the target. Partial identification means they imply a region. This is not a failed estimate; it is the sharpest conclusion warranted without stronger assumptions when the bounds are sharp.

For an outcome bounded between 0 and 1 with missing values, the missing outcomes can be placed at extremes to derive worst-case mean bounds. The interval may be wide, but replacing it with a model-based imputation point adds assumptions. Those assumptions can be useful if defended, yet they should appear as a new rung on an assumption ladder.

Ask whether the set is sharp: can every value inside it be generated by some data-generating process satisfying the assumptions and observed evidence? Non-sharp bounds may be valid but unnecessarily wide. Computational convenience should not be confused with the identified region.

First-principles check: Treat every value compatible with evidence and assumptions as live until another assumption rules it out. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Derive worst-case mean bounds with ten missing binary outcomes. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Allow the outcome range to be wider and see how the set changes. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain support restrictions, bound derivation, sharpness argument and any conservative approximation. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

3. Index assumption strength with an interpretable parameter

Back to contents

A sensitivity parameter can relax a baseline assumption gradually. It might bound selection on unobservables, departure from conditional independence, outcome misclassification or the fraction of contaminated observations. The parameter must have an operational meaning; a Greek letter alone does not make a sensitivity analysis interpretable.

Define the zero or baseline point, direction and maximum range. Show how parameter values restrict the admissible data-generating processes. Whenever possible, calibrate against observed covariates or domain knowledge, while stating that unobserved factors may differ.

For each parameter value, derive the identified set rather than only a p-value. The result is a set-valued function. Monotonic widening may occur under nested assumptions, but it should be proved or checked for the actual method.

First-principles check: Connect each parameter value to a concrete relaxation of the causal assumption. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Write a plain-language interpretation for three values of a selection sensitivity parameter. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Replace selection bias with outcome misclassification. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep parameter definition, baseline, domain, nesting, calibration and set-computation method. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

4. Build an assumption ladder

Back to contents

An assumption ladder begins with minimal credible restrictions and adds stronger, transparent conditions one at a time. For example: bounded outcomes; monotone treatment response; monotone selection; limited departure from unconfoundedness; then point-identifying independence. Each rung should show the resulting set.

The ladder reveals which claim depends on which assumption. A policy conclusion may hold under broad bounds even when a precise effect does not. Conversely, a narrow interval at the top can be driven almost entirely by an untestable rung.

Do not mix incomparable rungs. Adding one assumption while dropping another can change the model in more than strength. Label branches when restrictions are alternatives rather than nested refinements. The aim is an audit of inferential leverage, not a march toward the smallest interval.

First-principles check: Change one assumption at a time and record the contraction it produces. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Create four nested rungs and calculate or describe the set at each. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Add an alternative non-nested assumption justified by a different mechanism. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Archive each model, logical relation, set, contraction and substantive justification. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

5. Separate identified sets from confidence regions

Back to contents

The estimated endpoints of an identified set vary across samples. A confidence procedure can target the entire identified set or the unknown point parameter known to lie within it. Those coverage goals are related but not identical, and terminology should state the target.

An interval that simply plugs standard errors into each endpoint may fail near cases where the identified set becomes narrow or constraints bind. Valid methods account for joint endpoint behaviour and may require uniformity over a class of data-generating processes.

Graph identification width and sampling uncertainty separately when possible. More data can shrink endpoint uncertainty while leaving the population identified set wide. That pattern is informative: the limiting problem is assumption strength, not sample size.

First-principles check: Name the object covered by the confidence statement. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Write two captions, one for an estimated identified set and one for a confidence region around it. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Consider a sequence where the true identified set shrinks toward a point. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep coverage target, method, regularity conditions, resampling and finite-sample checks. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

6. Worked example: treatment effect with missing outcomes

Back to contents

In a randomised programme, 20% of outcomes are missing in treatment and 10% in control. Outcomes range from 0 to 100. The observed complete-case mean difference is 8 points, but complete-case analysis is unbiased only under additional missingness assumptions.

Worst-case replacement puts missing treated outcomes at 0 and missing control outcomes at 100 for the lower bound, then reverses the extremes for the upper bound. The resulting set includes negative and positive effects. A sensitivity parameter limits how different missing outcomes can be from observed outcomes within each arm. As the permitted difference narrows, the set contracts.

At the domain-justified sensitivity range, the lower bound remains below zero. The study cannot claim a uniformly beneficial effect. It can report the effects compatible with each assumption, the parameter value where the whole set becomes positive and a confidence procedure for the bound function. The midpoint of the set is not substituted for the effect.

First-principles check: Place missing values adversarially under the stated support before adding a missingness model. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Compute the worst-case arm means and effect bounds for a small numeric table. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Change the outcome to an unbounded measure and identify the extra restriction required. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Preserve missingness by arm, support, formulas, sensitivity grid, sets, confidence regions and claim. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

7. Make set-valued causal claims decision-relevant

Back to contents

A decision can use a set without choosing a midpoint. A robust-benefit rule acts only if every effect in the identified set exceeds a policy threshold. A minimax-regret rule compares worst-case losses. A precautionary rule may focus on the lower endpoint when harm is asymmetric.

State the decision criterion separately from identification. Statistics determines which effects remain compatible; values and consequences determine how to act across them. Two decision-makers can rationally choose differently from the same set because their loss functions differ.

Report sign, magnitude and threshold robustness. “The set excludes harm” is stronger than “most of the set is positive” unless a probability distribution over unidentified values is justified. An identified set is not a posterior distribution and has no natural centre or mass.

First-principles check: Apply a declared decision rule to all values in the set. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Compare robust-benefit, maximin and midpoint decisions for the same interval. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Change the cost of an unnecessary programme and missed benefit. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep the identified set, policy threshold, loss or rule, decision and sensitivity to values. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

8. Changed-context transfer: noncompliance in a trial

Back to contents

Random assignment can identify an intention-to-treat effect while the effect of actually receiving treatment remains partially identified without stronger assumptions. Instrumental-variable assumptions may identify a complier effect, but that estimand is not automatically the population average treatment effect.

Build the ladder: random assignment and observed uptake; exclusion restriction; monotonicity; then any assumption used to extrapolate beyond compliers. Show which set or point each rung identifies. A precise local effect should not be retitled as a universal treatment effect.

The transferable principle is that data plus assumptions define the target’s feasible values. What changes is the causal structure. Missing-outcome support bounds become principal strata and compliance restrictions. The audit must keep estimand shifts visible.

First-principles check: Track both the assumption and the estimand changed by an instrumental-variable step. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Draw compliance types and mark which outcomes are observed under assignment. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Allow defiers and see which identification result weakens. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain assignment, uptake, strata assumptions, exclusion, monotonicity and each estimand. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

9. Know the limits and common misconceptions

Back to contents

The width of an identified set is not a confidence level. A narrow set can rest on implausible assumptions; a wide set can be the honest answer from strong data under weak assumptions. More observations do not necessarily remove identification uncertainty.

The midpoint is not an unbiased or best estimate without an additional criterion. A sensitivity parameter is not estimated merely because a curve is plotted. “Robust up to c” means the stated claim holds under the model indexed through c, not that real confounding equals c.

Partial identification also does not license arbitrary pessimism. Bounds should respect all credible observed restrictions and be sharp where possible. Overly loose convenience bounds can hide useful information just as point estimates can overstate it.

First-principles check: Identify whether a statement concerns data uncertainty, assumption uncertainty or decision preference. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Correct five common interpretations of bounds and sensitivity parameters. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Explain a narrow confidence region around a wide identified set. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: State sharpness, assumptions, calibration limits, coverage target and what more data can or cannot change. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

10. Audit the set computation and claim

Back to contents

Reproducibility requires data versions, estimand code, constraint definitions, solver tolerances and endpoint checks. Optimisation failures can masquerade as narrow bounds. Verify active constraints, rerun boundary cases and use an independent implementation or analytic benchmark where feasible.

Create a claim ledger. For every sentence in the abstract, table and policy summary, record whether it requires one parameter value, all values through a range, the lower endpoint, the whole set or a confidence region. Link the sentence to the calculation that supports it.

Check monotonicity and nesting expected from the model. Unexpected contractions may reveal coding error or non-nested assumptions. Do not smooth them away. Preserve exploratory parameter ranges separately from the prespecified decision range.

First-principles check: Trace language to the exact set and coverage event it requires. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Audit one headline claim against saved endpoints and solver output. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Add a post hoc subgroup and identify which family and coverage statements change. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Archive data, code, constraints, tolerances, diagnostics, grids, sets, intervals and claim ledger. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

A geometry check helps readers understand the identified set. In one dimension it may be an interval; with several effects it may be a polytope, curved region or disconnected set. Reporting only coordinate-wise ranges can include combinations that are not jointly feasible. When a policy depends on several effects together, preserve the joint set or optimise the policy quantity over it. A rectangular summary may be conservative but should be labelled.

Sharpness should be audited constructively where possible. For candidate boundary points, exhibit or compute data-generating distributions that satisfy the model and reproduce the observed distribution. If the optimisation returns a point but no feasible witness, solver error or a relaxation may be hiding. Conversely, an outer bound can be useful when exact sharp computation is prohibitive, provided the report states that the true identified set may be smaller.

Sensitivity curves need simultaneous claim discipline. If the conclusion is “the lower bound remains positive for every c through 0.10,” pointwise intervals at each c do not automatically support the whole-range statement. Predefine the grid or continuum, use a valid uniform procedure where needed and include parameter selection or multiple outcomes in the protected family. This connects set-valued identification with the preceding sensitivity-band branch.

Calibration should be comparative, not magical. Use observed covariates to describe how much treatment assignment or outcomes change under measured differences, then compare that scale with c. Do not claim that an unobserved confounder must resemble an observed one. Present several benchmarks and explain which substantive mechanisms they approximate.

A decision table can show why a set is actionable. For each candidate action, compute worst-case benefit, worst-case harm or regret across the jointly feasible set. Add implementation cost and irreversibility. A pilot or information-gathering action may dominate both immediate adoption and rejection when the current set is wide. This is a legitimate result: partial identification can guide what to learn next.

Finally, plan evidence acquisition around set contraction. Identify which constraint determines the lower and upper endpoints. More sample size helps if endpoint uncertainty dominates; validation data, follow-up of missing outcomes, stronger design or a credible mechanism assumption may be needed if identification width dominates. State the expected contraction before collecting new data. A study should not spend heavily to reduce standard errors around endpoints that leave the policy conclusion unchanged.

Use a boundary-case test suite. Include no missing data, all outcomes missing in one arm, sensitivity equal to its baseline, sensitivity at the no-assumptions limit, a point-identified special case and a case with an empty model because restrictions conflict. The implementation should recover known results or fail transparently. Silent projection of an empty set into a plausible interval is a serious error because it conceals falsification of the maintained model.

Report feasibility alongside endpoints. If optimisation is numerical, save primal and dual residuals, constraint slack and the feasible witness at each boundary. Plotting endpoints without feasibility diagnostics can turn solver tolerance into apparent causal information. Re-run influential points with tighter tolerances and different starting values. Where analytic solutions exist for a simplified case, use them as regression tests for the full code.

Keep external validity separate. A sharply identified effect set for the study population may still transport only to a wider set of target-population effects under weak sampling assumptions. Do not add study identification and transport uncertainty as if they were independent standard errors. Build a second assumption ladder for sampling or setting variation, then optimise the target effect over the joint restrictions. The final claim should say whether uncertainty arises inside the experiment, in transport, or both.

11. Frequently asked questions

Back to contents

Is an identified set a confidence interval? No. It describes values compatible with population evidence and assumptions; a confidence procedure addresses sampling variability. Can I report the midpoint? You may as a descriptive convention only with clear justification, not as the identified effect.

Do bounds always shrink with more assumptions? Nested valid assumptions cannot enlarge the population set, but alternative or misimplemented models may not be nested. Does more data make bounds vanish? It reduces estimation uncertainty; identification width can remain. Is a sensitivity parameter the amount of actual bias? Not without separate identification or calibration.

What if the set crosses zero? A uniform sign claim is unsupported under that assumption. Can a decision still be made? Yes, using an explicit loss or robust decision rule. What should be published? Estimand, assumptions, sharp set, sensitivity domain, confidence target, computation, calibration and supported claim.

First-principles check: Answer by separating identification, estimation and decision. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Prepare a reader FAQ using one small numerical interval. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Explain the same result to a policy board seeking one yes-or-no choice. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Maintain definitions, examples, method limits, reviewed wording and links to analysis. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

12. The gold standard lets uncertainty remain set-valued

Back to contents

A strong partial-identification analysis begins with a causal estimand and the weakest credible restrictions. It derives the sharp set, then shows how transparent assumptions or sensitivity parameters contract that set.

Sampling uncertainty is added with a procedure whose coverage target is explicit. The analysis does not turn a set into a point by choosing its centre, and it does not describe a sensitivity index as observed bias. Decisions use rules applied across the compatible values.

The final test is linguistic: does the conclusion hold for every value and parameter setting it names? When it does not, the claim is narrowed or the assumption is strengthened openly. Set-valued results are not incomplete science; they are a precise record of what the study can and cannot determine.

First-principles check: End with the strongest claim shared by the entire relevant set. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than to hindsight.

Worked practice: Summarise the analysis as estimand, identified set, sampling uncertainty, decision and limit. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Imagine a larger sample halves standard errors but leaves the set width unchanged. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Publish the ladder, set-valued curve, confidence method, decision rule, code and explicit limit. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.