The Gold Standard of Experimental Design | Moment Inequalities, Constraint Selection and Calibrated Confidence Regions

1. Translate the design into population moment inequalities

Back to contents

A moment inequality has the form that an expected function of data and candidate parameter is at least or at most zero. The function must come from the design or maintained model: random assignment, monotonicity, utility optimisation, capacity or another defensible restriction.

Write every inequality with sign, conditioning set, unit and domain. A coding reversal can invert the feasible set. Avoid generating constraints simply because they improve precision in the observed sample. The family of moments is part of the analysis plan.

State which assumptions are testable, which are maintained and which parameters remain nuisance quantities. A mathematical inequality is only as meaningful as the design argument that produced it.

First-principles check: Derive each population restriction from an explicit design or model assumption. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write three inequalities for a simple capacity-constrained choice problem. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Reverse one monotonicity assumption and redraw the feasible set. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep derivation, sign convention, conditioning variables, parameter space and assumptions. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

2. Separate the identified set from a confidence region

Back to contents

The identified set contains parameter values compatible with the population restrictions. It may be an interval, polygon, curved region or disconnected set. A confidence region accounts for sampling uncertainty about that set or about the true parameter under the maintained model.

Do not call the sample-feasible region a confidence set. Sample moments fluctuate, so a candidate outside the raw sample region may still be statistically plausible, and one inside may rely on noisy slack. Define whether coverage targets the true parameter, the whole identified set or another object.

Visualise both layers where possible. The distinction prevents a narrow sample intersection from being sold as certainty and keeps partial identification separate from estimation error.

First-principles check: Name the object covered before choosing a calibration method. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Plot a population rectangle, sample analogue and 95% confidence region for two parameters. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Use a disconnected identified set created by a nonconvex restriction. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain target object, parameter space, sample analogue, coverage statement and visualisation. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

3. Standardise moments without erasing weak information

Back to contents

Moments measured on different scales require standardisation. Studentised statistics compare estimated violations with their standard errors, but tiny estimated variance can make a nearly deterministic constraint dominate. Variance estimation must match clustering, dependence and the sampling design.

Prespecify floors or regularisation for unstable variances. Check whether rare cells, extreme weights or simulation noise create misleading precision. Correlated moments should not be treated as independent evidence merely because each is separately standardised.

Report the raw moment, standard error, standardised value and effective sample support. Constraint selection is easier to understand when readers can see whether a moment is informative because it is close to binding or merely because its denominator is fragile.

First-principles check: Standardise by design-valid uncertainty while preserving the raw scale and support. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compute studentised moments for three constraints with different variances. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Add cluster sampling that doubles one standard error. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep raw estimates, variance method, dependence, floors, weights and effective sample size. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

4. Understand slack, binding and nearly binding constraints

Back to contents

A slack population inequality lies safely inside the allowed region and contributes little to the boundary near the truth. A binding moment equals zero. Nearly binding moments are close enough that sampling noise makes their role uncertain. In finite samples, their classification is not observed.

Least-favourable calibration treats many moments as potentially binding and can be conservative. Aggressive selection can improve power but risk deleting a constraint that matters. The method must balance these errors uniformly over plausible data-generating processes.

Use selection terminology carefully. A discarded slack moment is not “false”; it is uninformative for the local critical value under the stated rule. Keep the full moment family in the model record even when calibration focuses on a selected subset.

First-principles check: Treat binding status as uncertain and calibration-relevant. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Classify simulated moments under three selection thresholds and compare retained sets. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Move one population moment toward zero at the local sampling rate. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain full family, threshold, selected subset, standardised values and sensitivity. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

5. Use generalized moment selection deliberately

Back to contents

Generalized moment selection uses preliminary sample information to estimate which inequalities are likely to bind, then calibrates the test using those moments. Properly designed procedures can reduce conservatism while maintaining uniform size control under their assumptions.

Selection and testing are not independent because they use the same data. Valid procedures account for that dependence through their asymptotic or bootstrap construction. A home-made rule that drops all favourable moments and then applies an ordinary critical value can undercover severely.

Specify the selection function, tuning sequence, test statistic and critical-value method. Show results under least-favourable and reasonable alternative selection rules. Precision earned only under one opaque tuning choice should be presented as fragile.

First-principles check: Use a selection rule whose coverage properties include the selection step. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compare least-favourable and generalized moment-selection critical values for one candidate parameter. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Increase the number of clearly slack moments from five to fifty. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep tuning, selection function, statistic, bootstrap or limit method, seeds and coverage target. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

6. Worked example: capacity bounds with four markets

Back to contents

A study estimates a demand parameter constrained by four market-capacity inequalities. Two sample moments are close to zero, one is strongly positive and one appears mildly negative. The raw sample intersection excludes the policy-relevant value, but the negative moment has a large clustered standard error.

The analyst computes a max studentised violation statistic. Least-favourable calibration includes all four moments and yields a broad confidence interval. A prespecified generalized moment-selection rule drops the clearly slack moment while retaining the two near-binding and the uncertain negative moment. The calibrated interval is narrower but still includes the policy value.

Sensitivity analysis changes the tuning rule and cluster variance. The conclusion survives moderate changes but fails when the disputed market is treated as independent observations. The final report therefore attributes uncertainty to the design’s cluster structure, not to an arbitrary preference for a wider interval.

First-principles check: Let the calibrated joint violation, not the raw sample intersection, determine inclusion. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Calculate the four standardised moments, selected set and max statistic. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Make the disputed market’s variance half as large. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Preserve market moments, clusters, selected constraints, critical values, intervals and sensitivity. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

7. Calibrate the joint critical value

Back to contents

With several inequalities, a confidence region cannot use separate one-sided cut-offs without considering multiplicity and correlation. Max or quadratic statistics combine violations differently. Bootstrap, simulation or analytic methods approximate the joint reference distribution under stated conditions.

Use the same covariance and dependence structure that generated the sample. In high dimensions, multiplier bootstrap methods may be appropriate; in smaller structured problems, direct simulation or specialised asymptotics may be used. Numerical optimisation error must be smaller than the inferential tolerance.

Validate code on cases with known solutions and report Monte Carlo error. A confidence boundary that moves materially with simulation seed or optimiser start is not calibrated enough for a precise substantive claim.

First-principles check: Calibrate the joint statistic under the design’s dependence structure. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Simulate a critical value for four correlated standardised moments. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Change correlations from positive to mixed signs. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep statistic, covariance, resampling design, draws, seeds, convergence and Monte Carlo error. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

8. Changed-context transfer: many conditional moments

Back to contents

Conditional inequalities can generate many or infinitely many unconditional moments through instruments, bins or basis functions. The logic transfers, but dimension, approximation and multiple-testing burdens grow. Data-driven partitions can overfit apparent violations.

Prespecify or honestly account for instrument construction. Check whether sparse cells and extreme basis functions create unstable standard errors. Selection may need to remove both clearly slack and weakly informative constraints while preserving uniform validity.

What does not transfer automatically is the finite-moment critical value. Use methods designed for many inequalities and report how the number of moments grows relative to sample size. The richer constraint system can sharpen identification only if calibration keeps pace.

First-principles check: Preserve validity while expanding the moment dictionary. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Generate conditional moments from four bins and compare them with two smooth basis functions. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Allow the number of instruments to exceed sample size. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain dictionary construction, dimension, sparsity, variance screening, selection and calibration. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

9. Know the limits and common misconceptions

Back to contents

More inequalities do not always produce more credible precision. Redundant moments can add computation without information; noisy moments can inflate critical values; selected moments can create undercoverage if the procedure is not valid. A nonempty confidence region does not prove every maintained assumption.

An empty region may indicate model misspecification, sampling fluctuation, numerical error or a parameter-space problem. Diagnose before declaring theory rejected. Likewise, a wide region may be the honest consequence of weak identification rather than a failed method.

Asymptotic coverage does not guarantee good finite-sample behaviour in a small clustered design. Simulations should resemble the design, moment dependence and near-binding configurations that matter. Report power and coverage trade-offs rather than calling one rule optimal in all settings.

First-principles check: Distinguish information, validity, computation and model adequacy. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Diagnose an empty confidence region through four possible failure paths. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Reduce the number of clusters while keeping total observations fixed. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep diagnostics, numerical tolerances, simulations, rejected assumptions and scope limits. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

10. Audit the full constraint-to-region pipeline

Back to contents

The audit joins the design argument, moment coding, sample, variance estimation, parameter grid or optimiser, selection, joint calibration and reported region. Reproduce every boundary point from versioned code and data.

Run sign and unit tests, least-favourable comparisons, alternative valid tuning values, variance sensitivities and denser optimisation. Inspect which moments bind along each part of the boundary. A region can look stable in size while its substantive support shifts among constraints.

Report computation separately from inference. Record grid resolution, solver tolerances and failed starts. If approximation error could change inclusion, widen or qualify the region rather than presenting numerical output as exact.

First-principles check: Trace each confidence-boundary claim back to moments, uncertainty, selection and calibration. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Reproduce one boundary point by hand from stored intermediate values. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Double grid resolution and change optimiser initialisations. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Archive derivations, data, code, environments, selected moments, critical values, diagnostics and region files. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

11. Frequently asked questions

Back to contents

Why not keep every moment? Least-favourable methods can, but may be conservative when many moments are clearly slack. Why not drop every positive sample moment? Sampling error can make a binding moment look slack; unadjusted selection invalidates coverage. Is the identified set the same as a confidence set? No.

Can the region be disconnected? Yes. Does a narrower region mean a better method? Only if coverage and assumptions remain credible. What if no parameter passes? Investigate model, data, variance and computation before interpreting rejection.

Should one tuning rule be reported? Report the prespecified primary rule and informative valid sensitivities. Readers need to see whether the substantive claim depends on a narrow calibration choice.

First-principles check: Answer by separating population identification, sample uncertainty and computational approximation. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Prepare a decision guide for choosing least-favourable, selected and many-moment procedures. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Explain the guide to a policy reader without using asymptotic shorthand. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Maintain reviewed questions, definitions, references, examples and software version. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

12. The gold standard makes every constraint accountable

Back to contents

Moment inequalities are powerful because they let designs speak without forcing false point identification. Their honesty depends on showing how theoretical restrictions become coded moments and how sampling noise affects the apparent feasible set.

The gold standard standardises with design-valid uncertainty, treats binding status as unknown, uses selection rules with valid calibration and tests the joint system rather than isolated constraints. It verifies computation and exposes which moments shape the region.

The final claim is calibrated rather than merely feasible. Readers can see the identified object, coverage target, selection choice, uncertainty and limits. Partial identification then becomes a disciplined statement of what the design can support, not an excuse for a vague interval.

First-principles check: Require every reported boundary to survive a transparent identification and calibration chain. Write the answer before looking at the preferred outcome. Then identify the rule, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write the final claim for one parameter region with assumptions and sensitivities attached. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Use equality and inequality moments together in a new design. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep the final specification, code, simulations, region, interpretation, peer review and revisions. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a practical transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.