The Gold Standard of Experimental Design | Near-Minimum Relaxations, Tolerance Bands and Stability of Salvaged Conclusions

1. Begin with the refuted model and exact criterion

Back to contents

State the original restrictions, parameter space, evidence of incompatibility and relaxation objective. Minimum can mean fewest restrictions removed, least weighted removal, smallest slack or lowest loss. A near-minimum statement has no meaning until this metric and its units are fixed.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write the full model and three possible relaxation objectives. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: The sample criterion is nonzero although population refutation is uncertain. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Restrictions, data object, diagnostics, criterion, units, status and owner. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

2. Define the tolerance band before seeing conclusions

Back to contents

Choose an absolute, relative or substantively calibrated allowance around the minimum. Numerical solver tolerance is not automatically a scientific tolerance. Link the band to measurement error, prior plausibility, decision loss or sampling uncertainty, and report several defensible bands rather than one convenient cutoff.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Set bands for cardinality, weighted deletion and continuous slack. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: The exact minimum objective is zero. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Minimum value, band rule, rationale, scale, alternatives and prespecification. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

3. Enumerate every relaxation inside the band

Back to contents

Find all exact and near-minimum repairs, including ties, alternative assumption families and connected regions of continuous slack. Use search or optimisation certificates and test sensitivity to solver settings. Stopping after the first feasible repair understates model uncertainty.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Enumerate a toy system for objective values within five percent of minimum. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: There are millions of equivalent moment-level deletions. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Search space, grouping, algorithm, tolerances, certificates and retained repairs. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

4. Map each repair to parameter and decision sets

Back to contents

Recompute the identified or pseudo-true set for every admissible repair under a consistent construction. Track sharpness, components, gaps and decision functionals. A small objective increase can cross a threshold even when parameter sets appear close in a plotting scale.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Build a repair-by-bound-by-action matrix. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: One branch is an outer set and another sharp. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Repair, assumptions, objective, set construction, sharpness, decisions and code. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

5. Separate structural closeness from conclusion closeness

Back to contents

Two relaxations can be nearly equal by objective while implying disjoint conclusions; conversely, structurally different repairs can support the same action. Report both distances. Do not average incompatible estimates or convexify a disconnected union merely to produce one interval.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compare objective distance, set distance and action agreement. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: A rare near-minimum branch reverses a high-consequence decision. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Metrics, branches, set relations, actions, consequences and interpretation. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

6. Worked example: a two-percent band changes the action

Back to contents

The exact minimum drops restriction A and supports a positive effect. A repair with objective only two percent larger relaxes B and includes zero; another at three percent supports a negative region. The analysis reports the exact-minimum claim as branch-specific, shows the widening union and identifies “collect more evidence” as the only action robust across the prespecified five-percent band.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compute the stability path from zero to ten percent tolerance. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Weight uncertainty makes B exactly tied with A. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Criteria, weights, repairs, sets, actions, band path and conclusion. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

7. Build a stability path instead of one cutoff

Back to contents

Vary the tolerance continuously or over a prespecified grid and record when new repair families enter, set topology changes and actions switch. The breakpoints are often more informative than a yes-or-no robustness label. Report intervals of stability and the assumptions introduced at each transition.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Create a breakpoint table and stability curve. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Several branches enter at the same threshold. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Grid, entry points, branch labels, topology, actions and reproducibility. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

8. Add uncertainty in weights and grouping

Back to contents

Weights encode the relative cost or credibility of relaxing assumptions; grouping determines whether many moments count as one substantive restriction. Reanalyse plausible weight sets and group definitions. Near-minimum bands should not hide a result that depends on arbitrary duplication or scaling.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Perturb weights and regroup moments by assumption family. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: One family generates hundreds of sample moments. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Weight sources, ranges, group rules, alternatives, effects and retained claims. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

9. Separate sampling uncertainty from compromise tolerance

Back to contents

Sampling changes moment estimates, the minimum value, branch membership and set boundaries. Resample the full selection-and-mapping procedure where valid, rather than holding the observed repair fixed. Report how often conclusions are stable across both sampling and tolerance.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Bootstrap the near-minimum frontier and action set. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: A branch enters only in a small but consequential tail. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Resampling plan, branch frequencies, thresholds, sets, tail cases and limits. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

10. Use external evidence to rank without erasing branches

Back to contents

Validation studies, measurement knowledge or institutional facts can make some relaxations less credible. State this evidence independently of the estimated outcome and show the ranking rule. Retain excluded branches in the sensitivity record when the external evidence is uncertain or population-specific.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Apply an evidence rubric to three near-minimum families. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Validation supports one assumption only in a subgroup. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Sources, timing, scope, rubric, branch effect, exclusions and uncertainty. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

11. Audit tolerance, search and public reporting

Back to contents

Verify objective scaling, all minima, near-minimum coverage, solver convergence, grouping, weight provenance and branch-specific set calculations. Public summaries should distinguish exact-minimum, band-robust and evidence-ranked claims. A stable headline must not be manufactured by suppressing close alternatives.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Reproduce the frontier with an independent implementation. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Rounding in a table changes which repairs appear admissible. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Code, data, settings, certificates, branches, reports, discrepancies and corrections. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

12. The gold standard asks how much compromise a conclusion can bear

Back to contents

Model salvage is credible when the result is stable to defensible ways of measuring and slightly widening the repair. Strong practice prespecifies the tolerance scale, enumerates every admissible branch, maps sets and actions, includes sampling and weight uncertainty and reports breakpoints.

First-principles check: Name the decision, the people, material or population affected, the evidence needed and the limit that remains. Write the rule before looking at the preferred result. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write a report with exact, near-minimum and decision-robust layers. Keep every intermediate step visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Repeat when two minimality criteria cannot share one scale. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Refutation, criteria, bands, searches, sets, stability, evidence, decisions and audit. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.