The Gold Standard of Experimental Design | Minimum-Relaxation Sets, Pseudo-True Parameters and Best-Fitting Misspecified Models

1. Keep the rejected model and alternative target separate

Back to contents

If no parameter satisfies the maintained restrictions, the original identified set is empty. A minimiser of violations is not a member of that set and should not be described as the parameter the original model identifies.

Define a new target explicitly: the parameter or set minimising a stated population loss under a specified scaling and weight. Preserve the specification test and diagnostic evidence alongside the alternative estimate so that usefulness is not confused with model validity.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Rewrite a report that silently substitutes a minimum-violation estimate for an empty set. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: A confidence region is empty but the raw sample intersection is nearly feasible. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep original model, rejection result, diagnostic, alternative target, rationale and label. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

2. Define what is being relaxed

Back to contents

Relaxation can add a common slack to every inequality, family-specific slacks, relative tolerances, bounded assumption departures or removal of a minimum number of restrictions. These answer different questions and can lead to disjoint solutions.

Tie each relaxation to a substantive mechanism where possible. A common numerical slack may be convenient but can hide that measurement error, approximation and behavioural restrictions live on different scales. Do not infer the true failure mechanism merely because one relaxation is smallest.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Construct common, family-specific and discrete relaxations for four conflicting inequalities. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: One restriction is measured in dollars and another in probabilities. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain relaxation definition, units, mechanism, admissible range, alternatives and limits. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

3. Scale moments before comparing violations

Back to contents

Unscaled violations can be dominated by moments with large units or variance. Standardisation by standard error, substantively meaningful tolerance or another prespecified scale changes the geometry and therefore the minimiser.

Estimated scales introduce uncertainty and instability when variances are near zero. Record floors, regularisation and whether scaling is fixed or parameter-dependent. Compare plausible scales and treat large changes as evidence that “best fitting” is convention-dependent.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Calculate minimum violations under raw, variance-scaled and policy-tolerance scales. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: A near-deterministic moment receives an enormous inverse-variance weight. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep units, scale formula, estimates, floors, diagnostics, sensitivity and decision. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

4. Choose a loss and understand its geometry

Back to contents

Maximum violation minimises the worst constraint, squared loss spreads attention across many deviations and absolute loss produces different trade-offs. Weighting by covariance or substantive importance changes the result again. None is automatically correct under misspecification.

State whether the loss is descriptive, decision-based or derived from an inferential framework. Inspect whether it is convex, whether multiple local minima exist and which constraint families drive it. A precise optimiser output does not remove the normative content of the loss.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Solve a two-parameter example under maximum, absolute and squared violation losses. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Add ten correlated copies of one constraint family. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep loss formula, weights, geometry, optimiser, minima, active constraints and rationale. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

5. Find the minimum-relaxation set, not just one point

Back to contents

The smallest relaxation needed for feasibility may support a set of parameters, not a unique point. Report the entire minimum-relaxation set or a justified summary when possible. Selecting one convenient member can create artificial certainty.

Check closedness, boundaries and numerical tolerance. When several relaxation vectors tie, preserve their union and show which restrictions each one relaxes. Nonuniqueness is substantive evidence that the data and model do not determine a single compromise.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Find all minimisers in a linear inequality example and map the active faces. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Two distinct relaxation patterns yield disjoint parameter regions. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain relaxation threshold, set representation, algorithms, tolerances, nonuniqueness and active constraints. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

6. Worked example: incompatible capacity bounds

Back to contents

Two markets imply theta must be at least 0.62 and at most 0.55. Under a common absolute slack, the minimum value is 0.035 and theta equals 0.585. That midpoint looks natural only because both inequalities use the same scale and equal weight.

If the lower-bound moment is twice as uncertain, variance scaling shifts the compromise. If the upper bound is a hard engineering limit while the lower is an approximate behavioural restriction, a decision-based relaxation may preserve the upper bound and relax only the lower. The original model remains refuted in every case; the different pseudo-true targets describe different compromise rules.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compute the minimiser under equal, variance and asymmetric substantive weights. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Replace one bound with a nonlinear curve producing two minimising regions. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Preserve constraints, scales, losses, solutions, rejection, interpretation and comparative sensitivity. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

7. Interpret pseudo-true parameters cautiously

Back to contents

A pseudo-true parameter minimises a population objective within a misspecified model. It need not equal a causal, structural or policy quantity of interest. Its relevance depends on the decision problem and how the loss relates to consequences.

Explain what population features the objective prioritises and what changes under alternative misspecification. Avoid attaching the original parameter’s name and causal interpretation automatically. Where possible, connect the target to an explicit prediction or decision and test external relevance.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write substantive interpretations for two pseudo-true values generated by different losses. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: A value predicts outcomes well but violates the original causal exclusion restriction. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep target definition, population objective, substantive mapping, unsupported meanings and validation. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

8. Conduct inference under misspecification

Back to contents

Sampling uncertainty remains after the population target is redefined. Standard confidence procedures built for a correctly specified nonempty set can show spurious precision when the model is misspecified. Use methods whose guarantees match the pseudo-true target and diagnostic setting.

Account for estimated weights, selected constraints and possible nonunique minima. Report specification evidence separately from uncertainty around the alternative target. A narrow interval around a pseudo-true parameter does not rehabilitate the original assumptions.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compare a conventional confidence set with a misspecification-robust procedure in simulation. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: The minimum-relaxation set changes dimension across samples. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain inferential method, assumptions, selection steps, simulations, coverage evidence and interpretation. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

9. Test robustness across defensible relaxations

Back to contents

No single sensitivity run is enough when the compromise depends on scale and loss. Prespecify a small family of scientifically defensible relaxations and compare parameter sets, rankings and decisions.

Separate numerical stability from conceptual robustness. An optimiser can reproduce the same answer perfectly under one arbitrary loss while plausible alternative losses reverse the conclusion. Report the intersection, union or decision envelope according to the question, without implying that agreement was guaranteed.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Create a robustness grid crossing three scales and three losses. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: One policy choice survives even though point estimates move widely. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep the grid, reasons for inclusion, outputs, decision envelope, reversals and unresolved choices. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

10. Audit optimisation, coding and hidden conventions

Back to contents

Minimum-relaxation analysis adds layers where silent choices can enter: sign, scale, weight, parameter bounds, norm, penalty, solver, tolerance and tie-breaking. Reproduce hand-solvable cases and use multiple algorithms or certificates where available.

Record all local minima and failed runs. Inspect whether parameter bounds create the apparent optimum and whether duplicated or highly correlated moments receive unintended extra weight. Version code and preserve the original empty-model result.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Audit an optimisation script against three known geometric examples. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Changing the parameter box moves the minimiser to a boundary. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Archive code, tests, inputs, conventions, solvers, diagnostics, failures and environment. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

11. Frequently asked questions

Back to contents

Is the minimum-violation point the least wrong parameter? Only relative to a defined loss and scale. Does a pseudo-true value estimate the original causal effect? Not automatically. Can the minimum-relaxation set be empty? With well-defined compact problems it may exist, but bad domains or algorithms can still fail.

Should every conflict be averaged? No. Can one choose the loss after seeing which answer looks reasonable? That invites outcome-driven specification. What should policy use? Conclusions that are decision-relevant and robust across defensible alternatives, with model rejection still visible.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Prepare a collaborator-facing explanation of pseudo-truth without jargon. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Explain why two correct optimisers can return different best-fitting values. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Maintain reviewed answers, examples, cited methods, escalation routes and revision date. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

12. The gold standard makes compromise explicit

Back to contents

A best-fitting misspecified model is an alternative analytical object, not a repaired original. Strong practice defines the relaxed restrictions, units, scale, loss and weights before optimisation; finds the whole minimum-relaxation set; interprets pseudo-true targets only within that construction; and uses inference designed for misspecification.

Robustness spans credible relaxations, not just solver settings. The result may still support a decision, but its authority comes from transparent compromise and demonstrated stability, never from relabelling the nearest point as if all assumptions held.

First-principles check: State the decision this chapter protects, the evidence needed for it and the limit that remains. Write the rule before looking at the preferred outcome. Then identify the authority, observation or calculation that could change the answer. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write a final report that keeps rejection, alternative target, uncertainty and decision robustness separate. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Repeat when domain experts disagree about which restriction family is approximate. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep original model, relaxations, losses, sets, inference, sensitivities, decisions and review. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.