Make the inference visible before trusting it
When a refuted model can be repaired under several incomparable criteria, one weighted score can hide real disagreement. This guide builds a Pareto frontier and asks which conclusions survive every defensible compromise.
When a refuted model can be repaired under several incomparable criteria, one weighted score can hide real disagreement. This guide builds a Pareto frontier and asks which conclusions survive every defensible compromise. The practical aim is not to force one neat answer. It is to show what is known, what remains conditional and what action is justified now.
A gold-standard process protects the chain from evidence to inference. It uses original worked examples, changed-context transfer checks, explicit limits and a record that another person can test. Readers can enter through the closest route below or move chapter by chapter.
Choose the route closest to your decision
Chapter contents
Route 1: chapters 1–3
Route 2: chapters 4–6
Route 3: chapters 7–9
1. Begin with refutation, not optimisation
Establish that the maintained restrictions are incompatible with population evidence or rejected beyond a declared tolerance. A frontier is meaningful only after the conflict is explicit. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Which restrictions fail together, and under what sampling uncertainty? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Four moment inequalities have no common parameter value at the chosen tolerance, so the analysis documents the empty set before proposing repairs. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the check to a randomised experiment with noncompliance by separating design failure from model misspecification. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. A finite-sample empty estimate can differ from population refutation; do not collapse them. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Record restrictions, data version, test or diagnostic, tuning choices and uncertainty around refutation. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
2. List relaxation criteria in their native units
State each admissible repair and its cost without forcing premature conversion: number of restrictions removed, maximum slack, total weighted slack, probability contamination or domain-specific loss. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. What does one unit of each criterion mean in the scientific story? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Dropping one exclusion restriction and allowing a 0.04 probability slack are recorded separately because they are not naturally exchangeable. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the inventory to measurement models where bias, variance and missingness adjustments use different scales. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. A criterion that is mathematically convenient may be substantively meaningless. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Keep formulas, units, admissible range, interpretation, symmetry assumptions and excluded repairs. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
3. Refuse meaningless addition
Do not add incommensurable costs until a defensible preference model or conversion is supplied. Retain the vector of costs. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Would changing units alter the selected repair? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Measuring slack in percentage points rather than proportions changes a weighted sum unless weights are transformed; the Pareto relation remains unchanged under positive monotone rescaling. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the test to cost, time and ethical burden in an adaptive trial. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Normalising all criteria to zero-to-one does not create a value-free common unit. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Record scaling, transformations, any elicited weights and sensitivity to alternative normalisations. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
| Checkpoint | Question |
|---|---|
| Object | Would changing units alter the selected repair? |
| Worked comparison | Measuring slack in percentage points rather than proportions changes a weighted sum unless weights are transformed; the Pareto relation remains unchanged under positive monotone rescaling. |
| Transfer | Transfer the test to cost, time and ethical burden in an adaptive trial. |
| Boundary | Normalising all criteria to zero-to-one does not create a value-free common unit. |
4. Define dominance and the frontier
A repair dominates another when it is no worse on every declared criterion and strictly better on at least one. The Pareto frontier contains repairs not dominated within the admissible set. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Can every excluded repair be paired with a feasible repair that is at least as good on all costs? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Repair R1 drops one restriction with slack 0.03; R2 drops two with slack 0.05, so R2 is dominated. R3 drops none with slack 0.08 and remains nondominated. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the definition to interval widths and assumption departures. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Pareto efficient does not mean good, true or acceptable; it means not dominated under chosen criteria. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Keep the candidate set, feasibility checks, dominance comparisons, ties and numerical tolerances. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
5. Map the criterion-specific identified sets
For each frontier repair, compute the parameter or conclusion set it supports. The object of interest is both the cost frontier and the induced inference. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Do nearby cost vectors imply nearby scientific conclusions? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Two nondominated repairs have similar total slack but imply opposite signs for the treatment effect; the sign is therefore not frontier-robust. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the mapping to policy thresholds rather than point estimates. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Smooth cost changes need not produce smooth identified sets when active constraints switch. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Record active restrictions, identified sets, solver diagnostics and conclusion summaries for every frontier region. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
| Checkpoint | Question |
|---|---|
| Object | Do nearby cost vectors imply nearby scientific conclusions? |
| Worked comparison | Two nondominated repairs have similar total slack but imply opposite signs for the treatment effect; the sign is therefore not frontier-robust. |
| Transfer | Transfer the mapping to policy thresholds rather than point estimates. |
| Boundary | Smooth cost changes need not produce smooth identified sets when active constraints switch. |
6. Work a three-criterion example
Compare cardinality, maximum slack and weighted total slack without declaring one supreme. Enumerate nondominated repairs and intersect their supported claims. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Which claims hold at every defensible frontier point? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Repair A removes one moment and supports a positive effect; B keeps all moments with moderate maximum slack and includes zero; C has low weighted slack but a negative lower bound. Only a bounded magnitude claim survives all three. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Change the weight vector and confirm which repairs enter or leave the frontier. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. An intersection can be empty because criteria encode genuinely conflicting compromises, not because computation failed. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Save the table of repairs, costs, induced sets, robust intersection and reasons for any criterion exclusion. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
7. Separate analyst values from model facts
Weights, caps and admissible repairs express priorities. Elicit them openly from decision owners or report results across a range. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Which choice is empirical, which is mathematical and which is normative? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. A regulator forbids relaxing a safety restriction but permits modest measurement slack; this rule changes the feasible frontier and is labelled as policy input. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the separation to an ethics committee that caps contamination assumptions. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Stakeholder input cannot turn an unsupported assumption into evidence. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Record who set each rule, conflict-of-interest controls, alternatives considered and the empirical results before preferences were applied. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
8. Build conclusions robust to every nondominated repair
Classify claims as frontier-robust, region-specific or contradicted. Prefer bounded statements that survive the full admissible frontier. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Would the decision change at another nondominated point? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Every frontier repair rules out a large benefit, although the sign varies. The defensible conclusion is no evidence for a benefit above the policy threshold. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the method to ranking treatments where the winner changes but one option is never best. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. A very broad robust statement can be unhelpful; report what is learned and what remains unresolved. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Keep claim logic, frontier subset, threshold, exceptions and any loss from using a robust decision. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
| Checkpoint | Question |
|---|---|
| Object | Would the decision change at another nondominated point? |
| Worked comparison | Every frontier repair rules out a large benefit, although the sign varies. The defensible conclusion is no evidence for a benefit above the policy threshold. |
| Transfer | Transfer the method to ranking treatments where the winner changes but one option is never best. |
| Boundary | A very broad robust statement can be unhelpful; report what is learned and what remains unresolved. |
9. Carry sampling uncertainty through the frontier
Estimated costs and dominance can change with samples. Use confidence procedures, resampling or sensitivity bands appropriate to the model. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Is a repair apparently dominated only because of estimation noise? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. Bootstrap draws move two repairs across the frontier; the analysis reports a frontier band and withholds a unique ranking. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the uncertainty to clustered or dependent data with an appropriate resampling design. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Applying pointwise confidence intervals independently can miss joint uncertainty and selection effects. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Record sampling scheme, critical values, tuning, simulation error and coverage target. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
10. Stress-test numerical and specification choices
Vary solver tolerance, grid resolution, constraint selection, criterion bounds and transformations. Confirm that the reported frontier is not a computational artefact. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Do conclusions persist under reasonable implementation changes? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. A coarse grid misses a narrow nondominated region whose identified set includes zero; refinement changes the robust claim. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the test to nonconvex problems using multiple starts or certified bounds. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Solver convergence messages are not proofs of global optimality. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Keep code version, seeds, tolerances, diagnostic plots, failed runs and independent replication where feasible. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
| Checkpoint | Question |
|---|---|
| Object | Do conclusions persist under reasonable implementation changes? |
| Worked comparison | A coarse grid misses a narrow nondominated region whose identified set includes zero; refinement changes the robust claim. |
| Transfer | Transfer the test to nonconvex problems using multiple starts or certified bounds. |
| Boundary | Solver convergence messages are not proofs of global optimality. |
11. Answer the questions decision-makers ask
Explain why there is more than one best repair, why weights matter, what Pareto robust means and when a narrower conclusion requires an explicit preference. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Can the answer be stated without pretending the frontier identifies the true model? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. A manager asks for one estimate; the analyst shows the range across frontier repairs and states which weight judgement would select each point. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the explanation to students using two everyday costs that cannot be naturally added. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. Do not use Pareto language to avoid making an authorised decision when a clear preference is legitimately available. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Publish concise questions, examples, decision rights, uncertainty and review triggers. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
Can one rule settle every case?
No. Use a declared rule for the present purpose, then test whether a changed purpose, scale or decision threshold requires a different answer.
Does more data always improve the decision?
No. More data can duplicate the same blind spot, add poorly controlled observations or create false confidence if provenance and uncertainty are weak.
When should work pause?
Pause an irreversible decision when a plausible unresolved condition could change it and when waiting or gathering targeted evidence is proportionate to the risk.
What makes an answer auditable?
A reviewer must be able to find the inputs, transformations, assumptions, alternatives, authority, decision and residual uncertainty without relying on memory.
12. Audit the frontier and the resulting decision
Archive the original refutation, criteria, frontier, preferences, robust conclusions and downstream action. Revisit when evidence or objectives change. In experimental and observational models whose restrictions conflict with the data, the central claim is which conclusions remain defensible across nondominated relaxations measured by different, incommensurable costs. This chapter turns that claim into a bounded question so that evidence is not collected without knowing the decision it must support.
First-principles check. Could an independent analyst reproduce both the frontier and the policy choice? Start with the object, pathway or restriction that could make the answer wrong. State the comparison, unit, time window and authority before calculating a score or accepting a familiar label. A result that cannot be traced to those choices is not yet a defensible result.
Worked practice. A later dataset removes one conflict; the team rebuilds rather than merely updating the old preferred repair. The useful move is to expose the intermediate reasoning: what was observed, what was inferred, which alternative explanation remained possible and which action followed. That sequence makes the example reviewable and prevents the final answer from carrying more certainty than the evidence.
Changed-context transfer. Transfer the audit to sequential experiments by versioning each frontier over time. Keep the reasoning structure but replace the local quantities, controls and decision threshold. If the conclusion survives only because the old context supplied a hidden assumption, it has not transferred; it has merely been copied.
Limit and misconception. A stable decision can rest on a changing frontier, so both must be reviewed. A common mistake is to treat the most legible number, newest file or neatest optimisation as decisive. The gold standard is to identify what the evidence cannot establish, especially where an arbitrary scale or weight chooses the preferred repair and presents a value judgement as a mathematical necessity.
Evidence record. Keep reproducible inputs, outputs, approvals, residual disagreements and verified routes to related methods. Add the source, date, operator or analyst, method version, exclusions and uncertainty. Link raw observations to the derived judgement without overwriting earlier states, so a later reviewer can reproduce the chain and challenge a step without losing the record.
Decision-quality drill. Before closing this chapter, write one sentence naming the supported decision and one sentence naming the strongest unsupported claim. Then change one plausible condition and ask whether the action reverses. If it does, label the decision sensitive and set a review trigger rather than hiding the boundary.
Continue through verified routes
Use the core routes to return to the wider Gold Standard framework, then follow the specialist articles that extend this decision without duplicating it. The external sources below define or test the authoritative concepts used here.
Core routes
Authoritative sources
- Li: Discordant relaxations of misspecified models
- Andrews and Kwon: Misspecified moment inequality models
- Marmer and Otsu: comparison under a chosen measure of fit
- Multi-objective polynomial optimisation and Pareto points
Evidence and guidance can change. Recheck the linked source, the local data and the current decision context before carrying the conclusion into a new case.