Find your next learning step
A counterfactual result can stop travelling even when the archived simulator still runs perfectly. The target queue may lose overlap with the source, acquire a new feedback loop or attach a different cost to the same delay. When that happens, escalation should narrow or suspend the claim. A shadow baseline can remain useful as a non-authoritative comparator, while benchmark retirement closes the old decision role without destroying the archive or its learning value.
Open the chapter contents · Queueing Theory root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Recognise a transport failure
A transport failure occurs when evidence no longer supports carrying a source-setting intervention contrast into the target setting. Typical signals include missing overlap, reversed policy ranking, new service classes, unstable effect modifiers or errors concentrated where the operational decision matters.
Do not confuse transport failure with software failure. A reproducible archive may still answer historical questions. The failed object is the current decision claim, not necessarily the code, data or every scientific insight behind it.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Escalate according to the threatened decision
Link each trigger to an action: investigate, narrow the population, lower authority, suspend one policy comparison or retire the benchmark. Name who decides and how quickly. A dramatic diagnostic with no decision pathway is only an alert.
Use consequence as well as statistical size. A small shift near a capacity-switch boundary can matter more than a larger shift where every reasonable policy chooses the same action.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Build a transport-failure escalation register
A clinic transports an archived surge-capacity comparison into a queue that now includes teleconsults and same-day priority.
| Signal | Observed state | Decision exposure | Escalation | Interim rule |
|---|---|---|---|---|
| Covariate overlap | Teleconsult share outside archive | High | Suspend transport | Target data only |
| Policy ranking | Reverses in 3 of 8 weeks | High | Independent review | No automatic switch |
| Wait calibration | Within 6% | Medium | Continue monitoring | Diagnostic use |
| Overtime tail | Understates 4 of 8 weeks | High | Block cost claim | Report range |
| Routine in-person stratum | Stable overlap and ranking | Low | Narrow claim | Bounded comparison |
The broad transport claim is suspended, but a narrow routine in-person comparison remains available with its limitations visible.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Work a shadow-baseline decision
The retired benchmark runs beside the target-only model for six weeks. It cannot control capacity; it can expose disagreement and support diagnosis.
| Week | Shadow action | Live action | Estimated regret gap | Use |
|---|---|---|---|---|
| 1 | Open 4 rooms | Open 4 rooms | 0.1 units | Agreement control |
| 2 | Open 5 rooms | Open 4 rooms | 1.8 units | Review overtime trade-off |
| 3 | Open 3 rooms | Open 5 rooms | 4.6 units | Escalate missing feedback |
| 4 | Open 4 rooms | Open 4 rooms | 0.2 units | Agreement control |
| 5 | Open 5 rooms | Open 5 rooms | 0.5 units | Monitor |
| 6 | Open 3 rooms | Open 5 rooms | 5.1 units | Retirement evidence |
Shadow output stays labelled and access-controlled so it cannot become an unofficial recommendation. Agreement weeks are retained; they help detect a pipeline that reports disagreement everywhere.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Govern a shadow baseline without restoring authority
Freeze the shadow version, inputs, transformation, schedule and output audience. Log who views it and what actions follow. If operators routinely copy its recommendation, the system has authority in practice even if a document calls it diagnostic.
Set an end date and a question. Useful shadow questions include whether a new feedback mechanism explains disagreement or whether a safe subset still exists. Endless shadow operation can waste review effort and blur accountability.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Retire the benchmark decision role
Retirement should specify what ends: target performance comparison, capacity recommendation, cost claim or fallback use. Preserve the historical package, validation record, failures, permitted research uses and a pointer to its successor.
Useful criteria include persistent overlap failure, repeated rank reversal, unsupported cost structure, exhausted repair budget or a replacement benchmark that covers the target decision. Retirement is stronger than a warning label and narrower than deletion.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: parcel dispatch
Move the method to a parcel-dispatch queue. The archived benchmark assumes manual sorting; the target uses automated induction whose failures create bursts and rework. Historical arrival totals overlap, but the dependence structure does not.
A shadow run reveals that the old policy opens fewer lanes precisely during automation recovery. The team suspends transport, retains the archive for manual-mode contingencies and builds a target benchmark around burst recovery. That is a new decision role, not a silent refresh.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connect drift, replay and archive governance
Start with archive-replay benchmark drift and counterfactual transport limits. Use champion-archive replay and fallback-drill governance to recover the archive’s approved roles.
Return to post-handover drift and champion-archive governance when a shadow baseline reveals a successor-monitoring gap.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
A failed transport claim does not prove the source study was wrong. It means the source evidence, assumptions and target differences no longer support the intended target decision. Conversely, stable average error does not rescue a policy ranking that reverses near the action threshold.
A shadow baseline is not a no-risk mode. It can influence people, create anchoring and consume sensitive data. Retirement does not require erasing the benchmark or forbidding every historical and training use.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence and retirement checks
Archive the source and target definitions, transport assumptions, overlap diagnostics, decision boundary, disagreement strata, shadow configuration, access logs, escalation owner, interim restrictions and dated review outcomes.
For retirement, record scope, effective date, reason, systems and reports affected, replacement benchmark, remaining permitted uses, retention period, citation language and the authority required for any future requalification.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can a retired benchmark remain a fallback?
Only if fallback is a separately validated role with explicit scope. Retirement from ordinary comparison does not prove fallback fitness.
How long should a shadow baseline run?
Long enough to answer its predeclared question across relevant strata, with an end condition fixed before routine operation normalises it.
Can a narrow transported claim survive?
Yes, when overlap, mechanisms and decision stability are supported for that subset and unsupported areas remain excluded.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
NASA’s 2024 Standard for Models and Simulations requires defined intended use, acceptance criteria, documentation and credibility assessment. The 3 February 2026 NASA implementation handbook expands the lifecycle, risk and reporting practices supporting that standard. The Federal Reserve’s 17 April 2026 revised model-risk guidance links validation, limitations and ongoing monitoring to use and risk. The clinic and parcel examples are original teaching cases.
Continue through the Queueing Theory root, the Gold Standard of Education and How X Works.
Top · Contents · Previous · How X Works