Find your next learning step
Retiring an old queue surrogate does not automatically qualify its replacement. The successor must show that it preserves the decisions that matter, improves the weaknesses that forced retirement and remains usable under the live traffic regimes it will face. A controlled champion–challenger handover makes that evidence visible before authority changes hands.
Open the chapter contents · Queueing Theory root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Separate retirement from successor approval
The incumbent can lose approval because its domain has narrowed, even when no successor is ready. Keep the retirement decision, the fallback decision and the successor qualification decision as three separate records.
Qualification begins with an intended-use statement: which queues, horizons, routing rules and decisions the new surrogate may support. A model that predicts average delay well may still be unsuitable for a tail-risk staffing decision.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Define the champion–challenger contract
The champion is the currently authorised method; the challenger is the proposed successor. Both receive the same frozen inputs and are judged against a higher-fidelity queue experiment or observed outcome, not merely against each other.
Predeclare comparison windows, traffic strata, metrics, tolerances, tie rules and who can stop the trial. This prevents a favourable slice from becoming the whole qualification argument.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Build a qualification matrix
A service team compares an incumbent response surface with a new surrogate across four decision strata.
| Stratum | Decision metric | Champion error | Challenger error | Qualification state |
|---|---|---|---|---|
| Ordinary weekday | Mean wait | 6.1% | 3.8% | Pass |
| Monday peak | 95th-percentile wait | 14.0% | 7.2% | Pass with monitoring |
| Overflow routing | Action ranking | Wrong top action | Correct top action | Pass |
| Rare outage | Tail probability | Not supported | Wide interval | Hold out of scope |
The matrix makes partial approval possible. It is safer to state one unsupported regime than to bury it inside a favourable overall average.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Worked parallel-run exit decision
For eight weekly decisions, the challenger and champion recommend staffing while operations continue to follow the champion. A detailed simulator supplies confirmatory loss estimates.
| Checkpoint | Weeks observed | Challenger wins | Worst excess loss | Exit decision |
|---|---|---|---|---|
| Early review | 2 | 2 | 0.6 | Continue |
| Midpoint | 4 | 3 | 0.6 | Continue; peak evidence thin |
| Primary gate | 6 | 5 | 0.3 | Provisional handover |
| Stability gate | 8 | 7 | 0.3 | Exit parallel run |
After handover, the former champion remains a shadow comparator for four weeks. Authority changes at the gate, while observation continues long enough to catch implementation drift.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Use decision agreement and outcome checks
Prediction error and decision agreement answer different questions. Small numerical errors can reverse actions near a threshold, while larger harmless errors may leave the preferred action unchanged.
Track calibration, ranking, constraint violations and realised outcomes. Where real outcomes are delayed, use a confirmatory simulator but record its own uncertainty and independence.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Protect the trial from shared failure
Champion and challenger may share code, data or assumptions. Parallel agreement is weak evidence when both inherit the same fault, so include independent implementation checks, stress cases and a simple conservative benchmark.
Freeze evaluation data before tuning. If a failed stratum drives challenger changes, move that stratum back to development and reserve fresh evidence for the next gate.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: emergency department rostering
Transfer the method to an emergency department where a new surrogate estimates waits by acuity and roster. The challenger improves average fit but initially merges two triage routes that have different safety constraints.
The transfer check asks whether the successor preserves the operational boundary, not just the numerical score. The merged route fails intended-use qualification even if aggregate mean error falls.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connect screening, retirement and handover
Begin with the surrogate screening and confirmatory experiment guide. Use the retirement and model-risk register guide to establish why the incumbent is leaving.
Return to the nested Monte Carlo guide when simulation noise can change the handover verdict.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
A longer parallel run is not automatically stronger. Repeated observations from one quiet regime do not replace coverage of the peak or routing change that matters.
The challenger need not beat the champion on every metric. It must meet the declared tolerances for its intended use and show how trade-offs are governed.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive champion and challenger versions, shared dependencies, frozen inputs, traffic strata, decision metrics, higher-fidelity comparator, tolerances, exclusions and reviewer assignments.
At exit, preserve the gate calculation, unresolved regimes, fallback, shadow-monitoring period, rollback trigger, owner and date when the old champion loses operational authority.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can the challenger control some queues before full handover?
Yes, through a staged scope with explicit routing and a tested fallback for queues that remain with the champion.
What if champion and challenger tie?
Use the predeclared tie rule: often retain the champion, choose the simpler controlled method or collect evidence in the most decision-sensitive stratum.
Does handover end validation?
No. It ends the qualification phase and begins monitored operation under the successor’s declared controls.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
The Federal Reserve’s 17 April 2026 revised model-risk guidance discusses developmental evidence, independent validation, outcomes analysis and ongoing monitoring in its regulated context. NIST’s AI RMF Playbook covers measurement against deployment conditions and post-deployment risk management. The handover matrices are original teaching examples.
Continue through the Queueing Theory root, the Gold Standard of Education and How X Works.
Top · Contents · Previous · How X Works