Find your next learning step
A shadow eligibility rule may outperform the live wake–sleep rule in retrospective audits, but it has seen a world the live system did not. Promotion must therefore begin prospectively. Wealth quarantine keeps historical shadow evidence out of the live e-process, while a controlled trial compares eligibility, delay and regret under declared rules. Only evidence generated after the promotion boundary can support the new live claim.
Open the chapter contents · Topology root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Define what promotion changes
State whether the challenger changes wake thresholds, features, channel meaning or feedback handling. A threshold adjustment may preserve the null; a new feature or channel may require a fresh process.
Freeze the audited shadow version before the prospective trial. Continued tuning after outcomes arrive creates another development version.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Quarantine retrospective shadow evidence
Historical shadow wealth is diagnostic, not spendable. Keep it in a closed audit ledger and initialise the prospective rule with fresh declared capital.
If any transfer is proposed, justify it from the exact martingale or e-process construction and filtration. Similar channel names are not enough.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Build a promotion ledger
A residual channel challenger reduced false sleeps in replay. The team runs it prospectively beside the approved rule.
| Account | Historical shadow value | Transfer to trial | Fresh trial capital | Status |
|---|---|---|---|---|
| Old shadow residual | 4.8 | 0.0 | 0.0 | Quarantined |
| Approved live channel | 2.1 | 0.0 | 0.5 | Control |
| Prospective challenger | 0.0 | 0.0 | 0.5 | Trial |
| Common audit reserve | 0.0 | 0.0 | 0.5 | Unused |
Separate capital prevents the audit result from becoming evidence merely because the challenger looked strong in hindsight.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Worked promotion gates
The challenger is evaluated after the declared mature-feedback window.
| Gate | Requirement | Observed | Result |
|---|---|---|---|
| Predictability | Features available before increment | 100% | Pass |
| False-wake rate | No higher than control +2% | Control +0.8% | Pass |
| False-sleep reduction | At least 20% | 31% | Pass |
| Evidence validity | No filtration breach | None found | Pass |
A trial can improve regret yet fail evidence validity. All gates are conjunctive when the channel also supports anytime-valid decisions.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Use a prospective comparison schedule
Declare trial length, maturity delay, review times and stopping conditions. Do not stop at the first favourable checkpoint unless the sequential analysis permits it.
Stratify by stable regime, boundary approach and sensor version. Promotion should not rely on average performance from easy periods.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Preserve fallback and restart accounting
Promote first in limited regimes. If a validity or harm gate fails, revert to the approved rule and close or pause the challenger according to the ledger.
Record whether promotion is a continuation, restart or new channel. That state determines what evidence and wealth can cross the boundary.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: active-set screening
Transfer the trial to active-set screening where a challenger wakes coefficient diagnostics earlier near a support transition.
The transfer check ensures challenger features are available before the optimisation increment and that retrospective support labels do not enter prospective evidence.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connect audit, delay and predictable wealth
Start with the shadow-channel audit guide. The sleeping-channel guide distinguishes pause, retirement and restart.
Use the predictable wealth guide before any live weight transfer.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
Retrospective outperformance does not estimate prospective gain without accounting for selection and tuning. The challenger may have been chosen because it fit this history.
Wealth quarantine does not erase the usefulness of old evidence. It preserves it for design and diagnosis while preventing double use in the new validity claim.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive live and challenger code, version freeze, retrospective selection path, features, timestamps, nulls, filtrations, quarantined ledger, fresh capital and trial plan.
At review, save matured outcomes, delay distribution, gates, sequential method, fallback events, promotion scope and any proposed restart or transfer proof.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can the best shadow rule be promoted immediately?
No. Freeze it and test it prospectively; retrospective selection creates optimism.
Why not transfer its historical wealth?
Because the shadow account was computed after design choices and may not satisfy the live process’s predictable update contract.
What if the challenger passes only in one regime?
Promote only in that declared regime and keep the control elsewhere. Scope is part of the evidence claim.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Cortes and colleagues study online learning with sleeping experts and feedback graphs. Joulani and colleagues analyse online learning under delayed feedback. Ramdas and colleagues survey safe anytime-valid inference. The promotion ledgers are original teaching examples.
Continue through the Topology root, the Gold Standard of Education and How X Works.
Top · Contents · Previous · How X Works