Find your next learning step
A sleeping-channel portfolio is valid only if its wake–sleep rule represents when a channel can honestly contribute. Eligibility misspecification occurs when a useful channel sleeps, an invalid channel wakes or the decision uses information that arrived later. Delayed feedback makes those errors harder to see. A shadow-channel audit runs alternative eligibility rules without allowing them to change live weights, creating evidence for repair while preserving the approved process.
Open the chapter contents · Topology root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Write the eligibility contract as a prediction
For each residual, feasibility or noise channel, state what must be known before the next increment for the channel to be awake. Eligibility is a predictable decision, not a label assigned after the loss or evidence is observed.
Record the data clock as well as the mathematical condition. A feature computed later cannot retroactively justify an earlier wake decision even if its timestamp belongs to the same checkpoint.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Name the four audit outcomes
Compare approved eligibility with a prespecified shadow rule. True awake and true asleep rounds confirm agreement; false wake rounds expose invalid participation; false sleep rounds expose lost evidence or regret.
The shadow rule is not automatically correct. It is a challenger whose disagreements must be evaluated using later feedback, validity checks and operational consequences.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. An eligibility-confusion ledger
A manifold monitor audits 200 checkpoints after all delayed labels have matured. The table separates channel-state error from outcome quality.
| Approved state | Audited eligible | Audited ineligible | Interpretation |
|---|---|---|---|
| Awake | 132 | 8 | Eight possible false wakes |
| Asleep | 17 | 43 | Seventeen possible false sleeps |
| Total | 149 | 51 | Matured checkpoints only |
Stratify the ledger by regime, sensor version and delay length. A small global error rate can conceal systematic false sleeps exactly when the active manifold changes.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Worked delayed-feedback audit
The team reviews false-sleep candidates only after their outcome windows close. Each increment is joined to the decision-time eligibility record using an immutable event identifier.
| Audit group | Checkpoints | Mean feedback delay | Shadow gain after maturity | Action |
|---|---|---|---|---|
| Stable regime | 60 | 1.2 | 0.01 | Keep rule |
| Boundary approach | 22 | 3.8 | 0.14 | Revise wake threshold |
| Sensor replacement | 9 | 7.1 | Uncertain | Start fresh channel |
| Missing outcome | 6 | Not observed | Not identifiable | Do not score |
Do not code missing feedback as zero gain. Missingness may depend on the channel decision or regime and can make the comparison unidentifiable without extra assumptions or data.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Keep shadow weights out of live wealth
A shadow channel may compute hypothetical weights and evidence, but it must not receive or transfer live wealth unless the governance process promotes it. Otherwise the audit changes the system it is measuring.
Use separate namespaces, capital ledgers and outputs. Promotion starts from a declared initial account or an explicitly justified transfer compatible with the common null and filtration.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Update only when feedback becomes available
Maintain a pending-feedback queue keyed by the decision time. When an outcome arrives, apply the update to the correct historical decision and preserve the arrival time used by the live filtration.
Delayed feedback can increase online regret and can also create selection bias if only quick outcomes are audited. Report the delay distribution and mature-window rule.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: sparse support screening
Transfer the audit to sparse optimisation, where a coefficient-specific diagnostic sleeps when screening rules deem the coordinate ineligible. The shadow rule asks whether that sleep decision missed an approaching support change.
The transfer check distinguishes a delayed coefficient update from a redesigned feature space. A redesign changes channel meaning and may require a fresh process rather than a threshold repair.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connect sleeping channels, feedback and safeguards
Begin with the sleeping-channel and restart guide. Use the regret-controlled portfolio guide to measure the operational cost of missed eligibility.
Return to the predictable reweighting guide before promoting a shadow rule into live wealth updates.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
A shadow channel with lower retrospective loss is not automatically valid. It may have used later information, benefited from selective feedback or violated the null that governed the live channel.
Sleeping-expert regret and anytime-valid evidence are different guarantees. Repairing eligibility in an online learner does not by itself repair an e-process, false-unlock budget or multiple-testing account.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive eligibility code, feature timestamps, awake sets, shadow rules, decision identifiers, feedback-arrival times, pending queues, missingness reasons, nulls, filtrations, live and shadow ledgers and promotion decisions.
Replay false wakes, false sleeps, long delays, sensor replacement and rapidly alternating regimes. Verify that shadow computation cannot alter live weights and that every delayed outcome joins the intended checkpoint once.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can the shadow rule see the final outcome?
It may be scored after the outcome matures, but its eligibility decision must be reconstructed from information available at the original decision time.
Should every false sleep wake the channel immediately?
No. Investigate whether the disagreement is systematic, valid and decision-relevant. Promote a corrected rule through the declared review process.
How are never-observed outcomes scored?
They are not silently scored as zero. State the missingness problem, use justified methods or collect targeted feedback; some comparisons remain unidentified.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Cortes and colleagues analyse sleeping experts with feedback graphs. Joulani, Gyorgy and Szepesvari study online learning under delayed feedback. Ramdas and colleagues survey safe anytime-valid inference and evidence processes. The audit ledgers here are original teaching examples.
Continue through the Topology root, the Gold Standard of Education and How X Works.
Top · Contents · Previous · How X Works