GOLD STANDARD · PROBABILITY · HOW X WORKS
Find the decision hidden inside the model
An active-review policy can look efficient because it chooses the cases most likely to help. That same choice also changes which outcomes are observed. A gold-standard evaluation records the exploration probabilities, reconstructs credible alternatives and asks whether any apparent saving survives review-budget and subgroup safety checks.
This guide connects the Gold Standard of Probability, Gold Standard of Education, and How X Works hub.
Chapter contents and reading routes
Choose the route closest to your present task. Every chapter remains expanded, so the article also reads naturally from start to finish.
Orient the evaluation problem
Build the counterfactual estimate
Control variance and budget risk
Transfer and govern
Use evidence and next routes
CHAPTER 1 OF 12 · ORIENT THE POLICY
1. Policy logs are part of the evidence
An active reviewer does not see a random slice of the unresolved queue. The policy favours cases with high uncertainty, large estimated influence, particular reviewer availability or a predicted chance of changing the population total. Outcomes observed under that policy therefore describe the selected queue, not automatically the full candidate population.
For every decision point, preserve the eligible actions, the information available then, the probability assigned to each action, the chosen assignment, its cost and the later label. Without that time-stamped log, a new policy cannot be evaluated from old review data without relying almost entirely on modelling assumptions.
CHAPTER 2 OF 12 · ORIENT THE POLICY
2. Define value at the downstream decision
The policy target may be label accuracy, overlap-count error, uncertainty in the zero-truncated generalised inverse Gaussian–Poisson total, or the chance of making the wrong capacity decision. These targets need not prefer the same assignments.
Write the reward before comparing policies. One defensible reward is the reduction in expected loss about the final population total minus review cost, with separate safety constraints for consequential false links and under-reviewed groups. A policy that resolves many easy labels can score well on accuracy while adding little decision value.
CHAPTER 3 OF 12 · ESTIMATE COUNTERFACTUAL VALUE
3. Exploration gives alternatives observable support
If the logging policy always sends a case type to one reviewer, the outcome under another reviewer is not observed. A small randomised exploration probability keeps plausible alternatives represented and makes counterfactual evaluation less dependent on extrapolation.
Exploration need not ignore ethics or expertise. Unsafe or ineligible assignments remain excluded. Within the safe set, use known probabilities and a floor that prevents a relevant action from becoming vanishingly rare. The floor is a design choice that trades immediate efficiency for future learning.
CHAPTER 4 OF 12 · ESTIMATE COUNTERFACTUAL VALUE
4. Work through an importance-weighted comparison
Suppose the log contains 1,000 decisions. A candidate policy would select action A for 300 of them. Under the old policy, A was actually chosen in 120 of those cases with probability 0.40. Each observed reward for that matching action receives weight 1/0.40 = 2.5 when estimating the candidate value.
If those 120 rewards average 6 units of decision benefit before weighting, their contribution estimates what would have happened across the 300 supported decisions. But this arithmetic is credible only if the propensity is correct, the logged context contains the causes of assignment and outcome, and the candidate never chooses an action with zero logging probability.
| Logged situation | Propensity | Evaluation weight | Main concern |
|---|---|---|---|
| Action A selected | 0.40 | 2.50 | Moderate variance |
| Rare specialist action | 0.05 | 20.00 | High variance |
| Unsupported action | 0 | Undefined | Not identifiable |
CHAPTER 5 OF 12 · ESTIMATE COUNTERFACTUAL VALUE
5. Use direct and doubly robust estimates as cross-checks
A direct model predicts the reward of every eligible action from logged context. Importance weighting instead reweights observed matching actions. A doubly robust estimator combines them and can protect against failure of one working model under its stated conditions.
No estimator repairs missing support. Compare direct, weighted, self-normalised and doubly robust results. Large disagreement is evidence that the conclusion depends on modelling or a few high-weight cases, not a reason to report only the most favourable estimate.
CHAPTER 6 OF 12 · CONTROL RISK
6. Control weight variance before claiming improvement
Rare actions can carry enormous inverse-propensity weights. Effective sample size, maximum weight and contribution plots show whether a headline gain rests on a handful of assignments. Clipping or stabilising weights may reduce variance but introduces bias.
Predeclare sensitivity across reasonable caps and use confidence bounds that reflect adaptive collection. If a candidate is better only after aggressive clipping, the current log does not establish superiority. Collect targeted exploration instead of polishing an unstable point estimate.
CHAPTER 7 OF 12 · CONTROL RISK
7. Make review-budget safety a first-class outcome
A policy can improve average reward while exhausting specialist hours early, delaying urgent cases or leaving a late queue without feasible reviewers. Replay the policy under realistic arrivals, service times and hard budget constraints rather than evaluating assignments independently.
Suppose policy X gains 140 decision units using 90 specialist hours and policy Y gains 150 using 118 hours against a 120-hour cap. A small demand surge makes Y infeasible. Budget-safe selection can require a reserve, a high-probability spending bound and a fallback policy whose performance is also evaluated.
| Policy | Estimated benefit | Specialist hours | Stress result |
|---|---|---|---|
| X | 140 | 90 | Within reserve |
| Y | 150 | 118 | Fails under small surge |
| Budget-safe Y | 145 | 102 | Meets reserve rule |
CHAPTER 8 OF 12 · TRANSFER AND GOVERN
8. Changed-context transfer: medical chart review
Move from record linkage to validating diagnoses in medical charts. The stable structure is a costly review action, an adaptive selection policy, known exploration probabilities and a downstream estimate. The confidentiality rules, clinical losses and eligible reviewers change.
Transfer succeeds when the target population, reward, action set and safety constraints are rebuilt. Reusing the same exploration floor or cost weights would confuse the statistical structure with evidence specific to the original review programme.
CHAPTER 9 OF 12 · TRANSFER AND GOVERN
9. Protect sparse groups and policy changes
Report support and value by language, source pair, geography and case difficulty. A policy can look safe overall while one subgroup has tiny propensities and nearly no counterfactual evidence. Minimum coverage or worst-group constraints may be more appropriate than one average reward.
When the logging policy changes, version its probabilities and features. Pooling logs as if one mechanism generated them corrupts the weights. A new feature available only after a decision must not be smuggled into the historical context.
CHAPTER 10 OF 12 · TRANSFER AND GOVERN
10. Limits and common misconceptions
Off-policy evaluation does not prove what an arbitrary new policy will do. It estimates supported alternatives under assumptions about logged propensities, measured context, stable outcomes and policy interference. Reviewer learning, fatigue and queue effects can violate those assumptions.
Random exploration is not automatically wasteful; it purchases identifiability. Conversely, a recorded score is not necessarily a propensity. The probability must describe how the system actually randomised among eligible actions at that moment.
CHAPTER 11 OF 12 · CHECK AND CONTINUE
11. Questions review teams often ask
Use offline evaluation to eliminate clearly unsafe or unsupported candidates before a live trial, then confirm promising policies prospectively with guarded rollout and continuous budget monitoring.
Can a deterministic historical policy support evaluation?
Only for alternatives that choose the same actions on the same supported contexts, unless a credible outcome model supplies extrapolation. That limitation should be explicit rather than hidden inside a flexible predictor.
Should exploration continue after deployment?
Usually a small safe component remains useful for drift detection and future comparison. Its rate should reflect decision stakes, reviewer capacity and the cost of losing support, with exclusions for unsafe assignments.
CHAPTER 12 OF 12 · CHECK AND CONTINUE
12. Evidence, specialist routes and the next check
Wang, Agarwal and Dudík on optimal and adaptive off-policy evaluation explains why estimator choice depends on logging support and reward modelling. research on safe exploration in contextual bandits treats exploration as a constrained decision, while counterfactual learning with general data-generating policies examines evaluation beyond simple random logging.
Continue through active allocation and consensus stopping, reviewer consensus and dependence, and reviewer drift and recalibration. The next check is to replay every proposed policy with untouched propensities, budget stress and subgroup support shown on one audit page.
Contents · Previous chapter · Return to the Probability root