Find your next learning step
A forecast can be calibrated relative to the information contained in the forecast itself and still miss structure available in a richer information set. Auto-calibration asks whether outcomes behave like draws from the issued forecast after conditioning on that forecast. Stronger conditional checks add declared variables, groups or histories. The information being conditioned on is therefore part of the claim, not a footnote.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Name the outcome, forecast and information set
Let Y be the outcome and let G represent the information available before Y is observed. An ideal predictive distribution is the conditional law of Y given G. It uses all information declared to be available, not future outcomes or variables recorded only after the decision.
A published probability may contain less information than G. Calibration relative to the issued forecast asks one question; calibration after conditioning on location, horizon, subgroup or another pre-specified variable asks a stronger and different question.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Define auto-calibration without overclaiming
For a predictive distribution F, auto-calibration means that the conditional distribution of Y given F is F, under suitable regularity. For a binary event with issued probability P, this reduces to Pr(Y=1|P)=P almost surely.
The condition says that cases receiving the same forecast behave consistently with that forecast. It does not say that the forecast uses every informative variable, is causally correct or remains calibrated after the population changes.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Work an aggregate calibration ledger
Suppose 50 Group A cases and 50 Group B cases all receive forecast 0.40. Group A has 10 events and Group B has 30. Across all 100 cases there are 40 events, so the forecast looks calibrated when conditioned only on P=0.40.
| Conditioning view | Cases | Forecast | Events | Observed rate |
|---|---|---|---|---|
| All P=0.40 cases | 100 | 0.40 | 40 | 0.40 |
| Group A | 50 | 0.40 | 10 | 0.20 |
| Group B | 50 | 0.40 | 30 | 0.60 |
| A versus B | Same P | No separation | Different rates | Hidden structure |
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. See why a coarser forecast can be calibrated
The constant 0.40 forecast is auto-calibrated relative to the sigma-algebra generated by its own single value in this example. It is nevertheless less informative than a forecast that issues 0.20 for Group A and 0.60 for Group B.
Calibration alone does not reward resolution. A forecaster can discard useful information and retain calibration by predicting an appropriate base rate. Proper scores and discrimination diagnostics help compare calibrated forecasts with different information content.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Change the information set and change the verdict
Now issue 0.20 to Group A and 0.60 to Group B. The same 40 events yield calibration within both groups. If a pre-declared time-of-day variable then separates each group into different event rates, calibration fails relative to that richer information set.
| Forecast design | Information encoded | Aggregate result | Conditional result |
|---|---|---|---|
| Constant 0.40 | None beyond base rate | Calibrated | Fails by group |
| 0.20 for A; 0.60 for B | Group | Calibrated | Calibrated by group |
| Add ignored time pattern | Group only | May remain calibrated | Can fail by group and time |
| Use future outcome data | Invalid information | Artificial fit | Not a forecast |
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Move from full distributions to target calibration
Conditional T-calibration asks whether a reported functional such as a mean, quantile, expectile or event probability can be taken at face value under an identification condition. Auto-calibration of the full predictive distribution generally implies appropriate target-calibration properties.
The reverse need not hold. A correct conditional mean does not guarantee correct tails, quantiles or full distributional shape. State the target before choosing a reliability diagram or residual moment.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Connect the established probability routes
The reliability-diagrams guide shows how finite bins estimate calibration functions. The proper-scoring-rules guide explains why calibration should be paired with honest performance evaluation.
Use the posterior-mean guide for conditional means and squared loss, and How Probability Works for information, conditioning and base-rate reasoning.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Design conditional checks before seeing outcomes
Choose groups and covariates because they represent decisions, known mechanisms or deployment risks. Searching many partitions after seeing residuals can always find an apparent failure, especially with small samples.
Use held-out data, multiplicity control or a separate confirmation period. Preserve time order when forecasts are updated online, and avoid conditioning on variables affected by the outcome or intervention.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Know the limits and common misconceptions
Passing several conditional moment tests does not prove full auto-calibration. The tests inspect selected consequences of the condition and have limited power in sparse regions. Failing a test can arise from sampling noise, dependence, drift or a mis-specified uncertainty model.
Subgroup calibration is not automatically fairness, causal validity or equal decision quality. Groups can have different harms, thresholds and label processes even when probabilities match observed frequencies.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Use an evidence and source routine
Record forecast time, outcome window, available information G, forecast representation, target functional, conditioning variables, pre-specification, sample support, dependence, missing labels, estimation method, uncertainty, proper scores, resolution, subgroup impacts and drift checks.
Gneiting and Resin develop conditional target calibration, population reliability diagrams and score decompositions. Resin, Yang and Gneiting review calibration hierarchies and the relation between auto-calibration, predictive distributions and weaker properties.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can a constant forecast be calibrated?
Yes. A correct base-rate forecast can be calibrated while offering almost no resolution between cases.
Does auto-calibration use all available information?
No. It conditions on the forecast itself. Ideality relative to a richer declared information set is stronger.
Should I test every possible subgroup?
No. Pre-specify decision-relevant groups, then use exploratory findings as hypotheses for fresh data rather than final proof.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Continue with an information-set transfer check
Construct 120 binary cases with one constant forecast that is calibrated overall but fails across three groups. Replace it with group-specific probabilities, then add a time variable that reveals a second hidden split. For each stage, name the sigma-algebra and the exact calibration claim.
Continue through the Probability root, the Gold Standard of Education and the How X Works hub.
Top · Contents · Previous · How X Works