Find your next learning step
A reliability diagram compares announced probabilities with observed event frequencies. Its population target is a calibration function, but the displayed curve is an estimate built from finite, uneven data. Bin edges, empty regions, sampling noise and smoothing can change the picture. The useful question is therefore not whether dots look close to a diagonal, but which conditional claim was estimated, from which cases and with what resolution and uncertainty.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Define the population calibration function
For a binary outcome Y and forecast P in [0,1], define the calibration function c(p)=Pr(Y=1|P=p) when that conditional quantity is well defined. Perfect reliability means c(P)=P almost surely. A reliability diagram tries to compare c(p) with the diagonal p.
This is a conditional statement. It differs from matching the overall event rate, achieving a good average score or separating positive from negative cases. A model can have the right base rate yet be locally miscalibrated, or be calibrated but have little resolution because it nearly always predicts the same probability.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Translate a finite bin into two averages
For a bin B, the conventional point uses the mean issued probability x_B and the observed event fraction y_B among cases whose forecasts fall in B. The point estimates a bin-averaged relationship, not c(p) at every probability inside the bin.
If forecasts are concentrated at one edge and the calibration function bends across the bin, plotting y_B at x_B can hide that structure. Sample size, the distribution of forecasts within the bin and the bin rule belong in the interpretation.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Work an original two-bin ledger
Suppose ten cases in a low bin have mean forecast 0.18 and two events, while ten cases in a high bin have mean forecast 0.72 and eight events. The conventional points are (0.18,0.20) and (0.72,0.80).
| Bin | Cases | Mean forecast | Events | Observed fraction |
|---|---|---|---|---|
| 0.10–0.30 | 10 | 0.18 | 2 | 0.20 |
| 0.60–0.80 | 10 | 0.72 | 8 | 0.80 |
| Combined | 20 | 0.45 | 10 | 0.50 |
| Unoccupied middle | 0 | — | — | Not estimable |
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Separate sampling noise from a systematic gap
An observed fraction is a noisy estimate. With only ten cases, two events rather than one or three can move a point substantially. A visible departure from the diagonal is therefore not automatically a stable property of the forecasting system.
Use uncertainty intervals or resampling suited to the data structure, and preserve clusters or time dependence when resampling. Repeated decisions from the same person, place or storm are not necessarily independent Bernoulli trials.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Change the bin edges and expose finite-bin bias
Imagine forecasts at 0.21, 0.29, 0.31 and 0.39, with outcomes 0, 0, 1 and 1. A split at 0.30 produces two points with observed fractions 0 and 1. One broad 0.20–0.40 bin produces a single point near (0.30,0.50).
| Binning rule | Displayed groups | Observed fractions | What changes |
|---|---|---|---|
| Split at 0.30 | 2 + 2 cases | 0.00 and 1.00 | Sharp apparent step |
| One broad bin | 4 cases | 0.50 | Step is averaged away |
| Four singletons | 1 case each | 0 or 1 | Maximum noise |
| Population target | Conditional curve | Unknown here | Needs more data |
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Use smoothing without hiding its assumptions
Kernel smoothers, local regression and isotonic regression can reduce the arbitrariness of fixed bins, but each adds structure. Bandwidth controls a bias-variance trade-off; isotonic regression assumes a non-decreasing relationship between forecast and event frequency.
Show the forecast distribution beneath the curve and state the smoothing rule. A smooth line in a sparse tail may be mostly extrapolation or pooling. Reproducibility requires the implementation choices, not only the final graphic.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Connect scoring, calibration and discrimination
The proper-scoring-rules guide explains why honest distribution forecasts are evaluated in expectation. The conditional-median guide shows that an evaluation target must match its loss.
Use How Probability Works for conditional probabilities and base rates. Calibration diagnoses whether probabilities can be taken at face value; discrimination asks whether cases receive meaningfully different forecasts. They are complementary, not substitutes.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Check subgroups, time and decision ranges
A model can look calibrated in aggregate while being miscalibrated within important groups, a form of aggregation masking. Check pre-specified subgroups, seasons, forecast horizons and operational ranges while accounting for smaller samples and multiple comparisons.
Calibration at probabilities that never trigger a decision may matter less than calibration near an action threshold, but narrowing to a threshold after seeing results invites selection bias. Declare the decision range and audit plan before judging the plot.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Know the limits and common misconceptions
A diagonal-looking plot does not prove calibration at every probability. Coarse bins can average positive and negative errors, sparse bins can look volatile, and adaptive choices made after viewing outcomes can produce a flattering display.
Recalibration can improve reliability on one population while reducing sharpness or failing after distribution shift. A reliability diagram is diagnostic evidence, not a causal explanation and not a complete forecast comparison.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Use an evidence and source routine
Record the outcome definition, sampling frame, forecast time, bin or smoother rule, number of cases per region, confidence method, dependence structure, missing outcomes, subgroup plan, distribution shift, decision thresholds, proper score and discrimination diagnostic.
Dimitriadis, Gneiting and Jordan explain instability in classical binning and develop consistent, optimally binned, reproducible diagrams. Arrieta-Ibarra and colleagues analyse the resolution-versus-confidence trade-off and cumulative alternatives to binned calibration displays.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
How many bins should I use?
There is no universal number. It depends on sample size, forecast density and the scale of departures worth detecting; report sensitivity to reasonable choices or use a principled estimator.
Does calibration mean the forecasts are useful?
No. Constant base-rate forecasts can be calibrated but offer little resolution for decisions.
Can I read a bin point as c(p) at its centre?
Usually not. It is a finite-sample average over forecasts in that bin and is plotted at a representative probability.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Continue with a changed-context transfer check
Create 40 forecasts concentrated near 0.20 and 0.80, then compare equal-width bins with equal-count bins. Keep outcomes fixed. Mark empty regions, compute every displayed point and explain which apparent differences come from the estimator rather than the forecasting system.
Continue through the Probability root, the Gold Standard of Education and the How X Works hub.
Top · Contents · Previous · How X Works