Find your next learning step
Two forecasts can each look acceptable when inspected alone yet behave differently once one forecast is used to condition the other. Cross-calibration asks whether a forecast remains trustworthy relative to information carried by competing forecasts. Forecast dominance asks a different question: whether one forecast has no larger expected loss across a declared class of consistent scores. The conditioning set, target and score class must therefore be named before a winner is claimed.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Name the forecasts, outcome and comparison claim
Let Y be the outcome and let F1 and F2 be forecasts issued before Y is observed. A comparison needs the forecast target, issue time, horizon, available information and loss or scoring rule. Without those declarations, “better” can mean sharper, more calibrated, lower loss for one user or merely more convenient.
Calibration asks whether stated probabilities agree with realised frequencies. Cross-calibration conditions that check on another forecast or set of forecasts. Dominance compares expected scores over a stated family. These are related questions, not synonyms.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. See why individual calibration can be too coarse
A forecast can be calibrated relative to its own values while ignoring information visible in another forecast. The omission is not revealed if cases with different risks are pooled into the same issued probability.
Cross-calibration is stronger because it asks whether the conditional outcome distribution still agrees with F1 after the information in F2 is exposed. The direction matters: F1 may fail relative to F2 even when F2 passes relative to F1.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Work a directional cross-calibration ledger
Consider 100 binary cases. Forecast F1 always issues 0.50. Forecast F2 issues 0.20 for 50 cases and 0.80 for 50 cases. There are 10 events in the first group and 40 in the second.
| Conditioning cell | Cases | F1 | F2 | Events | Observed rate |
|---|---|---|---|---|---|
| F2=0.20 | 50 | 0.50 | 0.20 | 10 | 0.20 |
| F2=0.80 | 50 | 0.50 | 0.80 | 40 | 0.80 |
| All cases | 100 | 0.50 | Mixed | 50 | 0.50 |
| Verdict | — | Calibrated alone | Calibrated | — | F1 fails given F2 |
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Turn the same ledger into a score comparison
Under squared probability loss, the constant 0.50 forecast has average loss 0.25 in both groups. The conditional 0.20 and 0.80 forecast has expected loss 0.16 in each correctly calibrated group. F2 therefore improves the Brier score in this constructed population.
The improvement comes from useful resolution, not from calibration alone. A calibrated base-rate forecast can be beaten by a calibrated forecast that separates cases according to genuine pre-outcome information.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Change the score class before claiming dominance
One lower average score establishes superiority only for that score and population. Conditional dominance over a family of consistent scores is stronger: it asks whether every user whose loss belongs to the declared family weakly prefers one forecast in expectation.
| Comparison statement | What was checked | What may be concluded | What remains open |
|---|---|---|---|
| Lower Brier score | One proper score | Better under squared probability loss | Other losses and users |
| Lower log score | One logarithmic score | Better log predictive performance | Tail-sensitive alternatives |
| Mean-forecast dominance | Class of mean-consistent scores | Uniform preference in that class | Other targets |
| Cross-calibration | Conditional reliability | Forecast is trustworthy relative to named forecasts | Decision loss and sharpness |
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Separate nested information from empirical comparison
If F2 is the ideal conditional probability from a richer information set that contains the information behind F1, proper-score theory gives a clean route to improvement. In practice, information sets may overlap rather than nest, models may be estimated and both forecasts may be misspecified.
Empirical dominance tests therefore examine score differences with sampling uncertainty. They do not reconstruct the forecasters’ full information sets, and failure to reject equality is not proof that the forecasts are equivalent.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Connect the established forecast routes
The conditional-calibration guide explains why every reliability claim is indexed by information. The proper-scoring-rules guide connects honest reporting to the target being elicited.
Use the reliability-diagrams guide for finite-sample diagnostics and the posterior-mean risk guide when squared loss and conditional means are the decision.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Design a comparison that survives deployment
Lock the outcome definition, issue time, horizon, cases, score orientation, missing-outcome rule and comparison set before reading results. Keep forecasts paired on the same cases and preserve temporal order when errors are dependent.
Report conditional results by horizon, regime and decision-relevant subgroup only where support is adequate. Use a later period or external dataset to confirm partitions discovered during exploration.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Know the limits and common misconceptions
Cross-calibration is directional and relative to the named forecasts; it is not proof of calibration against every available variable. Dominance in one score is not dominance for all users, and dominance over a consistent score class does not establish causal correctness.
A more variable forecast is not automatically more informative. Resolution helps only when its variation tracks conditional outcomes. Estimated score gaps can also reverse under drift, rare-event sampling or a changed case mix.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Use an evidence and source routine
Record outcome, target, forecast issue process, horizons, forecast information available for conditioning, score class, pairing, dependence, sample support, pre-specification, uncertainty interval, calibration diagnostics, resolution, subgroup checks, drift window and deployment decision.
Strähl and Ziegel develop cross-calibration for probabilistic forecasts and stress its directional relation to other forecasters. Krüger and Ziegel derive conditions for forecast dominance, including cases with non-nested information sets and misspecification.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can two forecasts both be calibrated?
Yes. They can differ in resolution and score, and one may reveal conditional failures hidden by the other.
Is cross-calibration symmetric?
No. A forecast can be cross-calibrated relative to a second forecast while the reverse direction fails.
Does one significant score difference prove dominance?
No. It supports a comparison for the tested score and design; class-wide dominance needs the corresponding theory or tests.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Continue with a comparison transfer check
Recreate the 100-case ledger, compute both Brier scores and reverse the labels F1 and F2. Then perturb the event counts to 15 and 35. Recheck individual calibration, directional cross-calibration and the score gap as three separate claims.
Continue through the Probability root, the Gold Standard of Education and the How X Works hub.
Top · Contents · Previous · How X Works