Find your next learning step
A probability forecast should be judged as a probability before the outcome is known. A proper scoring rule makes a forecaster minimise expected loss by reporting the distribution they actually believe; a strictly proper rule makes that report unique. This incentive result is mathematical and conditional. It does not guarantee that the belief is well calibrated, the model is well specified or the forecasting process is free from strategic and sampling effects.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Define the report, outcome and orientation
Let Q be a reported predictive distribution and Y the realised outcome. A negatively oriented score S(Q,Y) is a loss, so smaller values are better. Propriety means that if the true or believed distribution is P, the expected score E_P[S(Q,Y)] is minimised by Q=P.
Strict propriety adds uniqueness, apart from distinctions the score cannot see. State the orientation because some authors use rewards to maximise instead of losses to minimise. Reversing the sign without changing the optimisation direction reverses the incentive.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. Work the binary Brier-risk identity
For a binary event Y in {0,1}, report q=P(Y=1) and use Brier loss S(q,Y)=(q−Y)². If the event probability is p, the expected loss is p(1−q)²+(1−p)q².
Expanding gives p(1−p)+(q−p)². The first term is unavoidable uncertainty under P; the second is the excess risk from misreporting. It is non-negative and vanishes only at q=p, so the binary Brier loss is strictly proper.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Compare honest and shifted reports
Suppose p=0.70. The expected Brier loss for an honest report q=0.70 is 0.21. A report of 0.90 has expected loss 0.25, while q=0.50 has expected loss 0.25. A more confident number is not automatically a better forecast.
| Report q | Expected loss if p=0.70 | Excess over honest report | Reading |
|---|---|---|---|
| 0.70 | 0.210 | 0.000 | Truthful minimum |
| 0.90 | 0.250 | 0.040 | Overconfident shift |
| 0.50 | 0.250 | 0.040 | Underconfident shift |
| 0.10 | 0.570 | 0.360 | Large probability error |
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. See why one realised outcome cannot certify honesty
If Y=1, the realised Brier losses for q=0.70 and q=0.90 are 0.09 and 0.01. The more extreme report wins that one case even though its expected loss is worse under p=0.70. Propriety is an expectation property, not a promise that the honest report wins every round.
Evaluation therefore needs a sequence or population whose dependence, selection and changing base rates are understood. Confidence intervals, paired score differences and time ordering matter when comparing systems.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Change the true probability and recompute
Keep the same candidate reports but change the data-generating probability to p=0.40. The strictly proper optimum moves to q=0.40. The report q=0.70 now carries excess Brier risk (0.70−0.40)²=0.09.
| True p | Report q | Expected Brier loss | Excess risk |
|---|---|---|---|
| 0.70 | 0.70 | 0.210 | 0.000 |
| 0.70 | 0.90 | 0.250 | 0.040 |
| 0.40 | 0.40 | 0.240 | 0.000 |
| 0.40 | 0.70 | 0.330 | 0.090 |
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Distinguish Brier and logarithmic sensitivity
Logarithmic loss is −log q when Y=1 and −log(1−q) when Y=0. Its expectation is cross-entropy and is strictly proper under ordinary conditions. It strongly penalises assigning tiny probability to an event that occurs and becomes infinite at a reported zero.
Brier and logarithmic losses are both proper but not interchangeable. They emphasise probability errors differently and can rank two imperfect forecasting systems differently. Score choice should match the outcome space, tolerance for tail errors and stated evaluation purpose.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Connect elicitability to point and distribution targets
A statistical functional T(P) is elicitable when some loss is uniquely minimised in expectation by reporting T(P). The mean is elicited by squared loss, a median by absolute loss when uniqueness conditions hold, and a quantile by an appropriate tilted absolute loss.
A strictly proper scoring rule elicits the full distribution rather than one summary. Elicitability is valuable because it pairs a target with an honest evaluation rule. It does not say that every familiar statistic has a useful one-dimensional strictly consistent loss.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connect the established probability routes
The conditional-median guide shows how absolute loss can produce several Bayes actions. The posterior-mean and MMSE guide shows why squared loss selects a conditional mean.
Use How Probability Works for calibration, conditional belief and decision context. The existing expectiles and elicitability guide gives a contrasting asymmetric target.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Know the limits and common misconceptions
A proper score encourages honest reporting only relative to the forecaster’s belief, information and utility assumptions. Payments, risk aversion, score aggregation, team incentives or the ability to influence which cases are scored can change behaviour.
Good average score does not imply calibration in every subgroup, useful resolution, causal validity or robustness under distribution shift. Post-selection on favourable time windows and repeated tuning to a public test set can also make score comparisons optimistic.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Use an evidence and source routine
Record the outcome space, report format, score orientation, exact formula, handling of zero probabilities, aggregation weights, sampling frame, dependence, missing outcomes, calibration diagnostics, subgroup checks, paired uncertainty and decision consequences.
Waghmare and Ziegel’s review of proper scoring rules develops propriety, estimation and forecast evaluation. Brehmer and Gneiting’s Bayes-act construction states strict propriety and shows how scoring rules can be properised under regularity conditions.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Does the best realised score reveal the true probability?
No. One outcome is noisy evidence. Proper scores justify expected comparisons and need repeated, appropriately sampled cases for empirical evaluation.
Is accuracy a proper probability score?
Ordinary classification accuracy evaluates a chosen label and usually discards the reported probability. It cannot generally reward honest distribution reports.
Should every project use one universal score?
No. Use at least one declared primary proper score and examine complementary diagnostics when different error regions matter.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Continue with a changed-score transfer check
For p=0.20, compare reports q=0.10, 0.20 and 0.40 under Brier loss and logarithmic loss. Compute expected loss before looking at a realised outcome, then explain why both proper rules choose q=0.20 while the size of each penalty differs.
Continue through the Probability root, the Gold Standard of Education and the How X Works hub.
Top · Contents · Previous · How X Works