Find your next learning step
A single average score answers a single weighted decision question. A Murphy diagram opens that score into a family of elementary threshold losses. If one forecast has no larger average elementary score at every threshold, it dominates for every consistent score that can be represented as a non-negative mixture of those elements. If the curves cross, the preferred forecast depends on which thresholds—and therefore which user decisions—receive weight.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the decision
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE DECISION
1. Name the target before drawing the curve
A Murphy diagram is indexed by a statistical target such as a quantile, expectile, mean or an approved joint target. The elementary score family changes with the target. Forecasts, observations, issue times and horizons must be paired before any curve is computed.
Lower is better only under the chosen score convention. The horizontal coordinate is a decision threshold, not an observed probability density. State the target and convention beside the graph so that a visual comparison cannot drift into a different claim.
CHAPTER 2 OF 12 · DEFINE THE DECISION
2. See a consistent score as a weighted mixture
Many consistent scoring functions can be written as non-negative mixtures of elementary scores indexed by a threshold θ. The user-specific score chooses the mixing measure: it places more weight where that user experiences greater consequence.
This representation separates forecast performance from the user’s weighting. The Murphy curve supplies the elementary-score evidence; a particular user supplies the weights. Integrating the curve against those weights recovers that user’s average score.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Work an elementary-score ledger
Suppose two probability forecasts are compared at four decision thresholds. The numbers below are illustrative average elementary losses on the same 200 cases. Forecast A is lower at three thresholds but higher at the fourth.
| Threshold θ | Forecast A loss | Forecast B loss | Lower curve |
|---|---|---|---|
| 0.20 | 0.072 | 0.081 | A |
| 0.40 | 0.116 | 0.121 | A |
| 0.60 | 0.110 | 0.104 | B |
| 0.80 | 0.058 | 0.063 | A |
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Turn a user profile into a score difference
Give the four thresholds weights 0.10, 0.20, 0.60 and 0.10. The weighted losses are 0.1030 for A and 0.1011 for B, so B is preferred by this threshold-heavy user. With equal weights, A averages 0.0890 and B averages 0.09225.
The ranking reverses because the first user concentrates weight where B’s curve is lower. Neither calculation contradicts the other. Each answers a different declared decision problem using the same elementary evidence.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Distinguish pointwise dominance from a sampled plot
Population dominance requires the expected elementary-score curve for A to lie at or below B for every relevant θ, with strict improvement somewhere. A finite grid can miss a narrow crossing between plotted thresholds.
| Evidence | Supports | Does not establish | Next check |
|---|---|---|---|
| A below B on plotted grid | Gridwise empirical advantage | All-threshold dominance | Refine grid at crossings |
| Confidence band excludes zero | Uncertainty-qualified local difference | Unconditional truth | Account for dependence |
| Curves cross | User-sensitive ranking | Either forecast is useless | Apply real user weights |
| A below B everywhere sampled | Candidate dominance | Causal superiority | Replicate out of sample |
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Handle uncertainty and repeated forecast errors
Elementary score differences share the same cases and are strongly dependent across thresholds. Time-series forecasts can also have serially dependent errors, especially with overlapping horizons. Pointwise standard errors treated as independent can overstate certainty.
Use a resampling or testing method that preserves the paired and temporal structure. Report whether bands are pointwise or simultaneous. A wide band means the data do not separate the forecasts at that threshold; it is not evidence that their population curves are identical.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Connect calibration, scoring and comparison routes
The cross-calibration and conditional-dominance guide separates reliability from ranking. The proper-scoring-rules guide explains why the target and consistency class must be declared.
Use the reliability-diagrams guide for conditional frequency checks. Murphy diagrams answer a complementary question: which forecast performs better across a family of user weightings.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Design a diagram that remains interpretable
Use identical cases, outcome definitions and horizons. Plot both average elementary-score curves on the same axes and add a difference panel when crossings are close. Show the threshold range supported by forecasts and decisions rather than stretching the axis into empty regions.
Publish the number of cases, missing-outcome rule, dependence treatment, uncertainty method and grid resolution. If a threshold region was chosen after seeing the curves, label that exploration and confirm it on later data.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Know the limits and common misconceptions
A Murphy diagram is not a reliability diagram, receiver operating characteristic curve or calibration plot. It does not by itself identify why one forecast wins, and it does not turn a mis-specified target into the correct decision quantity.
Dominance within one consistent-score class is not dominance for every imaginable loss. Curves that touch are not automatically equivalent, and curves that cross do not make comparison impossible; they make the user’s threshold weights material.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Use an evidence and source routine
Record the target, forecast and outcome units, cases, horizon, elementary-score formula, threshold grid, score orientation, user weights, pairing, dependence, uncertainty bands, crossing locations, extrapolated regions, out-of-sample window and decision conclusion.
Ehm, Gneiting, Jordan and Krüger derive mixture representations and introduce Murphy diagrams for quantile and expectile comparisons. Ziegel and colleagues extend the graphical dominance idea to joint Value-at-Risk and Expected Shortfall forecasts and pair it with formal testing.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Does the lowest curve always identify the best forecast?
Only relative to the declared target, elementary-score family, population and score orientation.
What should I do when curves cross?
Map the crossing region to real decision thresholds, state the user weights and compare the resulting weighted scores with uncertainty.
Can I compare curves from different test samples?
Not cleanly. Case-mix differences can move the curves, so forecasts should be evaluated on the same outcomes whenever possible.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Continue with a weighting transfer check
Recalculate the four-threshold ledger with equal weights, then with 80% of the weight at θ=0.60. Add a fifth threshold between 0.50 and 0.60 and decide what value would remove the apparent crossing. Explain why that interpolation is evidence to collect, not a fact to assume.
Continue through the Probability root, the Gold Standard of Education and the How X Works hub.
Top · Contents · Previous · How X Works