An ensemble weather forecast contains multiple possible evolutions of the atmosphere, often generated by varying initial conditions, model components or other uncertainty representations. Its spread can communicate uncertainty, but raw ensemble frequency is not automatically a reliable probability of future events.
The gold standard is a calibrated predictive distribution that remains useful and independently verified on forecast cases not used to tune it. Calibration should improve the match between stated probabilities and observed frequencies while retaining meaningful discrimination and the dependence of weather variables across space and time.
eduKateSG · Weather Ensemble Forecast Calibration
Find your next learning step
Start with raw ensemble members, calculate event probabilities and CRPS, then learn how calibration, spread and independent testing change forecasts.
Full chapter index · Practice and transfer · Education root · How X Works
Full chapter index
Define ensemble predictions · Chapters 1–2
Calculate scores · Chapters 3–4
Test calibration · Chapters 5–7
Preserve usefulness · Chapters 8–10
Connected forecast routes · Chapters 11–12
CHAPTER 1 OF 12 · Define ensemble predictions
1. An ensemble is a sample of model possibilities, not certainty
Numerical weather prediction is sensitive to incomplete observations, initial-condition uncertainty and the representation of small-scale processes. An ensemble explores a range of possibilities rather than issuing only one deterministic path.
Different prediction systems perturb initial states, physical parameters, model configurations or learned components. The distribution of members is useful only to the extent that those perturbations resemble the errors in actual forecasts.
Five out of ten members predicting an event may suggest a nominal 50% ensemble proportion. That does not prove that events occurring with that nominal probability will happen half the time across many comparable cases.
Specify the event’s location, valid time, lead time, threshold and observed verification method. One forecast of daily rain over a region cannot be evaluated against a single hourly gauge measurement without a declared comparison.
Probability has operational meaning when it is checked over repeated cases. The aim is honest, useful uncertainty rather than a visually wide ensemble plume.
CHAPTER 2 OF 12 · Define ensemble predictions
2. Calibration, resolution and sharpness measure different qualities
Reliability or calibration asks whether events assigned probability p occur with frequency approximately p among sufficiently many comparable forecasts. Resolution asks whether probabilities differ meaningfully between situations with different observed risks.
Sharpness describes the concentration of predictive distributions independently of what ultimately occurred. More concentrated forecasts can be valuable, but artificially narrow intervals that miss observations often are not reliable.
An ensemble may have nearly correct mean temperature yet produce intervals that are too narrow. Conversely, it may produce broadly reliable intervals but little discrimination between easy and dangerous weather days.
A calibrated probability distribution may still be physically incoherent across locations if each station is corrected separately without preserving spatial dependence. Calibration quality must be evaluated against the intended decision.
ECMWF’s forecast guide and post-processing examples distinguish calibration from forecast information content. No one summary metric should be allowed to hide these different requirements.
CHAPTER 3 OF 12 · Calculate scores
3. Worked example: transform five ensemble temperatures
Consider five hypothetical temperature ensemble members at one station and valid time: [18, 19, 20, 21, 22] °C. Their mean is (18 + 19 + 20 + 21 + 22)/5 = 20 °C, and their population standard deviation is √[(4 + 1 + 0 + 1 + 4)/5] = √2 ≈ 1.41 °C.
For an arithmetic illustration only, suppose a previously validated calibration rule shifts mean to 21 °C and multiplies deviations about the original mean by 1.5. Define x’ = 21 + 1.5(x − 20).
The transformed members are [18.0, 19.5, 21.0, 22.5, 24.0] °C. Their mean is now 21 °C and population spread is 1.5√2 ≈ 2.12 °C. The transformation separately changes location and dispersion.
Imagine the verifying observation happens to be 23 °C. It lies outside the original member range [18,22] and within the transformed range [18,24]. One such case does not show that the mapping improved long-run probability calibration.
Most operational calibration fits parameters from historical forecast–observation pairs, with suitable time and region definitions. The numbers here merely demonstrate how a rule would be applied after its independent justification.
CHAPTER 4 OF 12 · Calculate scores
4. Worked example: a Brier score can improve without guaranteeing skill
For a yes/no event with forecast probability p and verifying outcome o equal to 0 or 1, the Brier score is BS = mean[(p − o)²]. Lower scores are better, and probabilities must be compared against the same event definition.
Take two invented cases. In case A, p = 0.80 and o = 1, giving squared error (0.8 − 1)² = 0.04. In case B, p = 0.60 and o = 0, giving (0.6 − 0)² = 0.36. The average BS is (0.04 + 0.36)/2 = 0.20.
Now imagine a hypothetical alternative that issues p = 0.70 for case A and p = 0.30 for case B. Squared errors are 0.09 and 0.09, so average BS = 0.09. The illustrative alternative gives a lower score on those two outcomes.
Because the alternative probabilities were selected for teaching after outcomes were described, this is emphatically not a valid calibration experiment. A real improvement must be learnt without using the evaluation cases and then verified on held-out observations.
Comparing a Brier score with a climatological or operational reference offers more context. A small score for a rare event can be achieved simply by often forecasting a small event probability, even without useful discrimination.
CHAPTER 5 OF 12 · Test calibration
5. Worked example: CRPS scores a full predictive distribution
The continuous ranked probability score (CRPS) compares a forecast cumulative distribution with an observed outcome across thresholds. Lower CRPS is better; for a continuous variable its units match the predicted variable.
For an empirical ensemble of n equally weighted scalar members xi and observed value y, CRPS = (1/n)Σ|xi − y| − [1/(2n²)]Σi,j|xi − xj|. The second term accounts for the ensemble’s internal spread.
Take the invented ensemble [0, 2, 4] in a consistent unit and verifying observation y = 2. The first term is (2 + 0 + 2)/3 = 4/3. Across all ordered pairs the absolute member differences sum to 16, making the second term 16/(2 × 9) = 8/9.
Thus CRPS = 4/3 − 8/9 = 4/9 ≈ 0.444 in the variable’s unit. This is a mathematical one-case result, not a claim of calibration or superiority.
Forecast calibration is evaluated over a representative archive. METplus documents empirical and distribution-fitting CRPS variants and warns against normal approximations for many precipitation distributions; the score’s calculation convention must be stated.
For the distinction between scoring this empirical forecast and estimating a larger predictive distribution's score, read Why Finite-Ensemble Score Corrections Need an Independence Assumption. Original finite ensembles test independent, copied, opposite-pair and partially dependent members without changing the empirical formula above.
CHAPTER 6 OF 12 · Test calibration
6. Rank histograms diagnose but do not uniquely identify problems
To construct a rank histogram, sort the ensemble members for each verifying case and record where the observation falls, including below the smallest member and above the largest. Accumulate these ranks across many matched forecasts.
Under appropriate exchangeability and sampling assumptions, a well-behaved ensemble can yield a relatively flat rank histogram. A pronounced U-shape is often associated with underdispersion, while a hump shape may indicate overdispersion.
Systematic displacement of the histogram toward one side may suggest forecast bias. However, ties, small ensemble size, observation error and heterogeneous forecast regimes can also shape the diagram.
An approximately flat histogram does not prove every location-specific or threshold probability is calibrated. Different errors can compensate in aggregate, and conditional biases may remain hidden.
ECMWF guidance discusses rank and reliability displays separately. Always look at forecast bias, event-specific reliability, climatological dependence and sampling uncertainty before adjusting a model.
Forecast members may have unequal skill or shared model biases, particularly in multimodel ensembles. Simple member counting assumes an empirical distribution whose reliability must be tested rather than assumed.
A raw system may be cold-biased while otherwise representing variability well, or it may have the right mean but too little spread. These different defects call for different post-processing methods.
Forecast lead and season matter. A rule derived for next-day temperature in one coastal climate may fail for ten-day precipitation in a mountain region.
Reference observations can also be uncertain: a point station and a grid-box mean are not identical targets. Misrepresenting their difference as forecast error can spuriously suggest underdispersion.
Ensemble size is finite. ECMWF notes that probability and score estimates vary when few members sample a continuous distribution, which makes apparent calibration sensitive to sampling and binning choices.
CHAPTER 8 OF 12 · Preserve usefulness
8. Statistical post-processing learns from past forecast errors
Ensemble model output statistics, often called EMOS, can fit a predictive distribution with parameters linked to ensemble mean, spread or other predictors. Bayesian model averaging is another approach that combines member-related predictive distributions with weights.
Calibration may correct mean bias, dispersion or probability thresholds. Model family and link functions should suit the predictand: rainfall has a point mass at zero and a strongly skewed positive distribution, unlike an often approximately Gaussian temperature variable.
Train on paired historical forecasts and observations with fixed lead, site or regional definitions. Reserve independent testing periods, since measuring success on the same cases used to fit the calibration will exaggerate performance.
Post-processing parameters may need rolling updates as the underlying forecast system or observation network changes. A new forecast-model version can invalidate a long training archive unless this effect is addressed.
ECMWF reports practical examples using ensemble post-processing, but their particular skill improvements belong to those evaluated systems and should not be copied as universal gains.
CHAPTER 9 OF 12 · Preserve usefulness
9. Dependence across time, variables and locations still matters
A forecast for precipitation in two adjacent districts, or wind and waves at a coastal site, is not merely a pair of unrelated marginal distributions. Their joint behaviour can determine the consequence of an extreme event.
Calibrating each station independently may yield reasonable marginal probabilities while breaking cross-site correlations. Ensemble copula coupling is one family of methods that can reorder calibrated quantiles using dependence information from an ensemble or another template.
Whether a dependence-restoring method works depends on the availability and relevance of its template, ensemble size, event type and downstream evaluation. A neat spatial map is not automatically a statistically coherent joint forecast.
Verify joint event frequencies, trajectories or spatial structures when the user decision depends on those. For a flood-warning chain, rainfall calibration alone is not proof of correct joint runoff across subcatchments.
Model design must respect the system that will consume the forecast rather than optimizing only a convenient scalar score.
CHAPTER 10 OF 12 · Preserve usefulness
10. Calibration can fail when the weather regime or archive changes
A training set that contains mostly ordinary conditions may provide little information about very rare extreme rainfall or wind speeds. Extrapolation into those tails can be especially consequential.
Time series and neighbouring forecast locations are correlated. Randomly withholding individual samples may leak nearly identical weather cases into training and testing; use dates, storms or blocks that reflect dependence.
Stratify evaluation by lead time, season, region and threshold where sample size allows. Compare both raw and calibrated systems on matched cases, with confidence intervals and a reference forecast.
Calibration may improve average CRPS while worsening a critical threshold or timing feature. Report multiple metrics aligned to the actual decision.
An operational claim needs versioned raw forecasts, calibration parameters, observation quality, training cutoff and a held-out verification result. Without these, apparent improvement may be a retrospective fit.
CHAPTER 11 OF 12 · Connected forecast routes
11. Practice, FAQs and a changed-outcome transfer
Practice A: Transform the toy members [18, 19, 20, 21, 22] using x’ = 21 + 1.5(x − 20). Verify the transformed mean of 21 °C and distinguish moving the mean from increasing dispersion.
Practice B: For an event with predicted probability 0.4 and observed outcome 1, the one-case Brier contribution is (0.4 − 1)² = 0.36. Explain why it is not enough to determine long-run reliability.
FAQ — Is an ensemble with more members always better calibrated? No. More members can represent a given modeled distribution more finely but cannot automatically remove structural biases.
FAQ — Does a flat rank histogram guarantee useful predictions? No. It is one diagnostic and can obscure conditional errors or poor discrimination.
FAQ — Is the narrowest predicted interval always most useful? No. Sharpness must be balanced against calibration and the cost of excluded outcomes.
The transfer test changes from continuous temperature to binary severe-rain occurrence. Reconsider the forecast distribution, verification unit, event base rate, calibration model and decision threshold rather than copying a temperature correction.
These verified eduKateSG routes connect the specialist method to related observables and interpretation problems. A neighbouring article can inform, but cannot substitute for, the required calibration or validation.
- The Gold Standard of Weather Forecasting — raw numerical and ensemble forecasting
- The Gold Standard of Weather Forecast Verification — deterministic and probabilistic score definitions
- The Gold Standard of Meteorology — physical atmospheric predictability
- The Gold Standard of Atmospheric Science — observational constraints and weather mechanisms
- The Gold Standard of Statistics — calibration, proper scores and independent testing
- The Gold Standard of Water Balance Modelling — downstream rainfall and storage decisions
- The Gold Standard of Wind Farm Wake Modelling — wind uncertainty in a distinct energy-system context
CHAPTER 12 OF 12 · Connected forecast routes
12. Connected weather science and statistical learning routes
Ensemble calibration builds on weather forecasting and verification but changes the inferential question. Producing many members is a modelling strategy, while demonstrating well-calibrated probabilities is an empirical performance result.
The linked roots help explain the physical sources of uncertainty, the appropriate verification score and the water or wind decisions that may use the calibrated output.
Institutional and scientific references for understanding the measurement, analysis and assumptions:
- ECMWF: Statistical Post-processing of Ensemble Forecasts — reliability, sharpness and calibrated distribution methods
- ECMWF: Probabilistic Data Statistical Concepts — rank histograms, reliability and ensemble spread
- METplus Verification Measures: CRPS — empirical continuous ranked probability score and fit conventions
- ECMWF: How Many Ensemble Members Are Desirable? — finite ensemble sampling and forecast uncertainty
- WMO Subseasonal Prediction Verification — reliability diagrams, discrimination and fair ensemble scoring
The numerical examples specify their own toy distributions, decision outcomes and conventions. Calibration rules in practice must be fitted before independent verification, and no operational skill claim follows from these examples.
The educational standard is to name the evidence, show the working, test uncertainty and transfer understanding. The immutable Clementi Secondary 1 Mathematics teaching reference illustrates this instructional discipline rather than supplying scientific evidence for Weather Ensemble Forecast Calibration.