The Gold Standard of Weather | Forecast Reliability, Sharpness and Decision Thresholds

Before interpreting a percentage, finish the event sentence. What must happen, where, during which period and under which observation rule? A forecast of measurable rain at a specified point during an hour differs from a forecast of heavy rain somewhere in a region during a day. The shared word rain does not make the events interchangeable.

For this fictional exercise, the event is rain meeting a teacher-supplied measurement criterion at the display location during a stated afternoon interval. The criterion is part of the exercise. It is not presented as the definition used by every meteorological service or as an operational warning threshold.

Write the issue time as well as the valid period. A prediction made the previous evening and an update issued ten minutes before the interval do not have the same lead time. If you combine them carelessly, a verification table can reward the system that received much later information without explaining that advantage.

Preserve the original wording. A statement that rain is possible cannot automatically be turned into a numerical probability after seeing the outcome. Similarly, an area forecast should not be scored against one convenient point without checking that the verification method matches the product.

Meteorological Service Singapore’s 24-hour forecast page provides a current official product route. Read each service’s own conventions. This guide does not assign a probability interpretation to icons or terms that the issuing service has not defined that way.

The first learning check is therefore a comparison of definitions. Give students two forecast sentences and ask whether they predict the same event. If the place, period or criterion differs, repair that mismatch before calculating which system performed better.

Contents · Next chapter

A probability describes uncertainty before the outcome is known. If an event is assigned 20%, both occurrence and non-occurrence remain possible. Observing one event does not retrospectively prove that its probability was 100%, just as one dry interval does not prove that it was zero.

Reliability, often discussed through calibration, concerns how issued probabilities relate to observed frequencies across suitable groups of forecasts. If a system repeatedly issues 20% for comparable events, one relevant question is whether those events occur about one fifth of the time over an adequate verification sample.

The word about matters. A finite sample fluctuates, and weather cases can be related. Ten forecasts at 20% need not produce exactly two occurrences. A perfect two-out-of-ten match is useful for teaching arithmetic but does not certify a forecasting system.

Imagine ten fictional forecasts, all at 20%. Two events occur. The observed frequency is 2 ÷ 10 = 20%. Now imagine another set of ten in which three occur. The frequency is 30%. The difference invites investigation of sample size, conditions and uncertainty; it is not enough by itself to announce that the second system is fundamentally unreliable.

ECMWF’s archived statistical concepts guide discusses reliability and probabilistic verification. The examples here are original teaching constructions, rather than ECMWF performance results.

Ask a learner to explain why a rare event can occur under a good forecast. Then ask what repeated pattern would raise a stronger concern. This separates ordinary surprise from evidence that a system’s stated confidence is systematically out of line with outcomes.

Contents · Previous chapter · Next chapter

Consider thirty fictional forecast cases from System A. Ten forecasts are 20%, ten are 50% and ten are 80%. The corresponding event counts are two, five and eight. Every record uses the same event definition, location rule and lead time for the purpose of this exercise.

The table has perfect bin agreement in this deliberately tidy sample. It also contains only ten cases in each bin. Both facts belong in the explanation. A line connecting the three points should not imply that probabilities between them were issued or evaluated.

The total event count is fifteen out of thirty, or 50%. That overall frequency does not replace the three conditional frequencies. The useful feature is that the observed event rate differs between low- and high-probability groups in the same direction as the forecasts.

Keep the denominator beside every percentage. A displayed 80% based on eight of ten cases has a different evidential basis from 800 of 1,000, even when the arithmetic ratio matches. Neither sample size alone settles representativeness, independence or whether the event definition was appropriate.

For a repair task, remove the case-count column and ask what the reader can no longer assess. For extension, change the group sizes to ten, one hundred and ten. The overall event frequency then needs weighting by case counts. Simply averaging three displayed percentages would describe an equally weighted group average, not necessarily the frequency across all cases.

Contents · Previous chapter · Next chapter

Now compare System A with System B, which issues 50% for every one of the same thirty cases. Fifteen events occur, so B’s single 50% group also matches the overall frequency perfectly in this sample. Yet B never distinguishes a lower-risk case from a higher-risk case.

Sharpness concerns the distribution of the issued probabilities: how much the system moves away from an uninformative middle towards more specific probabilities. It can be examined from the forecasts themselves. Reliability requires comparing forecasts with outcomes. A confidently wrong system can be sharp and poorly reliable.

Resolution concerns the system’s ability to separate cases associated with different observed event frequencies. In this toy dataset, A’s groups have frequencies of 20%, 50% and 80%, while B places every case in one group. Sharpness and resolution are related ideas, but they should not be treated as synonyms.

The practical aim is useful discrimination with trustworthy confidence. Always saying 50% may avoid some dramatic mistakes, but it can leave the user with little information beyond the background event rate. Always saying zero or one can look decisive while overstating what the available evidence supports.

Ask students to describe two systems without choosing a winner yet. Which issues a wider range of probabilities? Which separates observed outcomes into groups with different frequencies? Which has reliable probability groups in this sample? The questions reveal several aspects of quality rather than hiding them inside a vague word such as accurate.

Then change the event frequency or population. B’s apparent reliability at 50% may disappear. A claim about calibration belongs to the cases and conditions evaluated, not permanently to the name of a forecasting system.

Contents · Previous chapter · Next chapter

For a binary event, a Brier score averages squared differences between each forecast probability and its outcome. Write occurrence as 1 and non-occurrence as 0. Use probabilities between 0 and 1, not percentages between 0 and 100. Lower is better for this score under the stated convention.

For one 20% forecast followed by rain, the squared error is (0.2 − 1)² = 0.64. For a 20% forecast followed by no rain, it is (0.2 − 0)² = 0.04. The event is more surprising under the first prediction, and the score reflects that difference.

In A’s ten 20% cases, two events and eight non-events contribute 2 × 0.64 + 8 × 0.04 = 1.6. The ten 50% cases contribute 10 × 0.25 = 2.5. The 80% group contributes another 1.6 by the corresponding calculation.

Total squared error is 5.7 across thirty cases, so A’s Brier score is 5.7 ÷ 30 = 0.19. B’s constant 50% forecast scores 0.25 on every case, giving a mean of 0.25. A has the lower score on this matched fictional sample.

Do not square the difference between the overall mean probability and overall event frequency instead. Both systems have a mean probability of 50% and the observed frequency is 50%. That shortcut would produce zero while discarding the individual forecast-outcome relationships that the score evaluates.

ECMWF’s statistical guide supports the mean-squared-probability-error definition. The values 0.19 and 0.25 are calculated from the original classroom dataset here. They are not scores reported for any real forecast provider.

Contents · Previous chapter · Next chapter

Create System C by making A’s probabilities more extreme: change 20% to zero, keep 50%, and change 80% to 100%. Do not change the outcomes. The new system is sharper in this example, but its extreme probability groups no longer agree with their observed frequencies.

The low group contains two events that C assigned zero probability. Each contributes a squared error of 1; the eight non-events contribute zero. The middle group contributes 2.5. The high group contains two non-events assigned 100%, adding another 2. Total squared error is 6.5.

C’s Brier score is 6.5 ÷ 30, approximately 0.217. It is worse than A’s 0.19 even though C sounds more certain. It remains better than B’s 0.25 on these particular data. Confidence and performance have several relationships, which the arithmetic makes visible.

This does not mean that extreme probabilities are always inappropriate. A well-supported high-confidence prediction can be useful. The problem is increasing confidence without evidence that justifies it.

For a learner check, ask why C can be worse than A while still better than B. The answer should refer to the actual probability-outcome pairs, not a slogan that confidence is either good or bad. Good judgement preserves useful distinctions while keeping uncertainty honest.

Contents · Previous chapter · Next chapter

A score becomes easier to interpret when compared with a relevant reference forecast on the same cases. The reference might use a historical event frequency or another defined method. It must be constructed using information legitimately available for the task, rather than hindsight about the evaluation outcomes.

In the classroom calculation, take a supplied 50% reference forecast. Its Brier score is 0.25. A simple Brier skill score is 1 minus the system score divided by the reference score. For A, that gives 1 − 0.19/0.25 = 0.24, or 24% under this defined scoring comparison.

The 24% is a relative reduction in that score against that reference. It is not a statement that 24% more forecasts were correct, that rainfall amounts improved by 24%, or that a user’s costs fell by 24%. The denominator determines the meaning.

A skill score can be negative. If a deliberately poor fictional system scores 0.50 against a reference scoring 0.25, the skill is 1 − 0.50/0.25 = −1. That means the system has twice the reference’s mean squared probability error under this comparison. It does not mean a negative probability or a negative count of correct forecasts. The result is a relative scoring statement. Keep that interpretation beside the number, especially when percentages make a headline seem more intuitive than its actual definition.

A reference score of zero creates a different problem: this ratio is undefined. Do not force a division by zero into a dramatic skill percentage. State that the chosen relative-score formula cannot be evaluated with that denominator and retain the underlying scores.

Reference choice matters. A seasonal or location-specific baseline may be more informative than one pooled background frequency. Comparing scores across different populations can also change the underlying difficulty. A system evaluated during rare events and another during common events should not be ranked from raw scores alone without examining the samples.

For students, provide two reference scores and ask how the skill value changes while A’s score stays fixed. Then ask which reference is appropriate and why. The exercise makes it clear that a relative performance number belongs to a comparison, not solely to the forecast being advertised.

Keep the underlying scores available. A skill headline without its event definition, sample and reference makes independent interpretation difficult. A useful report allows another reader to reconstruct the ratio and understand the question it answers.

Contents · Previous chapter · Next chapter

A forecast and a decision are different objects. The forecast estimates an event probability. The decision combines that estimate with consequences, available actions and relevant requirements. Two users can reasonably make different choices from the same probability because their tasks differ.

In the fictional display example, moving replaceable materials indoors costs 10 units. Leaving them outdoors incurs a loss of 100 units if the event occurs and zero if it does not. Assume moving them prevents that loss completely and that these are the only costs. Those assumptions define a deliberately simple teaching model.

Expected loss from leaving the materials out is 100p, where p is the event probability. Moving costs 10. The model favours moving when 100p exceeds 10, or p exceeds 0.1. At exactly 0.1, the two options have equal expected cost under these assumptions.

The number 10% is a consequence of this invented cost-and-loss pair. It is not a weather safety threshold, a public warning rule or an instruction for deciding whether people should remain outdoors. Real safety obligations, uncertain consequences and available alternatives cannot be replaced by this classroom ratio.

Change the moving cost to 40 units. The corresponding model threshold becomes 40%. Change it to 70 and the threshold becomes 70%. The forecast can remain identical while the economically preferred action changes.

A good learner can explain both sides: the probability belongs to the forecast question, while the threshold belongs to the user’s stated decision model. Confusing them can lead someone to accuse a forecast of inconsistency when different users are actually making different decisions.

Contents · Previous chapter · Next chapter

Use A’s thirty-case table with a moving cost of 40 and an unprotected event loss of 100. Apply a rule that moves materials when the issued probability exceeds 40%. The twenty cases forecast at 50% or 80% incur moving costs of 20 × 40 = 800 units.

The ten 20% cases are left unprotected. Two events occur there, adding 200 units. Total realised cost under the supplied outcomes is 1,000 units. This is an ex-post calculation for the fictional sample, not a guarantee of future cost.

Compare two simple alternatives. Always moving costs 30 × 40 = 1,200. Never moving incurs losses for fifteen events, or 1,500. The forecast-based rule costs less in this particular dataset.

System B always issues 50%, so the same threshold rule moves materials in every case. Its cost is 1,200. The comparison illustrates why distinguishing cases can have decision value even when a constant forecast is calibrated to the overall frequency.

Now change the moving cost to 10. All three A probability groups exceed the model’s 10% threshold, so all materials move. The detailed distinction between 20%, 50% and 80% does not change this user’s action in the supplied model. Forecast quality and value for one particular decision are related, but not identical.

Ask the learner what changed: the forecast, the outcomes, the action cost or the decision rule? A clear answer keeps the comparison honest. Reporting that A saved 500 units without naming the reference policy and cost assumptions would conceal much of the result’s meaning.

Contents · Previous chapter · Next chapter

A comparison needs matched questions. If one forecast is evaluated at a six-hour lead and another at a one-hour lead, the later forecast may benefit from information unavailable earlier. That may be useful operationally, but it is not a like-for-like comparison of the same forecasting task.

Keep location, period, event criterion, issue time and observation method consistent or explain differences. A forecast for one point should not inherit the outcome from a nearby station simply because that station has a complete record. The representativeness question needs an appropriate method.

Record every eligible case before outcomes are known. Dropping difficult days, missing forecasts or cases with inconvenient observations can bias the apparent performance. A missing observation is not automatically a non-event. A missing forecast should not quietly disappear from the service-availability account.

Separate forecast quality from delivery quality. A probability can be well calibrated but arrive after the user has to decide. A less detailed forecast delivered in time can still have value, while the system should continue improving both prediction and communication.

For a classroom audit, give students two otherwise identical score tables with different lead-time descriptions. Ask which comparison the scores support and which broader claim remains unresolved. This develops a practical understanding of evidence fairness that transfers beyond weather to sports, demand forecasting and educational assessment.

Contents · Previous chapter · Next chapter

Thirty forecast rows do not necessarily represent thirty independent situations. Several forecasts may concern the same storm, overlapping periods or neighbouring locations. Their outcomes can be related. Counting every row as independent can overstate the amount of distinct evidence.

Group the records according to the question. A learner can mark which forecasts share an event and explain why a run of errors might reflect one developing system rather than many unrelated failures. Appropriate statistical uncertainty needs analysis that respects those relationships.

Conditions can also change. A verification record spanning different instruments, forecast versions or observation definitions may combine unlike periods. Preserve those changes. A more favourable score after an update is interesting, but differences in the evaluation population may also matter.

Leakage occurs when information from the outcome or later time enters what is presented as an earlier forecast. Rewriting a probability after observing rain produces a better-looking table while destroying its meaning as a prediction. A model trained or tuned using the same cases used for final evaluation also needs appropriate separation and disclosure.

For a useful demonstration, let one fictional system issue probabilities before outcomes and another choose probabilities after seeing them. The second can achieve a perfect score by writing one for events and zero for non-events. Its score does not establish forecasting ability because it did not solve the forecasting task.

The lesson is straightforward: preserve the time at which information was available. Verification becomes credible when the record distinguishes prediction, update, observation and retrospective analysis.

Contents · Previous chapter · Next chapter

Recalibration adjusts issued probabilities using evidence about their relationship with outcomes. If a system’s 80% forecasts repeatedly correspond to substantially fewer events in an appropriate historical sample, a suitable calibration method may change those probabilities. The method needs independent evaluation rather than a promise that a fitted historical curve guarantees future reliability.

The toy example suggests one issue: C’s 100% group contains only 80% observed occurrences. Simply changing that label to 80% would fit this sample. But using the same thirty cases both to choose the correction and to certify its performance would exaggerate the evidence.

A better learning design separates a supplied development sample from a later evaluation sample. Students propose a correction using the first, freeze the rule, then apply it to the second. They can compare what improved, what did not and whether the new cases resemble the earlier ones.

Calibration also does not automatically create information that was absent. A constant forecast adjusted to the correct background frequency can become reliable while still failing to distinguish cases. The repair should match the diagnosed weakness: confidence, discrimination, event definition, data quality or delivery timing.

For practical communication, explain the change in ordinary language. State that the revised probabilities use a defined historical relationship and identify where performance remains uncertain. Avoid presenting a more sophisticated statistical label as proof that every forecast is now trustworthy.

The strongest next step is a maintained evaluation process. Preserve forecasts, check observations, inspect relevant groups and revisit assumptions when the system or environment changes. That process supports learning far better than selecting one successful episode as permanent proof of skill.

Contents · Previous chapter · Next chapter

The repair route begins with a percentage and its denominator. Ask the learner to interpret two events in ten cases before introducing diagrams or scores. If they say a 20% forecast was wrong because rain occurred, return to the distinction between probability and one outcome.

The stabilisation route compares A and B. Both match an observed frequency under their grouping, but only A separates cases. Ask the student to explain reliability and information without relying on the words calibrated or sharp. Clear ordinary language reveals whether the concepts are understood.

The extension route adds the Brier score, reference skill and cost-loss model. Keep each calculation attached to its question. The score assesses probability-outcome pairs. The skill score compares scoring performance with a reference. The decision model compares consequences under stated assumptions.

Mix questions after the first guided examples. One asks for an observed frequency, another for a squared error, another for a decision threshold and another for a limitation. The learner must choose a method rather than repeat the operation from the preceding paragraph.

Finish with a changed setting such as a fictional stock-demand forecast for an event stall. Keep the probability reasoning while changing costs, lead times and consequences. The student should identify which mathematical relationships travel and which weather-specific definitions need replacing.

This is evidence of transfer: the learner can preserve uncertainty, calculate a comparison and explain why a decision changes even when the forecast does not. It is a stronger achievement than remembering that a particular probability sounds high or low.

Contents · Previous chapter · Next chapter

A forecast can match the overall event frequency while performing differently across meaningful subgroups. The pooled table may be mathematically correct and still miss a pattern important to a user. The next step is to examine groups connected to the forecasting task, with enough evidence to interpret them.

Consider twenty fictional forecasts, all at 50%. Ten concern Location East and ten Location West under a supplied comparable event definition. Nine events occur at East and one at West. Across both locations, ten of twenty events occur, so the pooled observed frequency is 50%.

The overall probability group matches perfectly in this tiny example. Within the locations, the observed frequencies are 90% and 10%. A single pooled reliability point conceals those different patterns. It does not establish that the system gives an equally useful probability for both places.

The example is deliberately extreme and small. It does not prove a persistent local bias from ten cases. It shows why a relevant conditional check can matter and why sample size remains important after splitting the data.

Now change the mix while keeping the two location frequencies as supplied model rates. A population dominated by East has a different overall event frequency from one dominated by West. A constant 50% forecast calibrated to the original balanced mixture may no longer match the new population. The forecast label has not changed, but its evaluation context has.

This is a reason to preserve location, season, lead time and other relevant fields. They allow a reviewer to ask whether an aggregate result applies to the actual decision. It is not a reason to divide data into hundreds of tiny groups until every random fluctuation looks like a discovery.

Choose subgroup questions from the task and evidence. A location-specific user may need a location check. A decision made the previous evening may need a lead-time check. A claim about a particular event type may need that event definition kept separate from easier cases.

For a classroom exercise, ask students to explain three levels of conclusion. The pooled arithmetic is exact for the supplied records. The subgroup pattern is also exact for those records. A claim that the pattern will persist in future weather needs additional evidence and an appropriate analysis. Separating those levels prevents both overclaiming and the mistaken idea that a small dataset teaches nothing.

Then ask what the next investigation should preserve. A new matched record across both locations, with forecasts saved before outcomes, addresses the question more directly than collecting unrelated forecasts from another region. A useful next sample follows the uncertainty revealed by the first one.

The same lesson applies to a school cohort average or a network-wide reliability rate. An overall match can conceal a mismatch affecting a particular group. Good verification looks beyond one headline while keeping each additional claim proportionate to the evidence available.

Contents · Previous chapter · Next chapter

Is a 20% forecast disproved when the event happens?

No. Occurrence is possible under that probability. Reliability is examined through appropriate collections of comparable forecasts and outcomes, with uncertainty and dependence considered.

Can a constant forecast be calibrated?

It can agree with the overall event frequency in a defined population while offering little discrimination between cases. Calibration and useful separation of cases are different aspects of forecast quality.

Is a sharper forecast always better?

No. More extreme probabilities can overstate confidence. Compare them with outcomes and relevant scores, rather than rewarding decisiveness by appearance alone.

Does the best Brier score guarantee the lowest cost for every user?

No. A user’s consequences, action options and decision rule matter. A probability forecast can have different value for different tasks even when its quality is assessed on the same cases.

Can the cost-loss ratio replace official weather procedures?

No. The display-material example is a simplified economic teaching model. It does not set safety thresholds or override current warnings and responsible organisational procedures.

What should a forecast-learning notebook preserve?

The prediction, issue time, valid period, location, event definition, observation source, outcome and any missing or changed information. Preserve updates as updates rather than rewriting the earlier record.

Contents · Previous chapter · Next chapter

For the wider chain, return to Weather. For general methods, read Forecasting, Probability, Measurement and Data Literacy. These complement the worked weather-verification task rather than replace its event definitions.

For consequences and communication, use Flood Early Warning and Urban Flood Resilience. For water timing, the Runoff Hydrographs guide distinguishes volume from a changing flow record.

ECMWF’s linked statistical concepts resource supports the limited definitions of reliability, sharpness and probability scoring used here. Its forecast-quality overview provides an operational verification route. MSS’s Heavy Rain Warning page is a current official information route, separate from the fictional decision model. Sources were checked on 21 October 2026. Systems A, B and C, all counts and all costs are original teaching examples.

Explore the How X Works library for another context, or return to Learning That Transfers into Life to plan the next learning step.

Contents · Previous chapter · Return to Weather