THE GOLD STANDARD OF HYDROLOGY · SPECIALIST STUDY
Blend forecast systems only after proving what each component adds and when it should leave
During a numerical-weather or hydrological forecast-system upgrade, teams may have a mature postprocessor for the retiring system and limited reforecasts for the new one. Running both can reduce transition risk, but a naïve mixture can double count the same meteorology, preserve old bias, understate dependence and keep a legacy pipeline alive indefinitely. The gold standard treats blending as a versioned transition policy with identifiable inputs, proper-score evaluation, dependence checks, gates, rollback and an explicit decommissioning path.
This guide belongs to the Hydrology subject root, the Gold Standard of Education, and the How X Works hub. It begins with a concrete decision, then tests whether the method still holds when the context changes.
Choose a reading route
Map the transition system · chapters 1, 2, 3
Build and diagnose the blend · chapters 4, 5, 6
Verify value and dependence · chapters 7, 8
Set version gates · chapters 9, 10, 11
Decommission and transfer · chapters 12, 13
The whole problem at a glance
| Layer | Old branch | New branch | Transition question |
|---|---|---|---|
| Meteorological forcing | Version O | Version N | Are forecasts and reforecasts version matched? |
| Hydrological model | Calibration O | Calibration N or shared | Which errors are shared? |
| Postprocessor | Mature P-O | Young P-N | Does each add independent skill? |
| Operations | Stable pipeline | New pipeline | What gate permits promotion or rollback? |
Every output needs a lineage that reaches forcing, hydrological model, postprocessor, observation archive and code version.
CHAPTER 1 OF 13 · Map the transition system
1. Write the version lineage before choosing weights
Name the numerical-weather model cycle, ensemble configuration, reforecast version, hydrological model and calibration, observation archive, postprocessor, code and issue-time convention for each branch.
A label such as “old” or “new” is insufficient when components change on different dates. A new atmospheric model may feed an unchanged hydrological model while reforecasts arrive later, creating a hybrid branch.
Attach lineage to every forecast object and verification record. If a score cannot be tied to the versions that generated it, it cannot support a transition gate.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Change only the ensemble member count; even with identical variables, spread and postprocessor inputs change, so record a new compatible lineage. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 1 check · what should survive an audit
- Version forcing, hydrology, postprocessor and observations.
- Identify hybrid branches.
- Reject untraceable scores from gate evidence.
CHAPTER 2 OF 13 · Map the transition system
2. Match real-time forecasts to compatible reforecasts
Reforecasts provide historical forecasts produced with a model version aligned to the real-time system. Calibration on a different version can encode obsolete bias and spread relationships.
When new-version reforecasts are short or sparse, state the limitation rather than silently combining them with a long old archive. Pooling versions can increase sample size while destroying exchangeability.
Create a compatibility matrix for real-time cycle, reforecast cycle, variables, lead times, ensemble size and preprocessing. Mark exact, bridged and incompatible cells.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Use on-the-fly reforecasts instead of a fixed archive; the model-version match improves, but sample availability and rolling reproducibility change. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 2 check · what should survive an audit
- Verify forecast-reforecast version match.
- Do not pool incompatible archives for convenience.
- Publish the compatibility matrix.
CHAPTER 3 OF 13 · Map the transition system
3. Define what the transition blend is meant to protect
A transition blend may aim to preserve calibration, reduce discontinuity, hedge new-system uncertainty or maintain service while the new postprocessor matures. These goals imply different weights and end dates.
Write the target variable, locations, lead times, seasons, thresholds and decision loss. A single global blend can hide that the new system improves floods but degrades low flows, or helps short leads but not week-two forecasts.
Choose a bounded transition horizon and review dates. Without them, “temporary blending” becomes an undocumented permanent ensemble.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Change the service objective from mean discharge to flood-threshold exceedance; weight learning and verification should follow the new decision target. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 3 check · what should survive an audit
- Name the transition purpose.
- Stratify by variable, lead, season and decision.
- Set an initial sunset date.
CHAPTER 4 OF 13 · Build and diagnose the blend
4. Start with a transparent linear pool
A linear pool combines predictive distributions or probabilities with non-negative weights summing to one. It provides a clear baseline and preserves the ability to trace each branch’s contribution.
In the fictional case, weights 0.40 and 0.60 combine exceedance probabilities 0.30 and 0.55 into 0.45. This is not an average of discharges; it is a mixture of predictive beliefs.
Linear pooling often increases dispersion. That may help an underdispersed ensemble or worsen an already broad forecast. Verify calibration and sharpness after pooling rather than assuming diversification.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Combine quantile forecasts instead of full distributions; averaging equal quantiles does not generally produce the quantiles of a valid mixture distribution. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 4 check · what should survive an audit
- State whether you pool members, parameters, quantiles or distributions.
- Keep weights non-negative and summing to one for a standard linear pool.
- Verify dispersion after combination.
CHAPTER 5 OF 13 · Build and diagnose the blend
5. Learn weights without leaking the transition outcome
Weights fitted on the same events used to judge the blend reward hindsight. Use rolling-origin, event-wise or blocked validation that respects hydrological dependence and operational availability.
Estimate only as many weights as the data can support. Lead-, site-, season- and threshold-specific weights can overfit a short new-version archive. Hierarchical shrinkage or a small set of regimes may be more stable.
Record the training window and availability date. A forecast cannot use observations or reforecasts that were not available at its issue time.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Add a severe event that arrived after the promotion decision; use it for forward monitoring, not to rewrite the historical gate as if it had been known. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 5 check · what should survive an audit
- Use time-respecting validation.
- Control weight complexity.
- Enforce issue-time data availability.
Worked mixture for one exceedance probability
| Component | Forecast P(Q > q*) | Mixture weight | Contribution |
|---|---|---|---|
| Old-system postprocessor | 0.30 | 0.40 | 0.12 |
| New-system postprocessor | 0.55 | 0.60 | 0.33 |
| Linear pool | — | — | 0.45 |
| Observed outcome | 1 | — | Used for scoring, not weight hindsight |
The arithmetic is easy. The difficult questions are how weights were learned, whether components are dependent and whether 0.45 is calibrated.
CHAPTER 6 OF 13 · Build and diagnose the blend
6. Detect dependence before claiming diversification
Old and new branches may share observations, hydrological structure, boundary conditions and similar meteorological dynamics. Their errors can therefore be strongly dependent even when model labels differ.
Measure residual correlation by lead, basin, season and event type. Inspect joint misses, not only average correlation. Both systems may fail together during snowmelt, tropical rainfall or rating-curve error.
Effective diversification comes from complementary error, not component count. If branches are nearly redundant, a 50:50 mix can blur accountability without adding resilience.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Add a third postprocessor trained on the same observations and raw ensemble; do not count it as independent merely because its algorithm differs. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 6 check · what should survive an audit
- Map shared data and structure.
- Inspect joint tail failures.
- Estimate incremental value beyond the stronger branch.
CHAPTER 7 OF 13 · Verify value and dependence
7. Avoid double counting meteorological and hydrological uncertainty
A branch may already contain an ensemble of weather members and a residual-error model. Mixing two such predictive distributions can count similar uncertainty twice or distort member probabilities.
Document whether each component is a calibrated distribution, an equally weighted ensemble, a set of scenarios or a deterministic correction. The meaning of a component weight differs across these objects.
If pooling members, preserve within-branch weights and provenance. If pooling distributions, score the resulting distribution directly. Do not convert both to a large unweighted bag unless that representation is justified.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: One branch changes from 20 to 100 ensemble members; unweighted member pooling would silently multiply its influence by five. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 7 check · what should survive an audit
- Identify uncertainty already represented in each branch.
- Keep component and member weights distinct.
- Test sensitivity to ensemble-size changes.
CHAPTER 8 OF 13 · Verify value and dependence
8. Use proper scores, calibration diagnostics and operational loss together
Proper scoring rules reward honest predictive distributions. CRPS or threshold-weighted variants can assess univariate forecasts, while multivariate scores address joint behaviour. Calibration diagnostics reveal whether probabilities match frequencies.
No one score captures every service consequence. Pair proper scores with reliability, sharpness, rank or PIT diagnostics, event performance and the operational loss tied to warnings or reservoir decisions.
Compare the blend with each branch and simple baselines over identical cases. Report paired uncertainty and missing-run handling. A blend that looks best only because weak cases were excluded has not passed.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Change the warning threshold; threshold-weighted performance and decision loss can reorder the preferred blend while overall CRPS stays similar. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 8 check · what should survive an audit
- Use proper and diagnostic measures.
- Compare on identical cases.
- Report event and decision-relevant performance.
CHAPTER 9 OF 13 · Set version gates
9. Build gates that separate scientific from operational readiness
Scientific gates cover calibration, skill, dependence, extremes and subgroup performance. Operational gates cover latency, missing inputs, reproducibility, monitoring, security and rollback. Both must pass for live promotion.
A new branch may be scientifically promising but late or unreliable. Conversely, a stable pipeline may preserve poor calibration. Keep the gate dimensions visible so one success does not excuse another failure.
Assign owners and evidence windows. A gate with no measurement source, reviewer or action is a slogan.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Move computation to a new platform without changing models; scientific evidence may transfer, while operational reliability and lineage gates must be re-established. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 9 check · what should survive an audit
- Separate scientific and operational gates.
- Assign owner, metric, window and action.
- Do not average away a failed safety-critical gate.
Version-gate dashboard
| Gate | Evidence window | Failure response |
|---|---|---|
| Lineage completeness | Every run | Quarantine output |
| Calibration and proper score | Rolling and event-stratified | Reduce scope or weight |
| Dependence and spread | Rolling joint diagnostics | Re-estimate or change combination |
| Operational reliability | Latency, missing runs and rollback drills | Return to approved branch |
A model can be statistically promising yet operationally unfit, or reliable yet scientifically inferior.
CHAPTER 10 OF 13 · Set version gates
10. Use staged weights and version locks during cutover
Begin with shadow output, then a bounded advisory phase, then controlled live weights. Each stage should have maximum scope, weight and duration, with explicit evidence for expansion.
Lock branch versions within an evaluation window. If the new model changes again while weights are being assessed, start a new lineage or use a pre-defined bridge; otherwise the evidence mixes targets.
Publish the active weight table by basin, lead and issue date. A forecast should be reproducible from its archived components and policy version.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: The new weather centre issues an emergency patch mid-window; quarantine or version it rather than pretending continuity. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 10 check · what should survive an audit
- Stage exposure and weight ceilings.
- Lock or explicitly bridge versions.
- Archive the active weight policy.
CHAPTER 11 OF 13 · Set version gates
11. Plan rollback as a forecast product, not a meeting decision
Rollback must specify the approved branch, data feeds, retained capacity, trigger, authority and communication. If the old branch has already lost inputs or staff support, it is not a real fallback.
Exercise rollback with archived components and a live drill. Confirm that issue times, labels and downstream warning systems handle the change without mixing products.
After rollback, preserve the failed run and decision evidence. Do not overwrite it with replacement forecasts; users and reviewers need the actual chronology.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Rollback occurs after some users received the new blend; version and communicate revised guidance rather than silently substituting files. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 11 check · what should survive an audit
- Keep the fallback operational.
- Run a rollback drill.
- Preserve superseded outputs and notices.
CHAPTER 12 OF 13 · Retire safely and transfer
12. Decommission the legacy branch with dependency evidence
List every job, dashboard, archive, alert, API, report and user that consumes the old branch. A statistical decision to assign zero weight does not remove operational dependencies.
Set a last production date, archival format, reconstruction method and exception policy. Monitor for calls to the retired endpoint and resolve them before shutdown.
Retain the minimum artefacts needed to reconstruct historical forecasts and verify transition claims: code or container, configuration, training data references, weight policy and lineage.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Retire only the old postprocessor while retaining its raw model archive; document which historical products remain reproducible and which do not. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 12 check · what should survive an audit
- Inventory technical and human dependencies.
- Define last run and exception rules.
- Preserve historical reconstruction assets.
CHAPTER 13 OF 13 · Retire safely and transfer
13. Write the transition conclusion as a bounded claim
State where the blend improved, where it did not, the uncertainty, the evaluation period, missing cases, dependence, operational incidents and the gate decision.
Avoid declaring the new system universally superior. A transition policy can be accepted for certain basins and leads while the old branch remains temporarily necessary elsewhere.
Carry monitoring into steady state. Once the legacy branch is gone, drift, observation changes and later model upgrades still require calibration and version control.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Apply the framework to a switch between observation networks rather than forecast models; lineage, overlap, bridging and retirement still matter, but the error pathways differ. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 13 check · what should survive an audit
- Limit the claim to tested strata and dates.
- Report non-improvements and incidents.
- Continue monitoring after retirement.
Verification lab: work the decision twice
Old and new postprocessors forecast flood-threshold probabilities of 0.30 and 0.55. A 0.40/0.60 linear pool gives 0.45. Score the mixture as a predictive distribution over many cases; do not decide from this one realised flood. Compare calibration, sharpness and operational loss with both components.
The new branch shares the old hydrological model and observation-error correction, so residual correlation is 0.88. Explain why nominal component count exaggerates diversification. Test whether the blend adds paired proper-score value beyond the stronger branch and whether joint tail misses persist.
Change the new ensemble from 20 to 100 members. A component-level 0.60 weight should remain 0.60 if members are internally normalised. An unweighted combined member bag would instead increase the new branch’s influence, demonstrating hidden double counting.
Limits and misconceptions
A blend is not automatically more robust
Highly dependent components can fail together and merely blur provenance.
More members do not deserve more policy weight
Ensemble size and branch weight are different quantities.
A long old archive cannot simply calibrate a new model version
Model-version mismatch can encode obsolete bias and spread relationships.
Zero statistical weight is not decommissioning
Pipelines, archives, users and rollback obligations can keep the old branch operationally alive.
Questions readers ask
Why blend during a forecast-system upgrade?
- To hedge transition uncertainty, preserve service or bridge limited new-version training data—provided the purpose, evidence and sunset are explicit.
Can I average old and new quantiles?
- Not generally as a faithful distribution mixture. Define the forecast object and use a mathematically coherent combination.
How should weights be learned?
- With time-respecting validation, controlled complexity and issue-time availability, evaluated by proper scores and decision-relevant diagnostics.
What is double counting?
- Giving repeated influence to the same uncertainty or branch, for example by unweighted pooling when one system has many more members.
When can the old branch be retired?
- After scientific and operational gates pass, rollback and dependencies are resolved, and historical reconstruction is preserved.
What if the new model upgrades again during evaluation?
- Create a new version lineage or apply a pre-specified bridge. Do not pool evidence as if the target system remained unchanged.
Practise, verify and continue
Build a transition matrix for two weather-model cycles, two reforecast archives, one hydrological model and two postprocessors. Mark compatible cells, define a linear-pool baseline, a time-respecting weight fit, proper-score and tail diagnostics, version gates, rollback and a last-run plan.
Replace the old/new system pair with three regional hydrological models feeding one national product. Reassess dependence, member weighting, geographic scope and decommissioning; the version-governance framework transfers, but the transition may no longer have a single chronological old and new branch.
Practical evidence check
Before implementation, write a one-page evidence map with the decision, unit of analysis, version or time boundary, inputs, outcomes, uncertainty, authority and stop rule. Attach one worked case that passes and one that fails. A reviewer should be able to reproduce the reasoning without relying on the memory of the person who built the system.
Run one changed-context rehearsal before approval. Alter a mechanism rather than a cosmetic detail, then identify which calculation, safeguard or governance gate must change. Record any conclusion that does not transfer. This short rehearsal is often the fastest way to expose a hidden assumption before it becomes an operational dependency.
Related eduKate reading
Reforecast postprocessor transfer across model upgrades
Reforecast postprocessing leakage and honest holdout skill
Multivariate ensemble verification and joint calibration
Ensemble score decomposition and operational loss
Conditional ensemble verification and regime stratification
Authoritative sources and scope
ECMWF explanation of reforecasts and model-version matching
ECMWF S2S model-update handling guidance
Bias correction and performance-weighted multi-model blending in hydrology
Hydrological ensemble preprocessing and postprocessing evaluation
The sources support version matching, model-update documentation and hydrological postprocessing evaluation. The probabilities, weights and gate examples are fictional. Operational agencies should apply their own forecast responsibilities, data policies and warning protocols.