THE GOLD STANDARD OF HYDROLOGY · SPECIALIST STUDY
Prove that the new hydrological postprocessor remains live, calibrated and independent after cutover
A forecast-system cutover is not complete when a new dashboard appears. Hidden clients may still call the legacy endpoint, caches may serve old products, missing inputs may trigger undocumented fallback and calibration can drift after overlap ends. This guide builds surveillance from per-forecast lineage and synthetic probes through incident replay, dependency closure and a decommission-assurance case.
This guide belongs to the Hydrology subject root, the Gold Standard of Education, and the How X Works hub. It begins with a concrete decision, then tests whether the method still holds when the context changes.
Choose a reading route
Establish post-cutover truth · chapters 1, 2, 3
Detect silent fallback · chapters 4, 5
Watch calibration and service · chapters 6, 7, 8
Replay incidents · chapters 9, 10, 11
Assure decommissioning · chapters 12, 13
The whole problem at a glance
| Question | Evidence | Unsafe assumption |
|---|---|---|
| Which branch produced it? | Immutable per-product lineage | The dashboard proves source |
| Did fallback occur? | Explicit mode, trigger and duration | A successful file means new system |
| Who uses the old path? | Access logs plus retirement probes | No tickets means no clients |
| Is calibration drifting? | Outcome-aligned diagnostics by regime | Average score is enough |
Status, scientific performance and dependency retirement are separate claims.
CHAPTER 1 OF 13 · Establish post-cutover truth
1. Define cutover scope and steady-state responsibility
Name forcing cycle, hydrological model, observation feed, postprocessor, calibration archive, code image, configuration, issue convention and endpoint. Mark every legacy component as stop, archive or approved fallback. Assign scientific, platform, data, service and incident owners and set evidence windows matched to operational, seasonal and rare-event claims.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Revise a rating curve after cutover; lineage and scientific responsibility change though endpoint does not. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 1 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 2 OF 13 · Establish post-cutover truth
2. Attach immutable lineage to every forecast
Carry issue and valid time, location, variable, ensemble, forcing, hydrology, postprocessor, training vintage, configuration hash and mode with each product and an independent ledger. Filenames are insufficient because caches and reprocessing can place old content under current paths. Block or label incomplete combinations.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Reprocess yesterday with corrected observations; preserve original and revised lineage. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 2 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 3 OF 13 · Establish post-cutover truth
3. Create end-to-end synthetic probes
Exercise input ingestion, model run, postprocessing, publication, API retrieval and downstream parsing with a harmless canary or signed marker impossible for the legacy branch. Verify content, provenance, latency, cache age and permissions. Exclude probe traffic from scientific skill data.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Place the API behind a cache; add cache key, age and origin lineage to the probe. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 3 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 4 OF 13 · Detect silent fallback
4. Make every fallback an explicit forecast mode
Predefine fallback product, trigger, maximum duration, label, notification, validation and authority. Raw ensemble, previous calibration, climatology and legacy branch have different scientific meanings. Emit mode in every product and record entry, exit, affected locations and users. Never quietly copy the last file forward.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Change from previous-calibration to raw-ensemble fallback; update interpretation and skill expectations. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 4 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 5 OF 13 · Detect silent fallback
5. Detect legacy endpoint calls across paths
Instrument authenticated APIs, service accounts, jobs, batch files, dashboards, notebooks, partner feeds, mirrors and storage. Classify migration, approved fallback, unknown, scanner and probe. Stage observe, notify, require exception and refuse so hard shutdown does not merely drive users to unmonitored copies.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A partner switches to a mirrored bucket; endpoint traffic is clean while dependency survives. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 5 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
Worked silent-fallback incident
| Time | Signal | Risk | Action |
|---|---|---|---|
| 02:00 | New input late | Postprocessor cannot run | Enter declared degraded mode |
| 02:05 | Legacy file under current name | Lineage ambiguity | Block unproven publication |
| 02:12 | Probe sees legacy marker | Silent fallback | Notify and preserve both products |
| After event | Outcome and logs available | Process and calibration learning | Replay and repair |
Fallback is safe only when declared, versioned and bounded.
CHAPTER 6 OF 13 · Watch calibration and service
6. Trace caches, aliases and data copies
Map DNS aliases, gateways, storage prefixes, cache keys, symlinks, topics and exports with TTL and invalidation. A new hostname can serve old content when version is absent from the key. Combine static code and infrastructure search with runtime evidence because constructed paths and external clients escape repositories.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Increase cache TTL from five minutes to a day; recalculate how long legacy products survive. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 6 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 7 OF 13 · Watch calibration and service
7. Monitor calibration by regime, lead and place
Use reliability, rank or PIT diagnostics, coverage and sharpness appropriate to the predictive object. Stratify by basin, lead, season, flow, event severity and observation quality with sample-size caution. Define warning and action thresholds with persistence and physical diagnosis.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Observation latency grows only at high flow; separate outcome-quality drift from forecast drift. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 7 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 8 OF 13 · Watch calibration and service
8. Distinguish model, pipeline and observation drift
Changes can come from forcing, catchment behaviour, hydrological state, postprocessor, network or rating curve. Monitor inputs, feature distributions, residuals, revisions and latency. Use archived-input shadow recomputation before retraining; a narrow ensemble can be a missing-member fault rather than confidence.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A rating curve is revised retroactively; rerun verification by observation version. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 8 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 9 OF 13 · Replay incidents
9. Join scientific and operational surveillance
Calibration is insufficient if products miss warning deadlines. Monitor completion, latency, missing basins, member count, file size, API freshness, schema and notification with skill. Keep failed runs in service denominators and assess whether severe events are disproportionately missing.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: New computation improves calibration but misses major-event deadlines; reassess the service claim. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 9 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
Decommission-assurance matrix
| Layer | Evidence | Residual duty |
|---|---|---|
| Traffic | Zero authorised calls in a defined window | Resolve scanners and exceptions |
| Code and jobs | Disabled schedules and dependency graph | Retain reconstruction |
| Data and caches | No mutable legacy publication path | Preserve provenance |
| Users and contracts | Migration and expiry notice | Support statutory access |
| Operations | Rollback formally closed | Keep replay artefacts |
Zero forecast weight does not prove the old system is gone.
CHAPTER 10 OF 13 · Replay incidents
10. Preserve an incident evidence bundle
For fallback, stale output, wrong lineage or drift alarm, preserve issue-time inputs and observations, code, configuration, environment, logs, traces, products, cache state, notification and decision chronology. Hash controlled artefacts before rotation and record which users received what product and correction.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Discover the incident after log rotation; bound reconstruction with durable lineage and note unknowns. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 10 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 11 OF 13 · Replay incidents
11. Replay production and test controls
Replay original versions and issue-time data before repair, then change one control at a time: lineage gate, cache key, timeout, fallback notice or dependency refusal. Do not add later observations and call hindsight a fix. When external services prevent exact replay, test stored responses and containment explicitly.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A remote feed cannot be reproduced; replay its stored response and test timeout plus labelling. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 11 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 12 OF 13 · Assure decommissioning
12. Build a dependency-closure case
Prove each consumer migrated, expired or has a bounded exception; jobs and publication paths are disabled; credentials revoked; caches invalidated; and legacy data cannot masquerade as current. Require scientific, operations, security, records and communication sign-off and preserve residual risks with owners and expiry.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A statutory archive needs old files but not live production; retain read-only provenance without callable generation. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 12 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
CHAPTER 13 OF 13 · Assure decommissioning
13. Prove shutdown while preserving reconstruction
Probe endpoints, paths, schedules and credentials for explicit refusal, monitor attempts through an assurance window and alert on any successful legacy output. Preserve source or image, configuration, dependencies, training references, schema, lineage and representative outputs without leaving production exposed. Formally close rollback and keep monitoring the new branch.
Build the evidence record around this decision. Name the unit being observed, the version or policy in force, the time boundary, the numerator and denominator, the missing cases and the person authorised to act. Keep the raw observation separate from the interpretation. A dashboard value without those boundaries can look precise while answering a different question.
Turn the finding into a controlled action. Predefine the trigger, permitted response, owner, expiry, rollback and verification window. Retain one case that passes and one that fails, because a system that records only success cannot reveal whether the control detects the problem it was designed to contain.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Move the archive after shutdown; verify checksum, access and reconstruction without reactivation. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 13 check · what should survive an audit
- State the decision and measurement unit in one sentence.
- Preserve version, timing, denominator and missing-case evidence.
- Connect the finding to a bounded action and a later verification.
Verification lab: work the decision twice
At 02:05 the current endpoint returns the right issue time but legacy lineage. Reconstruct input delay, mode, cache and client evidence, decide whether to block or correct and preserve the visible chronology.
Overall CRPS is stable while flood-threshold reliability worsens at long leads in three basins. Build a regime review with sample-size, observation checks and action thresholds.
Legacy API traffic is zero but a nightly partner file appears in a mirror. Extend the dependency graph, identify owner, stage shutdown, probe refusal and update the assurance case.
Limits and misconceptions
A green pipeline is not proof of the right branch
Lineage and end-to-end probes establish origin.
Zero endpoint calls do not prove zero dependency
Files, mirrors, queues and storage can preserve use.
Stable average skill does not rule out drift
Lead, basin, regime and observation quality can diverge.
Decommissioning is not deletion
Consumer closure, exceptions, records and reconstruction matter.
Questions readers ask
How is silent fallback detected?
- Mandatory mode lineage, blocking of unlabelled products and branch-distinguishing probes.
How long should surveillance last?
- As long as the claim needs; routing is quick, seasonal calibration and extremes take longer.
What should be watched for drift?
- Calibration, sharpness, inputs, observations, latency, completeness and decisions.
Why replay?
- To reproduce the path, test the control and separate issue-time behaviour from hindsight.
When can legacy close?
- After dependencies and exceptions resolve, refusal is tested and reconstruction preserved.
Must old production remain runnable?
- Usually a controlled reconstruction package is safer than an exposed service.
Practise, verify and continue
Build post-cutover controls for one basin: immutable lineage, a synthetic probe, explicit fallback, endpoint and mirror inventory, calibration and service dashboards, an incident bundle and closure matrix.
Move to a multi-agency flood service with partner caches and files; rebuild ownership, access evidence, notice, exceptions and archive duties.
Practical evidence check
Before implementation, write a one-page evidence map with the decision, unit of analysis, version or time boundary, inputs, outcomes, uncertainty, authority and stop rule. Attach one worked case that passes and one that fails. A reviewer should be able to reproduce the reasoning without relying on the memory of the person who built the system.
Run one changed-context rehearsal before approval. Alter a mechanism rather than a cosmetic detail, then identify which calculation, safeguard or governance gate must change. Record any conclusion that does not transfer. This short rehearsal is often the fastest way to expose a hidden assumption before it becomes an operational dependency.
Related eduKate reading
Postprocessor blending and transition decommissioning
Postprocessor transfer and concept drift
Rolling verification and repeated alerts
Conditional verification
Authoritative sources and scope
ECMWF S2S model-update handling
ECMWF reforecasts and model-version matching
Operational runoff ensemble postprocessing
Hydrological ensemble postprocessing evaluation
The sources support model-version handling, reforecast compatibility and hydrological postprocessing evaluation. The surveillance architecture, basins, thresholds and incident are fictional. Agencies must apply their warning, security, records and observation procedures.