THE GOLD STANDARD OF HYDROLOGY · SPECIALIST STUDY
Ask whether simulated catchments reproduce the hydrological signatures that matter, not merely whether one fitted hydrograph looks close
Posterior predictive checking turns a fitted hydrological model back toward the observations: draw replicated rainfall-runoff behaviour from the posterior, calculate discrepancies that matter for floods, recessions, seasonality and dependence, and compare those replications with the real catchment record. The gold standard checks residual scale, whiteness and event structure; separates calibration reuse from genuine validation; searches for regime shifts and state-dependent failure; and treats systematic discrepancy as evidence to revise, limit or retire the model rather than inflate an uncertainty band until everything fits.
This guide belongs to the Hydrology subject guide, the Gold Standard of Education, and the How X Works learning library. Choose a reading route, work the examples, then use the transfer checks to test whether the method survives a changed context.
Choose a reading route
Build the foundations · chapters 1–2
Apply the core method · chapters 3–5
Test evidence and alternatives · chapters 6–7
Handle limits and disagreement · chapters 8–9
Transfer and govern · chapters 10
The whole problem at a glance
| Diagnostic target | Posterior replication | Failure pattern |
|---|---|---|
| Magnitude | Flow quantiles and peaks | Biased high-flow tail |
| Timing | Event lag and recession | Systematic phase error |
| Dependence | ACF, runs and event clusters | Residual memory remains |
| Regime | Season and state signatures | Good average, failed subgroup |
One global fit statistic cannot reveal all structurally important failure.
CHAPTER 1 OF 11 · Build the foundations
1. Define the posterior predictive experiment
Core idea. A posterior predictive distribution generates replicated observations by drawing parameters and latent quantities from the posterior and then drawing data through the observation model.
Method. State whether replications condition on observed forcing, resample forcing errors or simulate new events; identify calibration data reused; retain every transformation and comparison scale.
Worked example. One check simulates discharge conditional on recorded rainfall; another also samples uncertain catchment rainfall, so their interval widths answer different questions.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Future climates, land use and instruments may lie outside the fitted generative model.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 1 is complete only when the claimed result survives that review.
Misconception to avoid. Posterior prediction automatically means out-of-sample forecasting. It often reuses the observed covariates and fitted data.
Practical evidence check
- Draw the generative chain for calibration replication and for a genuinely new storm.
- Return to chapter contents
CHAPTER 2 OF 11 · Build the foundations
2. Choose discrepancies before inspecting model winners
Core idea. A discrepancy is a feature used to compare observed and replicated data; hydrologically meaningful choices include peaks, volumes, timing, recession, low-flow duration and dependence.
Method. Predeclare a compact set tied to the model purpose, include raw and standardised residual features, avoid redundant fishing and preserve exploratory checks as clearly labelled discovery.
Worked example. A flood model is checked on annual maxima, peak timing, event volume and threshold duration rather than only mean squared error.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. No finite discrepancy set can detect every inadequacy and several checks create multiplicity and interpretation challenges.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 2 is complete only when the claimed result survives that review.
Misconception to avoid. Use whichever statistic makes the model look informative. Outcome-selected checks weaken evidence.
Practical evidence check
- Have a second analyst name failure signatures before seeing the fitted simulations.
- Return to chapter contents
CHAPTER 3 OF 11 · Apply the core method
3. Generate replicated series with the full observation model
Core idea. Checking only deterministic simulator output ignores rainfall, rating-curve, residual and measurement components that define the predictive claim.
Method. Draw all represented uncertainties in their dependency order, retain latent and observed-scale replications, and compare at the scale where decisions and measurements occur.
Worked example. Replicated latent discharge matches flood dynamics, but stage-derived observed discharge shows wider tails because rating-curve uncertainty is included.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Flexible observation error can conceal process failure by absorbing residual structure.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 3 is complete only when the claimed result survives that review.
Misconception to avoid. Adding more noise makes a model pass. Wide replications can be uninformative and still miss signatures.
Practical evidence check
- Run checks on latent process, observed response and decision summary separately.
- Return to chapter contents
CHAPTER 4 OF 11 · Apply the core method
4. Check residual location, scale and support by hydrological state
Core idea. Residuals can change bias, variance, skewness and bounds with flow, season, wetness, temperature or sensor regime.
Method. Plot standardised residuals by fitted flow and state, compare quantiles and exceedances, test impossible support and calculate replicated conditional summaries.
Worked example. A model is unbiased overall but underpredicts wet-season peaks and overpredicts dry-season baseflow; global cancellation hides both errors.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Subgroup checks lose power and state variables may themselves be uncertain or post-treatment.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 4 is complete only when the claimed result survives that review.
Misconception to avoid. Zero mean residual proves calibration. Opposing regime biases can sum to zero.
Practical evidence check
- Partition by predeclared flow and season bands and recompute bias and coverage.
- Return to chapter contents
CHAPTER 5 OF 11 · Apply the core method
5. Test whiteness, runs and event dependence
Core idea. Autocorrelated residuals indicate that observations contribute less independent information than an independent-error likelihood assumes and often reveal missing memory or event structure.
Method. Inspect ACF, PACF, run lengths, storm-cluster summaries and cross-correlation with forcing; compare observed diagnostics with replicated residual diagnostics rather than applying one threshold mechanically.
Worked example. Twenty consecutive positive storm residuals create a long run and high lag-one correlation that independent posterior replications rarely reproduce.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Irregular time steps, seasonality, transformations and observation error complicate classical whiteness tests.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 5 is complete only when the claimed result survives that review.
Misconception to avoid. White residuals prove the process model is correct. They only remove some detectable structure.
Practical evidence check
- Shuffle residuals, preserve event blocks and simulate AR(1) residuals to see which diagnostics respond.
- Return to chapter contents
Residual-whiteness example
| Residual pattern | Likely implication | Next check |
|---|---|---|
| Positive storm runs | Missed storage or rainfall bias | Event residuals and forcing |
| Seasonal bias | Regime mismatch | Split sample and covariates |
| Variance rises with flow | Heteroscedastic error | Conditional scale model |
| Lag-1 correlation | Unmodelled memory | ACF/PACF and state dynamics |
Forcing and observation errors can also create structured residuals.
CHAPTER 6 OF 11 · Test evidence and alternatives
6. Keep event-level and point-level evidence separate
Core idea. Hourly residuals within one storm share forcing and state errors, so treating them as many independent successes or failures exaggerates evidence.
Method. Define events independently of model residuals, calculate volume, peak, lag and recession discrepancies per event, use block replication and show event-to-event variation.
Worked example. A model fits thousands of ordinary hours but misses six of eight major storm peaks; point-level accuracy hides event-level failure.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Event boundaries are contestable and compound or overlapping events resist simple separation.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 6 is complete only when the claimed result survives that review.
Misconception to avoid. More time steps always produce stronger validation. Dependence can leave the effective number of events small.
Practical evidence check
- Repeat the assessment after aggregating to events and compare conclusions.
- Return to chapter contents
CHAPTER 7 OF 11 · Test evidence and alternatives
7. Search for regime shifts and nonstationary failure
Core idea. Catchment response can shift with drought, vegetation, urbanisation, snow, groundwater state or data-system change, making stationary parameters appear inconsistent.
Method. Use chronological residual plots, split-sample tests, change-point or hidden-state models and covariate hypotheses; distinguish observed-data changes from process changes.
Worked example. Runoff efficiency drops after a multiyear drought; a stationary model shows persistent positive rainfall-normalised residual changes rather than random error.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Change-point methods can overfit short series and covariates may be correlated without causing the shift.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 7 is complete only when the claimed result survives that review.
Misconception to avoid. Every parameter drift proves climate change. Instrument, forcing and structural explanations must compete.
Practical evidence check
- Replay the diagnostic after masking the suspected breakpoint and after changing observation products.
- Return to chapter contents
CHAPTER 8 OF 11 · Handle limits and disagreement
8. Compare calibration checks with validation and transfer
Core idea. Posterior predictive checks on fitted data can reveal structure the model cannot reproduce, but they do not measure performance under new periods, basins or forcing regimes.
Method. Separate in-sample replication, temporal validation, spatial transfer and stress tests; avoid recalibrating on validation failures before recording them.
Worked example. A model reproduces calibration flow duration but fails an independent dry decade, narrowing its valid scope even though its posterior checks looked acceptable.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Nonstationarity can make any historical holdout an imperfect proxy for the future.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 8 is complete only when the claimed result survives that review.
Misconception to avoid. A passed posterior predictive check is validation. It is one component of model criticism.
Practical evidence check
- Compare checks on calibration, forward holdout and a deliberately contrasted climate period.
- Return to chapter contents
Predictive-check certificate
| Question | Evidence | Boundary |
|---|---|---|
| What was replicated? | Posterior generative definition | Conditional on model |
| What was compared? | Predeclared discrepancy set | Multiple checks |
| Where did it fail? | Regime and event attribution | Small extremes |
| What changed? | Versioned repair or scope limit | No outcome fitting |
Posterior predictive p-values are diagnostic summaries, not ordinary frequentist p-values.
CHAPTER 9 OF 11 · Handle limits and disagreement
9. Treat posterior predictive p-values as diagnostics
Core idea. Posterior predictive tail areas compare observed discrepancies with replications but reuse data through the posterior and are not uniformly distributed classical p-values.
Method. Report the observed statistic, replicated distribution, graphical context and practical consequence; avoid binary thresholds and examine several mechanistically distinct discrepancies.
Worked example. An observed peak-bias statistic lies near the edge of replications while total volume sits centrally, pointing to timing or routing failure rather than universal rejection.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Different discrepancy choices can give conflicting signals and double use of data can make checks conservative.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 9 is complete only when the claimed result survives that review.
Misconception to avoid. A posterior predictive p-value below 0.05 invalidates the model. Model checking is comparative and purpose-specific.
Practical evidence check
- Construct a model that passes a mean check but fails variance and autocorrelation checks.
- Return to chapter contents
CHAPTER 10 OF 11 · Transfer and govern
10. Respond to discrepancy with revision, mixture or scope limits
Core idea. Failure can arise from forcing, observation, residual, parameter or structural assumptions; simply widening uncertainty may hide rather than repair the mechanism.
Method. Trace the failed signature through competing components, test simpler and alternative structures, use regime mixtures only with evidence, and narrow operational scope when repair is not supported.
Worked example. Persistent recession error remains after rating-curve and rainfall checks, motivating a storage-process revision rather than a larger iid residual variance.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Several components can remain confounded and added flexibility can damage identifiability and transfer.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 10 is complete only when the claimed result survives that review.
Misconception to avoid. Every failed check needs a more complex model. Sometimes the honest response is a narrower claim.
Practical evidence check
- For one failure, write repair, alternative-model and withdrawal options before selecting.
- Return to chapter contents
CHAPTER 11 OF 11 · Transfer and govern
11. Version checks, decisions and retirement triggers
Core idea. Predictive checking is an ongoing evidence process because gauges, forcing products, catchment regimes, code and operational consequences change.
Method. Version data and discrepancy definitions, retain replicated summaries, monitor checks by regime, record accepted exceptions, replay corrections and predefine triggers for recalibration, scope reduction or retirement.
Worked example. A revised rating curve removes one apparent high-flow bias but residual dependence remains; the audit updates only the affected claims and keeps the structural warning open.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Exact replication may be limited by proprietary inputs, stochastic computation or sensitive infrastructure data.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 11 is complete only when the claimed result survives that review.
Misconception to avoid. Once validated, a hydrological model stays validated. Evidence and domains change.
Practical evidence check
- Ask an independent reviewer to reproduce one passed check, one failed check and the resulting governance decision.
- Return to chapter contents
Build the specialist decision record
For “The Gold Standard of Hydrology | Posterior Predictive Checks, Residual Whiteness, Event Dependence, Regime Shifts and Model Discrepancy”, begin with one bounded decision rather than an abstract claim. Name the affected people, data or institutions; the time window; the authority for acting; and the conclusion the method is allowed to support.
Preserve source material before interpretation. Record definitions, exclusions, transformations and the person responsible for each judgement. A reviewer should be able to separate an observed fact from an inferred result, a policy choice and a presentation shortcut.
Work one ordinary case in full. Show the input, every material rule or calculation, the output and the action that follows. Then write what the same evidence would mean under the simplest credible alternative method.
Keep an explicit unresolved state. Missing, protected, conflicting or unevaluated information must not become zero, false, consent or absence merely because a form expects one value.
Finish the record with a claim ledger. For every sentence that could drive action, identify its supporting source, inferential step, confidence or legal status, intended reader and expiry condition. This keeps descriptive evidence, formal consequence, professional judgement and institutional authority from collapsing into one impressive-looking conclusion.
Run the boundary-case audit
Select a case near the method’s domain edge: a missing input, a disputed definition, an endpoint, a sparse group, a resource limit or a legal deadline. Boundary cases reveal the contract that routine examples can conceal.
Change one assumption at a time and retain every output. Mark what stays stable, what reverses and what becomes undefined. If several assumptions move together, report that dependence instead of attributing the change to one convenient cause.
Search deliberately for a counterexample to the strongest claim. A method that works on the published illustration can still fail under a permitted input, an alternative grouping, a changed unit or an inaccessible evidence pathway.
Translate the counterexample into plain language: who would receive a different decision, what evidence would be hidden, and which safeguard or narrower claim repairs the problem.
Include an adversarial but plausible case. Let a missing group, ambiguous identifier, extreme weight, conflicting rule or urgent deadline interact with normal system incentives. The aim is not theatrical failure; it is to learn whether ordinary pressure makes the method conceal uncertainty, shift burden or reward a misleading shortcut.
Test independent reproduction
Give a reviewer the bounded inputs, public method contract and permitted access, but not the desired answer. Ask them to reproduce the result and identify every place where judgement, local knowledge or unavailable evidence was required.
Compare the reviewer’s intermediate states, not only the final number or label. Matching outputs can hide different denominators, inference paths, missing-data treatments or authorities that will diverge on the next case.
Where reproduction fails, classify the cause: ambiguous definition, undocumented transformation, incompatible implementation, inaccessible source, calculation error or legitimate alternative interpretation. Each class needs a different remedy.
Retain the disagreement and resolution. A corrected record should show what changed, why it changed and which prior outputs or decisions must be revisited.
Run the reproduction on a changed but structurally similar case after the first discrepancy is repaired. A method is not transferable merely because its authors can explain the original example. The new case tests whether the public contract carries the reasoning or whether hidden expertise is still doing essential work.
Design correction, challenge and remedy
Assign a visible route for correcting inputs, challenging assumptions and reviewing the application of the method. Technical accuracy without a usable remedy leaves the affected person carrying an error they cannot contest.
Separate urgent live-case repair from later system learning. A current decision may need pause, reassignment, fresh evidence or a human review before the organisation finishes its longer investigation.
Protect confidential and privileged material during review, but do not use confidentiality as a reason for unrecorded authority. Layer public explanation, protected oversight and individual reasons according to lawful access.
Verify that correction reaches cached results, derived records, dashboards and future decisions. Closing a complaint is not enough if the old conclusion continues to travel.
Measure remedy from the affected person’s position. A revised database row is useful, but it is incomplete when a hearing, service allocation, published claim or downstream model still relies on the old result. Record notification, restored opportunity, compensation or procedural repair where the governing framework provides it.
Set review and retirement triggers
Review after a change in definitions, population, law, source access, measurement scale, software, vocabulary or incentives. A stable formula can become invalid when the world or the meaning of its inputs changes.
Monitor leading signals as well as final outcomes: coverage gaps, timeouts, overrides, complaints, disagreement, missingness and boundary frequency. These often reveal failure before a headline metric moves.
Predefine thresholds for recalibration, suspension and retirement. Do not force a method to answer after its assumptions fail merely because replacing it is inconvenient.
At retirement, preserve the versioned contract, evidence, known limitations and migration decision. State which historic results remain interpretable and which require warning, restatement or withdrawal.
Schedule one prospective review before confidence fades or controversy forces it. Assign named owners for evidence, implementation and rights impact, and invite at least one reviewer outside the design team. The review should be able to narrow the claim, change the method or stop use without requiring proof of catastrophic failure.
Questions readers ask
What is a posterior predictive check?
- A comparison between observed hydrological features and data replicated from the fitted probabilistic model.
What does residual whiteness mean?
- Residuals show no remaining detectable serial structure under the chosen diagnostics and scale.
Do white residuals prove the model is correct?
- No. They only remove some detectable patterns and can coexist with other failures.
Why check events separately?
- Many time steps within one storm are dependent and can hide poor peak, volume or timing performance.
Is a posterior predictive p-value a classical p-value?
- No. It reuses data through the posterior and should be interpreted diagnostically.
What if a check fails?
- Investigate forcing, observation, error and structure; repair, compare alternatives or narrow the claim.
Practise, verify and continue
Choose one consequential case. Reproduce the complete method, work one ordinary and one boundary case, test the strongest rival explanation, and state the evidence that would reverse or suspend the result. Build an evidence pack containing original inputs, definitions, calculations or rule steps, alternatives rejected, uncertainty, reviewer comments and the final action. Give the pack to a reader who did not design the method. Ask that reader to reproduce the result, identify one missing assumption and construct a case that would make the method return unresolved. Revise the public explanation only after the evidence pack and the operational process agree.
Move the method into an education or public-service setting. Keep the reasoning structure while replacing domain authority, population, privacy, evidence and correction routes. Start with a low-stakes historical example, then compare the specialist method with a transparent baseline. Check whether the same people, records and decisions remain inside scope; whether the scale or vocabulary still means the same thing; and whether a person affected by error can understand and challenge it. Pilot under human review, record reversals and burden, and stop the transfer if the formal resemblance hides a different ethical or legal problem.
Related eduKate reading
Authoritative sources and scope
Thyer et al., Critical Evaluation of Predictive Uncertainty Using BATEA
Evin et al., Heteroscedasticity and Autocorrelation in Hydrological Calibration
Westra et al., Diagnosing Hydrological Model Nonstationarity
Kavetski and Fenicia, Flexible Conceptual Hydrological Modelling
Combining Split-Sample Testing and Hidden Markov Modelling
This article is hydrological model-checking education, not an operational forecast or site-specific engineering assessment. Examples are original teaching cases. Posterior predictive checks are conditional on the data, forcing, priors, likelihood and model structure. Protect sensitive infrastructure data, use qualified local expertise and lawful authority, distinguish exploration from predeclared validation, and withdraw or narrow claims when failure is material.