Choosing one model can hide uncertainty about the model itself
Bayesian model averaging carries uncertainty across a defined set of candidate models by weighting model-specific posterior quantities with posterior model probabilities.
Analysts often compare several plausible models and then report the winner as if no alternatives existed. That discards uncertainty about variables, mechanisms, priors and functional form. It can make estimates too confident and predictions fragile.
Bayesian model averaging treats the model index as uncertain. Prior model probabilities and model-specific marginal likelihoods produce posterior model probabilities; posterior estimates and predictions are then mixed across models. The method is principled within the candidate set, but it depends strongly on that set, its priors and whether any candidate is adequate.
Chapter contents
- 1. Define the quantity of interest before the models
- 2. Construct a defensible candidate-model space
- 3. Assign prior model probabilities transparently
- 4. Use marginal likelihood to update model odds
- 5. Interpret posterior inclusion and model probabilities carefully
- 6. Worked example: average three forecasting models
- 7. Build predictive mixtures on the full distribution
- 8. Search large model spaces without pretending enumeration
- 9. Stress-test prior and model-space sensitivity
- 10. Compare BMA with stacking and selection honestly
- 11. Check calibration and residual structure after averaging
- 12. Use model averaging in decisions with explicit utility
- 13. Version model spaces and preserve dissent
Choose a reading route
- Route 1: Set the model space — question, candidates and priors
- Route 2: Update weights — marginal likelihood and posterior odds
- Route 3: Work a mixture — prediction and variance
- Route 4: Compute and criticise — search, sensitivity and diagnostics
- Route 5: Decide under model risk — shift, utility and governance
How to use this guide
Start with the chapters that match your immediate decision, then return to the worked example and audit chapters. Each chapter separates a model from the evidence needed to use it responsibly. A claim is not strengthened merely by being technical; it is strengthened when assumptions, observations, alternatives and limits can be inspected.
| Layer | Question | Minimum record |
|---|---|---|
| Concept | What exactly is being represented? | Definitions and boundary |
| Method | How will the reasoning be carried out? | Steps, inputs and assumptions |
| Transfer | Would it work in a changed setting? | A deliberately altered case |
| Audit | What could prove it wrong? | Evidence, limits and revision trigger |
Chapter 1. Define the quantity of interest before the models
The idea
Model averaging is meaningful only when candidate models address the same target and predictions are placed on compatible scales.
A disciplined method
Write the estimand or predictive target, decision horizon, data available at prediction time and transformations common to every candidate.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which quantity is being averaged and for what decision. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Several demand models all predict next-month units, while one predicts revenue and must be transformed or excluded.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Change the target from average demand to probability of a shortage and rebuild model outputs.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Compatible labels can hide different populations, conditioning sets or intervention meanings.
Evidence to retain: target definition, units, horizon, conditioning information and decision use
Common misconception: Any models fitted to the same dataset can be averaged directly.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 2. Construct a defensible candidate-model space
The idea
BMA conditions on a set of models. The set may vary covariates, mechanisms or distributions, but it should represent genuine alternatives rather than a convenient software menu.
A disciplined method
List scientific or operational hypotheses, map them to models, include a credible baseline, avoid duplicates and document important omitted families.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether the candidate set covers the major plausible structures. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A regression space varies inclusion of demand, price and season variables plus a no-change baseline.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Add a nonlinear family and observe how posterior mass and predictions redistribute.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: If all candidates share one misspecification, BMA can be confidently wrong.
Evidence to retain: model catalogue, rationale, equivalences, exclusions and baseline
Common misconception: A large candidate set automatically captures model uncertainty.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 3. Assign prior model probabilities transparently
The idea
Prior model probabilities encode belief or structural penalties before data. Uniform weights over models are not neutral when model counts differ greatly across families.
A disciplined method
Choose priors at the family or model level, inspect implied inclusion probabilities and complexity penalties, and run alternatives that reflect reasonable disagreement.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is how prior mass is distributed across duplicated or nested possibilities. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
One nonlinear family has 100 variants and a linear family has two; equal model priors give nonlinear structure nearly all family-level mass.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Collapse near-duplicate variants and compare family-weighted and model-uniform priors.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Prior sensitivity can dominate with weak data, while data reuse can invalidate apparently objective priors.
Evidence to retain: prior rule, implied family mass, inclusion probabilities, justification and alternatives
Common misconception: Equal probability for each model expresses complete ignorance fairly.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 4. Use marginal likelihood to update model odds
The idea
Posterior model probability is proportional to prior model probability times model evidence, the likelihood integrated over model parameters under their priors. This rewards fit while accounting for prior volume.
A disciplined method
Compute marginal likelihood with a validated method, retain normalising constants, compare posterior odds and inspect sensitivity to parameter priors.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which integrated predictive support changes the model weight. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A flexible model fits best at its optimum but receives lower evidence because much of its prior parameter space predicts poorly.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Narrow a parameter prior using justified external knowledge and show the weight change.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Marginal likelihood can be highly prior-sensitive and numerically difficult.
Evidence to retain: model priors, parameter priors, evidence method, numerical error and posterior odds
Common misconception: Posterior model probability is simply normalised maximum likelihood.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 5. Interpret posterior inclusion and model probabilities carefully
The idea
Posterior model probabilities rank candidates only within the specified space. Inclusion probabilities sum weights for models containing a feature but do not automatically establish causal importance.
A disciplined method
Report candidate set, priors and denominator; distinguish predictive inclusion from causal claims and compare correlated features jointly.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether probability refers to model identity, feature inclusion or a real-world hypothesis. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Two correlated predictors divide inclusion mass, while the family containing either is strongly supported.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Replace one predictor with a measurement-equivalent proxy and observe inclusion instability.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Model probabilities are not calibrated truth probabilities when the true process lies outside the set.
Evidence to retain: posterior weights, inclusion definitions, correlations, alternatives and sensitivity
Common misconception: A variable with 90 percent inclusion probability causes the outcome.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 6. Worked example: average three forecasting models
The idea
A small example exposes posterior weights, model-specific predictions and the mixture variance without hiding arithmetic.
A disciplined method
Assign prior weights to baseline, trend and seasonal models; update to posterior weights from stated evidences; mix their predictive means and variances.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is how between-model disagreement contributes to uncertainty. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Weights 0.2, 0.5 and 0.3 combine predictions 100, 110 and 125; total uncertainty includes within-model variance and spread among means.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Change the next period to a holiday and alter only the seasonal model prediction, then recompute.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: The fictional numbers demonstrate calculation, not evidence that three models are adequate.
Evidence to retain: priors, evidences, posterior weights, component predictions, mixture calculation and check
Common misconception: The model-averaged variance is the weighted average of component variances only.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
| Component | Contribution | Audit question |
|---|---|---|
| Within-model variance | Weighted component uncertainty | Are likelihoods calibrated? |
| Between-model variance | Disagreement in component means | Is the model set meaningful? |
| Posterior weight | Prior times evidence | How prior-sensitive is it? |
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 7. Build predictive mixtures on the full distribution
The idea
For prediction, BMA mixes model-specific posterior predictive distributions, not just point estimates. Mixtures can be skewed or multimodal even when each component is simple.
A disciplined method
Draw or integrate from each predictive distribution according to posterior model weights, compute intervals from the mixture and retain component contributions.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether a single mean hides decision-relevant modes. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Two models predict either stable demand or a sharp surge; the mixture mean describes neither mode well.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Increase the cost of the surge outcome and compare mean-based and distribution-based decisions.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Mixture intervals remain conditional on the candidate set and priors.
Evidence to retain: component distributions, weights, samples or quadrature, mixture summaries and tail checks
Common misconception: Averaging predictions and then adding one standard error reproduces the predictive mixture.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 8. Search large model spaces without pretending enumeration
The idea
Variable-selection spaces grow exponentially. Screening, Markov chain Monte Carlo model composition and other search methods approximate posterior mass and can miss isolated high-weight models.
A disciplined method
Describe the search algorithm, initialise multiple chains, monitor visited mass and transitions, compare small enumerable cases and retain search uncertainty.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether unvisited models could materially change the mixture. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A model chain remains in one correlated-variable cluster; another start discovers an alternative cluster with similar evidence.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Tighten screening and measure lost posterior mass on a tractable subset.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Convergence in parameter space does not prove adequate exploration of model space.
Evidence to retain: seeds, starts, visited models, transition diagnostics, retained mass and exact benchmark
Common misconception: The highest visited model proves the important model space was explored.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 9. Stress-test prior and model-space sensitivity
The idea
BMA can look stable while weights change sharply under plausible prior, baseline or candidate-set choices. Sensitivity is part of the result.
A disciplined method
Vary model priors, parameter priors, candidate families and likelihood assumptions; compare weights, predictions, decisions and tail risk.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which conclusion survives reasonable modelling choices. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
The averaged mean changes little, but shortage probability doubles when a heavy-tailed candidate is added.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Remove a dominant near-duplicate family and reallocate prior mass explicitly.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: There is no universal sensitivity range; choices must reflect the domain.
Evidence to retain: scenario grid, posterior weights, predictive metrics, decision changes and unresolved disagreements
Common misconception: If the averaged point estimate is stable, model risk is negligible.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 10. Compare BMA with stacking and selection honestly
The idea
BMA is coherent when one of the candidate models is treated as the data-generating model. In an open-world setting, predictive stacking may weight models by out-of-sample predictive utility instead.
A disciplined method
State the modelling regime, compare BMA, single-model selection and cross-validated stacking with proper scores, and avoid calling distinct weights the same probability.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether weights are posterior beliefs or predictive optimisation coefficients. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
BMA concentrates on one misspecified model while stacking blends complementary predictions on held-out data.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Add a well-specified candidate and observe whether methods converge.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Cross-validation has sampling noise, and stacking does not transform models into true explanations.
Evidence to retain: method definitions, validation folds, scores, weights, uncertainty and decision performance
Common misconception: Any weighted ensemble is Bayesian model averaging.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 11. Check calibration and residual structure after averaging
The idea
Averaging cannot rescue shared misspecification. Posterior predictive checks, held-out calibration and subgroup diagnostics remain necessary.
A disciplined method
Evaluate predictive log score, interval coverage, residual patterns and performance by time, region or relevant group; compare component and mixture failures.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether the mixture corrects or conceals component errors. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
All candidate models underpredict extremes, so their weighted mixture does too despite good central fit.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Move to a period with structural change and monitor weight and calibration drift.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Small samples limit comparison, and aggregate metrics can hide local harm.
Evidence to retain: holdout design, predictive distributions, scores, coverage, residuals and drift triggers
Common misconception: Good posterior weights guarantee calibrated predictions.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 12. Use model averaging in decisions with explicit utility
The idea
Posterior predictive mixtures can feed expected utility, reliability or value-of-information analysis. The decision still requires consequences, constraints and model-risk safeguards.
A disciplined method
Compute action outcomes under each model and the mixture, inspect worst credible models, compare reversible actions and state when additional information is worth collecting.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which action changes because model uncertainty is represented. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A capacity expansion has positive average value but unacceptable loss under a credible low-demand model, favouring a staged option.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Increase the cost of delay and compare staged and immediate choices.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Expected utility can hide distributional, legal or safety constraints.
Evidence to retain: action table, component outcomes, averaged utility, constraints, sensitivity and review
Common misconception: The action with highest BMA expected value is automatically robust.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 13. Version model spaces and preserve dissent
The idea
Candidate sets and priors evolve with evidence and purpose. Silent changes make posterior weights incomparable and can manufacture confidence.
A disciplined method
Version model definitions, priors, data and code; record additions and removals; preserve minority models and explain why weights changed.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether a weight shift came from data or analyst redesign. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A new mechanism enters the space and absorbs weight from earlier models; the report separates structural revision from new observations.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Transfer the analysis to another population and rebuild rather than relabel the original model set.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Versioning cannot remove judgement from choosing models and priors.
Evidence to retain: model-space version, prior version, data snapshot, code, dissent and change summary
Common misconception: New posterior probabilities can be compared directly even when the candidate set changed.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Field workbook
Create three simple candidate models for one prediction target. Assign explicit prior model probabilities, calculate posterior weights from supplied or simulated evidences, form a predictive mixture and test sensitivity to one added model and one prior change.
- Write the decision question in one sentence and name the decision owner.
- List observations separately from interpretations, with dates and provenance.
- Record at least two plausible alternatives before choosing a preferred account.
- Run the changed-context test and note which assumptions fail.
- State the limit, confidence and next review trigger in plain language.
| Layer | Question | Evidence |
|---|---|---|
| Target | Are model outputs compatible? | Estimand and horizon |
| Space | Which alternatives are included? | Catalogue and omissions |
| Weights | How were odds updated? | Priors and evidence |
| Decision | What changes under model risk? | Mixture and sensitivity |
Continue through the verified ecosystem
- Bayesian networks — structured probabilistic dependence
- Gaussian processes — posterior prediction within a function model
- Influence diagrams — beliefs connected to choices
- Multistage stochastic programming — decisions and recourse across scenarios
Frequently asked questions
What is Bayesian model averaging?
A posterior mixture of model-specific quantities weighted by posterior probabilities over a defined candidate-model set.
Where do posterior model probabilities come from?
From prior model probabilities multiplied by model-specific marginal likelihoods and normalised across candidates.
Does BMA find the true model?
Not necessarily. Its probabilities are conditional on the included models and priors.
Why average full predictive distributions?
Because model disagreement can create skewed, heavy-tailed or multimodal uncertainty that point averages hide.
What is posterior inclusion probability?
The total posterior probability of models that contain a specified feature or structure.
How is stacking different?
Stacking chooses predictive weights using out-of-sample performance rather than interpreting weights as posterior model probabilities.
Authoritative sources and further reading
- Hoeting et al., Bayesian Model Averaging: A Tutorial — foundational Statistical Science tutorial
- Adrian Raftery Bayesian model averaging research — primary author resource and bibliography
- Bayesian Model Averaging tutorial PDF — author-hosted paper copy
- Yao et al., Using Stacking to Average Bayesian Predictive Distributions — predictive alternative in the M-open setting
- Bayesian model averaging and model search strategies — model-space computation and search
This guide explains statistical judgement under a defined model space. It does not establish causal truth or remove the need for domain validation, calibration and accountable decision review.