Why Mathematics? | Simulation-Based Inference for Reliability: Summary Statistics, Approximate Bayesian Computation, Posteriors and Model Checks

Extended practice laboratory

This laboratory turns simulation-based inference for reliability into a sequence of checkable decisions. Students should keep a ledger of definitions, calculations, assumptions and later evidence. A parent or educator can ask what the model knew at the time and what would make the conclusion change. The goal is not to imitate advanced software; it is to understand the mathematical mechanism well enough to detect a wrong denominator, a leaked timestamp, an unsupported extrapolation or an overclaim.

A strong project on simulation-based inference for reliability includes a simple baseline, one worked example, a sensitivity check and an untouched evaluation. It reports limits with the same care as results. This is how school algebra, probability, graphs and functions become transferable reasoning tools.

For a capstone, build a small reproducible investigation of simulation-based inference for reliability. Freeze the question and data contract first. Preserve every intermediate value so another student can recompute the result. Compare at least two defensible methods, evaluate on later or held-out evidence, and include a failure case. End with a one-page decision record: what is supported, what remains uncertain, what action is allowed, and what new evidence would trigger review. The capstone succeeds when its reasoning survives a changed example, not when it produces the most impressive chart. A simulation-based-inference capstone should use a simulator with at least one case where an exact or high-accuracy reference posterior is available. Freeze the prior, parameter domain, summaries, scaling, distance and simulator version. Compare rejection ABC, sequential ABC and one learned density estimator under matched simulation budgets. Track acceptance, effective sample size, coverage, rank statistics and wall-clock cost. Deliberately use one insufficient summary and one misspecified simulator to show how a tight approximate posterior can still be wrong. Run posterior predictive checks on extreme and temporal features not used solely for fitting. Repeat across seeds, tolerances and priors. The final inference card must separate algorithm approximation, Monte Carlo error, neural estimation error and simulator discrepancy; name the supported parameter region; and state when the method abstains. No deployment decision should rest on a posterior whose calibration was checked only inside the same simulator that generated it.

Identifiability deeper audit

Use identifiability to practise sensitivity analysis. Parameters are identifiable when distinct values produce distinguishable data distributions for the observations available. Start from this example: Wear rate and shock rate can trade off to produce similar failure counts. Change one assumption while keeping the others fixed and predict the direction before calculating. A narrow learned posterior can hide simulator non-identifiability. Use profile simulations and pair plots. Report whether the decision is stable or depends heavily on the changed assumption. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Simulation-based calibration deeper audit

Build a small counterexample for simulation-based calibration. Generate parameters from the prior, simulate data, infer posteriors and check whether true parameters have uniform rank statistics. Across many synthetic cases, 90% credible regions should contain about 90% of generating values under the full procedure. Then create a nearby case where the shortcut fails. This matters because passing simulated calibration does not validate the simulator against reality. Separate algorithm calibration from model adequacy. Explain the failure in plain language and connect it to the exact mathematical step that changed. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Posterior predictive checks deeper audit

Treat posterior predictive checks as a model-checking problem. Draw parameters from the posterior, simulate replicated data and compare features not used solely to fit. Check extreme downtime, run lengths and failure clustering. Preserve inputs, intermediate values and output. A model can match chosen summaries while missing important structure. Predefine decision-relevant checks. Evaluate the same rule on later or held-out evidence and separate numerical correctness from real-world validity. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Model misspecification deeper audit

Connect the mathematics of model misspecification to a decision. If the simulator omits a mechanism, inference may distort available parameters to compensate. Unmodelled corrosion makes the estimated wear rate appear too large. Calculate the quantity, then translate it into an allowed action and an action that the evidence does not support. A tight posterior can be confidently wrong. Add discrepancy checks and alternative simulators. This distinction prevents an accurate calculation from becoming an overclaim. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Worked posterior interpretation deeper audit

Turn worked posterior interpretation into a transparent worksheet. Suppose accepted ABC draws place 80 of 100 effective samples above a wear-rate threshold. The approximate posterior probability is 0.80 under the chosen prior, summaries, distance and tolerance. List every quantity, unit and timestamp, then recompute the example after changing one input. It is not a frequentist failure rate or proof the threshold is true. Repeat under alternative summaries and priors. Finish by naming the output as description, prediction, diagnosis or causal evidence and justify that label. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Student activity deeper audit

Begin with the operational question behind student activity. Infer the bias of a simulated coin using only the number of heads, then repeat using an unnecessarily poor summary. Now examine the illustrative case: Observe how summary choice and tolerance change accepted probabilities. Draw the relevant timeline or graph before calculating. The important boundary is that a coin has a tractable likelihood, so abc is pedagogical rather than necessary. Compare with the exact binomial posterior. Compare a simple baseline and record what evidence would reverse the conclusion. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Parent and educator guidance deeper audit

For this case, audit the mechanism rather than memorising vocabulary. SBI connects probability, simulation, inverse problems and scientific checking. Ask what the simulator can generate, what was compared and what evidence was discarded. Ask which denominator, risk set, neighbourhood or candidate pool makes the result meaningful. Realistic graphics do not make a simulator complete. Begin with a coin or queue whose exact answer is known. A second student should be able to reproduce the answer from the saved ledger alone. Record the result in plain language for a student and technical language for a reviewer. The two versions should agree on the quantity and limitation even though their vocabulary differs.

Practice case 1: Forward simulation and inverse inference

Setup: Use a small example for forward simulation and inverse inference.

Task: Check identifiability before interpreting parameters.

Checking target: Different parameter combinations can generate similar outputs.

Turn forward simulation and inverse inference into a transparent worksheet. A forward model generates data from parameters; inference works backward from observed data to plausible parameters. A degradation simulator maps wear rate and shock frequency to inspection histories. List every quantity, unit and timestamp, then recompute the example after changing one input. Different parameter combinations can generate similar outputs. Check identifiability before interpreting parameters. Finish by naming the output as description, prediction, diagnosis or causal evidence and justify that label.

Practice case 2: Why the likelihood is hard

Setup: Use a small example for why the likelihood is hard.

Task: List every simulator component and random source.

Checking target: Likelihood-free does not mean assumptions-free.

Begin with the operational question behind why the likelihood is hard. Complex event simulation may be easy to run but hard to turn into a tractable probability density for the observed record. Now examine the illustrative case: A repairable-system simulator includes queues, inspections, shocks and policy feedback. Draw the relevant timeline or graph before calculating. The important boundary is that likelihood-free does not mean assumptions-free. List every simulator component and random source. Compare a simple baseline and record what evidence would reverse the conclusion.

Practice case 3: Bayesian target

Setup: Use a small example for bayesian target.

Task: Show prior, approximate likelihood mechanism and posterior.

Checking target: A prior can dominate when observations are weak.

For this case, audit the mechanism rather than memorising vocabulary. The posterior is proportional to prior times likelihood, even when the likelihood is approximated through simulations. Prior knowledge about crack-growth rate updates after observed inspection data. Ask which denominator, risk set, neighbourhood or candidate pool makes the result meaningful. A prior can dominate when observations are weak. Show prior, approximate likelihood mechanism and posterior. A second student should be able to reproduce the answer from the saved ledger alone.

Practice case 4: ABC rejection

Setup: Use a small example for abc rejection.

Task: Measure acceptance rate and effective sample size.

Checking target: Naive rejection becomes inefficient in high dimension.

Use abc rejection to practise sensitivity analysis. Sample parameters from the prior, simulate data and accept draws whose simulated summaries are close to observed summaries. Start from this example: Accept a wear-rate draw when simulated mean degradation and failure count are within tolerance. Change one assumption while keeping the others fixed and predict the direction before calculating. Naive rejection becomes inefficient in high dimension. Measure acceptance rate and effective sample size. Report whether the decision is stable or depends heavily on the changed assumption.

Practice case 5: Summary statistics

Setup: Use a small example for summary statistics.

Task: Justify summaries through mechanism and simulation.

Checking target: Insufficient summaries discard parameter information.

Build a small counterexample for summary statistics. Summaries compress raw observations into features used for comparison. Failure count, mean time, quantiles and autocorrelation summarise an event history. Then create a nearby case where the shortcut fails. This matters because insufficient summaries discard parameter information. Justify summaries through mechanism and simulation. Explain the failure in plain language and connect it to the exact mathematical step that changed.

Practice case 6: Distance function

Setup: Use a small example for distance function.

Task: Report scaling and sensitivity.

Checking target: Arbitrary weights change the approximate posterior.

Treat distance function as a model-checking problem. A distance measures discrepancy between simulated and observed summaries, often after scaling. Use weighted Euclidean distance so hours and counts do not compete by units alone. Preserve inputs, intermediate values and output. Arbitrary weights change the approximate posterior. Report scaling and sensitivity. Evaluate the same rule on later or held-out evidence and separate numerical correctness from real-world validity.

Practice case 7: Tolerance

Setup: Use a small example for tolerance.

Task: Plot posterior and acceptance rate over ε.

Checking target: A tiny tolerance with few samples creates Monte Carlo instability.

Connect the mathematics of tolerance to a decision. ABC accepts distances below ε; smaller ε gives closer matches but fewer accepted draws. At ε=1, 2% of draws are accepted; at ε=0.5, only 0.3% pass. Calculate the quantity, then translate it into an allowed action and an action that the evidence does not support. A tiny tolerance with few samples creates Monte Carlo instability. Plot posterior and acceptance rate over ε. This distinction prevents an accurate calculation from becoming an overclaim.

Practice case 8: A simple ABC example

Setup: Use a small example for a simple abc example.

Task: Report effective sample size and repeat seeds.

Checking target: Two hundred correlated or weighted draws may contain less than 200 independent equivalents.

Turn a simple abc example into a transparent worksheet. Draw 10,000 parameter values and accept 200 simulations within the tolerance. The acceptance rate is 200/10,000=2%. List every quantity, unit and timestamp, then recompute the example after changing one input. Two hundred correlated or weighted draws may contain less than 200 independent equivalents. Report effective sample size and repeat seeds. Finish by naming the output as description, prediction, diagnosis or causal evidence and justify that label.

Practice case 9: Sequential ABC

Setup: Use a small example for sequential abc.

Task: Track weights, ancestry and particle spread.

Checking target: Poor kernels or weight calculations can collapse diversity.

Begin with the operational question behind sequential abc. Move populations of particles through decreasing tolerances to spend simulation effort near plausible regions. Now examine the illustrative case: Start broadly, reweight accepted particles and perturb them at each stage. Draw the relevant timeline or graph before calculating. The important boundary is that poor kernels or weight calculations can collapse diversity. Track weights, ancestry and particle spread. Compare a simple baseline and record what evidence would reverse the conclusion.

Practice case 10: Synthetic likelihood

Setup: Use a small example for synthetic likelihood.

Task: Check simulated summary distributions.

Checking target: Normal summaries can be wrong in skewed or multimodal systems.

For this case, audit the mechanism rather than memorising vocabulary. Approximate the distribution of summary statistics, often as multivariate normal, from repeated simulations at each parameter. Estimate summary mean and covariance for a proposed degradation rate. Ask which denominator, risk set, neighbourhood or candidate pool makes the result meaningful. Normal summaries can be wrong in skewed or multimodal systems. Check simulated summary distributions. A second student should be able to reproduce the answer from the saved ledger alone.

Practice case 11: Neural posterior estimation

Setup: Use a small example for neural posterior estimation.

Task: Validate coverage on simulated test cases.

Checking target: Training distribution and architecture can create unsupported confidence.

Use neural posterior estimation to practise sensitivity analysis. Train a conditional density estimator on simulated parameter-data pairs to approximate the posterior. Start from this example: A flow model maps sensor summaries to a distribution over wear parameters. Change one assumption while keeping the others fixed and predict the direction before calculating. Training distribution and architecture can create unsupported confidence. Validate coverage on simulated test cases. Report whether the decision is stable or depends heavily on the changed assumption.

Practice case 12: Neural likelihood and ratio methods

Setup: Use a small example for neural likelihood and ratio methods.

Task: Use simulation-based calibration and benchmark tasks.

Checking target: Good classifier loss is not sufficient calibration evidence.

Build a small counterexample for neural likelihood and ratio methods. Other SBI approaches learn a likelihood surrogate or likelihood-to-evidence ratio instead of the posterior directly. A classifier distinguishes joint parameter-data pairs from independently shuffled pairs. Then create a nearby case where the shortcut fails. This matters because good classifier loss is not sufficient calibration evidence. Use simulation-based calibration and benchmark tasks. Explain the failure in plain language and connect it to the exact mathematical step that changed.

Practice case 13: Amortised inference

Setup: Use a small example for amortised inference.

Task: Detect shift and retrain or abstain.

Checking target: A new fleet or sensor protocol may lie outside the simulation distribution.

Treat amortised inference as a model-checking problem. A trained model can infer many future cases quickly after an expensive simulation phase. One network analyses thousands of similar assets. Preserve inputs, intermediate values and output. A new fleet or sensor protocol may lie outside the simulation distribution. Detect shift and retrain or abstain. Evaluate the same rule on later or held-out evidence and separate numerical correctness from real-world validity.

Practice case 14: Simulation budget

Setup: Use a small example for simulation budget.

Task: Budget pilots, validation and final checks separately.

Checking target: Parallel capacity does not reduce total evidence needs.

Connect the mathematics of simulation budget to a decision. Every parameter draw, replicate and training example consumes computation. One million simulations at 0.2 seconds each take about 55.6 compute-hours before parallel overhead. Calculate the quantity, then translate it into an allowed action and an action that the evidence does not support. Parallel capacity does not reduce total evidence needs. Budget pilots, validation and final checks separately. This distinction prevents an accurate calculation from becoming an overclaim.