Choose your reading route
eduKateSG · Why Mathematics? · Explainable AI for Reliability: Time-Dependent SHAP Values, Feature Effects, Calibration and Causal Limits
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 8–14
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 15–21
Check assumptions
See where design and calibration matter.Route 4 · Chapters 22–27
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 28–33
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important in explainable AI for reliability? A model may predict a whole survival curve, while a technician needs to know why one asset’s risk changed and whether that explanation is stable. Time-dependent SHAP values, accumulated local effects and sensitivity checks translate model behaviour into structured comparisons. They do not reveal causes automatically. This guide develops the mathematical mechanism, worked examples, limits and responsible ways for students, parents and practitioners to learn from explanations.
Full chapter index
Chapters 1–7 · Understand the question
Chapters 8–14 · Reproduce the example
Chapters 15–21 · Check assumptions
Chapters 22–27 · Explore applications
Chapters 28–33 · Study and verify
A reliability model can output a risk score, hazard, cumulative hazard, survival probability or remaining-life quantile. An explanation is always about one chosen output. Saying that temperature contributed +0.08 is incomplete unless the scale and horizon are named. Begin with a sentence such as 'the explanation decomposes predicted 90-day failure probability relative to this background population.'
A local explanation describes one prediction. A global explanation summarises behaviour across many cases. Averaging local magnitudes can reveal common drivers, but it can hide opposite effects in different regimes. Keep the two questions separate: why did this asset receive its forecast, and how does the model behave across the fleet?
Shapley values come from cooperative game theory. Features act like players, and the prediction difference from a baseline is allocated by averaging each feature’s marginal contribution across possible coalitions. The construction aims to satisfy properties such as efficiency and symmetry for the chosen value function. The practical result still depends on how missing features and background data are defined.
For an additive explanation, baseline plus feature contributions equals the explained output: f(x)=E[f(X)]+Σφj. If baseline risk is 0.18 and contributions are +0.06, -0.02 and +0.03, the explained risk is 0.25. Check the sum. For nonlinear link scales, explain whether additivity holds on risk, log hazard or another transformed output.
| Term | Contribution |
|---|---|
| Baseline risk | 0.18 |
| Feature A | +0.06 |
| Feature B | -0.02 |
| Feature C | +0.03 |
| Predicted risk | 0.25 |
With two features A and B, A’s Shapley value averages its contribution when added first and when added after B: φA=0.5[v({A})-v(∅)]+0.5[v({A,B})-v({B})]. If the two differences are 0.10 and 0.04, φA=0.07. The averaging makes order explicit; it does not prove A causes a seven-point change in real failures.
| Order | Marginal contribution |
|---|---|
| A first | 0.10 |
| A after B | 0.04 |
| Average | 0.07 |
The baseline and coalition values depend on a background distribution. Using the whole fleet, one site or only healthy assets can produce different explanations for the same prediction. Choose a reference that matches the decision question and report it. If the model is deployed across fleets, compare references rather than presenting one explanation as context-free.
Interventional SHAP breaks feature dependence when simulating missing inputs; conditional SHAP respects an estimated conditional distribution. With correlated sensors, the choice can reallocate credit substantially. Neither is universally correct. Interventional explanations describe model behaviour under independent feature substitution, while conditional explanations depend on a model of relationships among inputs.
Temperature, pressure and load can move together. A model may use any one as a proxy, so importance is divided or shifted across correlated features. Grouped explanations can be more stable and scientifically meaningful. Repeat the analysis after removing or grouping proxies. Do not rank individual sensors as if the model had identified unique physical causes.
A survival model predicts a function over time, not one scalar. SurvSHAP(t) assigns contributions that vary with horizon, allowing a sensor to raise early risk and have little effect later. Plot contributions as curves with the baseline survival. Always label whether the explained quantity is survival, cumulative hazard or risk, because signs reverse between survival and failure probability.
Suppose high vibration contributes -0.04 to predicted survival at 30 days and -0.12 at 180 days. On the survival scale, negative means lower predicted survival, with a larger later effect. The same evidence would be positive on a failure-risk scale. The numerical sign belongs to the selected output, not to an intrinsic good-or-bad property of the sensor.
A time-dependent method evaluates model predictions for coalitions across a grid of times, then estimates Shapley contributions at each time. Computation grows with features and grid size, so approximations sample coalitions. Report the number of samples, background cases and time points. Repeated runs can reveal Monte Carlo noise in the explanation.
CHAPTER 12 OF 33 · Reproduce the example
12. From local curves to global importance
Aggregate absolute time-dependent contributions to rank features globally, but keep direction and time profiles available. A large absolute contribution means the model changes its prediction with that feature in the chosen context. It does not mean the measurement is reliable, actionable or causal. Show distributions rather than one mean bar.
Permutation importance measures the performance loss after disrupting a feature. It answers a different question from SHAP: how much the fitted model depends on a feature for predictive accuracy. Correlation can mask or exaggerate results, and unrealistic permutations can leave the data manifold. Use grouped or conditional variants when appropriate.
Partial dependence averages predictions while setting one feature to selected values. It can show nonlinear trends, but correlated inputs may create impossible combinations. Add a data-density rug and individual conditional expectation curves. Avoid extrapolating into sparse regions. For survival outputs, choose horizons or show a surface over time and feature value.
ALE divides the feature range into intervals, measures local prediction changes where data exist and accumulates them. It usually handles correlated inputs better than ordinary partial dependence because it avoids sweeping every case across the full range. Results still depend on binning and sample density. Bootstrap curves to assess stability.
LIME-style methods fit a simple model near one case using perturbed samples and locality weights. The explanation can change with kernel width, perturbation distribution and random seed. Check local fidelity: the surrogate should reproduce the black-box predictions in the neighbourhood it claims to explain. A simple equation with poor fidelity is not a faithful explanation.
A counterfactual asks what smallest input change would move a prediction to a target. In reliability, unconstrained optimisation may recommend physically impossible sensor combinations or changes that are consequences of damage. Add feasibility, actionability and temporal constraints. Call the result a model counterfactual unless a causal model supports an intervention claim.
Vary inputs across plausible distributions and measure how prediction variance changes. Sobol-style indices separate main and interaction effects under assumptions about the input distribution. This is useful for simulation models and reliability digital twins. Correlation and uncertain input ranges must be handled explicitly; otherwise the sensitivity statement answers an artificial experiment.
An explanation is estimated from a fitted model, finite background sample and sometimes random coalition sampling. Refit the entire model across bootstrap samples or seeds, then compare contribution curves. Stable average performance does not guarantee stable explanations. Report intervals or qualitative stability, especially when explanations will guide investigation.
A faithful explanation matches how the model responds, not whether the model is correct about the world. Test additivity, local fidelity and perturbation predictions. Then separately validate the base model’s calibration and transport. A perfectly faithful explanation of a biased model can confidently describe the wrong relationship.
Domain experts may prefer explanations that match known physics. Agreement increases usability but is not proof of correctness; experts and models can share the same mistaken assumption. Disagreement is a prompt to inspect data, leakage, proxies and mechanisms. Preserve both the model evidence and the expert review rather than forcing one to disappear.
A feature recorded after failure may dominate SHAP values and produce a clear story. The explanation reveals dependence but does not tell you the timestamp is invalid. Combine attribution with data-lineage checks. For every important feature, verify when it becomes available and whether it contains maintenance or outcome information unavailable at prediction time.
If predicted probabilities are not calibrated, explaining a precise 0.73 failure risk can create false confidence. Evaluate calibration at the same horizon and population before presenting probability contributions. When explaining a ranking score, label it as a score. The presentation should never be more precise than the validated output.
Compare explanation distributions across sites, equipment types and operating regimes when sample support permits. Differences may reflect genuine environments, proxy use or model instability. They are descriptive, not automatic evidence of discrimination or cause. Predefine important groups and include prediction performance, calibration and data quality beside attribution.
A long waterfall chart can overwhelm rather than clarify. Match the explanation to the user: technicians may need sensor trends and thresholds; engineers may need time-dependent contribution curves; managers may need limitations and action pathways. Test whether readers interpret signs, baselines and uncertainty correctly. Explainability is partly a communication design problem.
A feature can be measurable but not changeable. Asset age may strongly influence risk, while replacing age itself is impossible. Map each explanation to an appropriate response such as inspect, verify data, adjust load or seek engineering review. The response should follow approved procedures and causal evidence, not an attribution bar alone.
SHAP, permutation importance and ALE describe a predictive function under selected distributions. They do not estimate what would happen if a feature were intervened on. Confounding, mediation, feedback and measurement error remain. Use causal diagrams, target trials or experiments for intervention effects. Predictive explanations can help generate questions, not answer causal ones by themselves.
CHAPTER 28 OF 33 · Study and verify
28. Common misconception: positive SHAP means dangerous
The sign depends on the output and baseline. Positive contribution to survival can be protective on that scale; positive contribution to failure risk has the opposite meaning. A positive log-hazard contribution is different again. Put the output name and reference value on every plot. Avoid red-equals-bad colour schemes when the mathematical sign is context-dependent.
CHAPTER 29 OF 33 · Study and verify
29. Common misconception: explanation equals transparency
A model may produce attractive explanations while its training data, loss, calibration and governance remain opaque. Transparency includes provenance, cohort construction, missing-data handling, validation, limitations and monitoring. Explanation is one layer. Treating it as the whole system can make a complex pipeline look safer than it is.
Use a baseline of 0.18 and three contributions +0.06, -0.02 and +0.03 to verify additivity. Then compute the two-feature Shapley average from two marginal contributions. Repeat on survival and failure-risk scales to see the sign change. Finally, draw two correlated sensors and explain why credit allocation becomes ambiguous.
Begin with averages, functions and conditional probability, then add combinations, weighted averages and model validation. Ask the learner to state the baseline and output before interpreting a contribution. Encourage one diagram linking data, model, explanation and decision. These skills develop careful AI literacy and support many study paths without guaranteeing a particular job.
A time-dependent explanation can show a feature changing direction across the survival curve. A single global importance bar would hide that behaviour. This is why functional outputs need functional explanations—and why the time axis is part of the meaning, not decoration.
Read the SurvSHAP(t) paper at https://doi.org/10.1016/j.knosys.2022.110234. Connect explanation to https://edukatesg.com/2026/11/03/why-mathematics-random-survival-forests-reliability-bootstrap-trees-censoring-variable-importance-calibration/, https://edukatesg.com/2026/11/03/why-mathematics-deep-survival-learning-reliability-neural-hazards-censoring-regularisation-validation/ and https://edukatesg.com/2026/11/03/why-mathematics-causal-reliability-analysis-interventions-confounding-target-trials-counterfactual-failure-risk/. Mathematics makes explanation accountable by naming the output, baseline, distribution and limits.
Extended practice laboratory
Build explanations only after the output, horizon, reference population and calibration status are fixed. Verify Shapley additivity, compare methods that answer different questions and measure explanation stability across refits. Use timelines to detect leakage and physical constraints to reject impossible counterfactuals. Present local curves, global distributions and uncertainty rather than one authoritative bar chart. Close every analysis with the causal boundary: the explanation describes how a predictive function uses recorded features under selected background data. It can guide inspection and question formation, while intervention claims require a different design.
Students should name the baseline, scale and horizon before interpreting any contribution. Parents can ask whether the explanation describes prediction or cause. That single distinction supports responsible AI literacy without making the mathematics less approachable.
A second capstone is an explanation evidence pack for one high-risk case. Include the raw time series, feature lineage, model output, calibration status, reference population, local time-dependent contributions, global feature distribution and bootstrap stability. Place a causal diagram beside the predictive explanation, with a clear note that the two answer different questions. Ask a technician to identify the next diagnostic check, an engineer to identify unsupported operating regions and a reviewer to identify which claims exceed the evidence. Record their interpretations and revise the design until signs, baselines and horizons are understood. Then repeat the pack after a model update and compare not only predictions but also proxy use and contribution stability. This exercise shows why explainability is not a decorative chart. It is a chain from mathematics to communication and governance. Weighted averages define credit, background distributions define comparison, calibration defines whether probabilities are honest and causal design defines which actions can be justified. A responsible system keeps all four visible. Add an adversarial reading exercise: remove the baseline label, swap survival for failure risk and use an unsupported red-green colour scale. Ask readers what changed. Restoring the missing context demonstrates that many explanation errors come from presentation rather than algebra. Visual honesty is therefore a testable part of technical accuracy. Record interpretation accuracy before and after revision so clarity becomes measured evidence rather than an assumption made by the chart designer.
Practice case 1: Additivity
Setup: Use a baseline and three contributions.
Task: Reconstruct the prediction.
Checking target: Label scale, horizon and reference.
Choose a baseline predicted failure risk of 0.18 and contributions +0.06, -0.02 and +0.03. Verify that the explained prediction is 0.25. Repeat on survival scale, where signs reverse if risk is one minus survival. Then move the background population from the whole fleet to one site and note how the baseline and contributions change. Every chart should display output, horizon and reference population. Additivity is a bookkeeping property for the chosen explanation, not evidence of causal decomposition.
Practice case 2: Two-feature Shapley
Setup: Average two marginal contributions.
Task: Calculate both feature values.
Checking target: Define the value function.
For two features, compute phi_A=0.5[v(A)-v(empty)]+0.5[v(A,B)-v(B)]. With marginal contributions 0.10 and 0.04, phi_A=0.07. Calculate B's value and check that both sum to the prediction difference. Next create correlated features that substitute for each other and observe how conditional and interventional background choices reallocate credit. The exercise shows why a Shapley value is mathematically precise only after the value function and missing-feature distribution are defined.
Practice case 3: Time-dependent SHAP
Setup: Explain three survival horizons.
Task: Check additivity at every time.
Checking target: Keep the time axis visible.
Explain one survival curve at 30, 90 and 180 days. Plot baseline survival and time-dependent feature contributions. A vibration signal may contribute -0.04 to survival early and -0.12 later, while another feature changes direction. Check additivity at every time point and repeat coalition sampling to measure approximation noise. Summarising the curve with one bar can hide the timing that makes the explanation operationally useful. Store the grid and background sample with the result.
Practice case 4: Method comparison
Setup: Compare permutation, SHAP and ALE.
Task: Explain different rankings.
Checking target: Different tools answer different questions.
Compare permutation importance, SHAP and ALE for correlated temperature and load. Permutation asks how performance changes when a feature is disrupted; SHAP allocates a prediction difference; ALE accumulates local changes where data exist. They need not rank features identically because they answer different questions. Group correlated sensors and repeat the analysis. If conclusions change, report the ambiguity rather than choosing the method that supports the preferred story.
Practice case 5: Stability
Setup: Bootstrap model and explanation.
Task: Plot contribution distributions.
Checking target: Do not narrate unstable signs confidently.
Bootstrap the model at the asset level and regenerate one local explanation in every refit. Plot the distribution of contributions rather than one waterfall. A feature whose sign flips frequently should not drive a confident operational narrative. Separate instability from Monte Carlo noise by fixing or increasing coalition samples. Then compare prediction stability: the output can be stable while credit allocation is not. The deployment report should reflect whichever layer is least certain.
Practice case 6: Leakage
Setup: Add a post-event feature.
Task: Inspect its attribution and timeline.
Checking target: Explanation cannot replace provenance.
Create a leaked feature recorded after a failure and fit a simple model. It will likely receive dominant, stable attribution and appear easy to explain. Use the data timeline to reveal why the explanation is misleading. Remove the feature, rebuild validation and document the performance drop. This experiment demonstrates that explainability cannot replace provenance. For every important input, record availability time, physical meaning, missingness and whether maintenance or outcome recording can influence it.
Practice case 7: Counterfactual
Setup: Optimise a lower-risk input.
Task: Check feasibility and actionability.
Checking target: Model counterfactual is not causal proof.
Generate a counterfactual recommendation that reduces predicted risk by lowering vibration and raising coolant pressure. Check physical feasibility, sensor ranges, dependence and actionability. If the combination is impossible or vibration is a consequence of damage, reject the recommendation. A model counterfactual describes the fitted function; a causal maintenance recommendation needs intervention evidence and safety review. Add constraints and compare with an approved diagnostic action such as inspection rather than automatic control.
Practice case 8: Audience design
Setup: Build technician, engineer and governance views.
Task: Test interpretation.
Checking target: Link every view to model limits.
Design three explanation views: a technician view with sensor trends and next checks, an engineer view with time-dependent contributions and uncertainty, and a governance view with data, validation and causal limits. Test readers on the sign, baseline and horizon. Record misunderstandings and revise labels. The final package should link every explanation to model version, calibration status and escalation procedure. An explanation succeeds when it helps the right person ask a better question without overstating what the prediction proves.
Practice case 9: Background sensitivity
Setup: Use whole-fleet and site-specific references.
Task: Recompute one local explanation.
Checking target: Baseline choice is part of the claim.
The same asset may look high-risk relative to the whole fleet and ordinary relative to a harsh site. Compare baselines and contributions side by side, then state which decision each reference supports. A small convenience background can add sampling noise, so repeat samples. Do not merge explanations from incompatible references. Store the reference cohort date and filters with every exported chart.
Practice case 10: Conditional dependence
Setup: Create correlated pressure and load features.
Task: Compare interventional and conditional SHAP.
Checking target: Credit allocation depends on the missing-feature model.
Interventional substitution may generate pressure-load combinations absent from the fleet, while a conditional method estimates plausible combinations but depends on another model. Recompute explanations under both and group features when individual credit is unstable. Neither output is a unique truth about causation. Report the choice and explain it in plain language so users understand why the bars changed.
Practice case 11: ALE binning
Setup: Calculate local effects with coarse and fine bins.
Task: Inspect density and curve stability.
Checking target: Sparse bins should not drive confident stories.
Fine bins can reveal structure but become noisy; coarse bins smooth important thresholds. Plot observation counts and bootstrap curves. Compare with partial dependence only within supported ranges. Select binning without the final narrative in mind. For time-dependent outputs, repeat at meaningful horizons and avoid a visually dense surface that hides uncertainty. ALE explains the model near observed data, not physical response to intervention.
Practice case 12: Local fidelity
Setup: Fit a surrogate around one asset.
Task: Measure prediction error in the claimed neighbourhood.
Checking target: Simplicity without fidelity is misleading.
Vary locality width and perturbation distribution, then compare surrogate and black-box predictions for plausible nearby cases. A narrow neighbourhood can yield a stable-looking line supported by almost no data; a wide one may average different regimes. Report fidelity alongside coefficients. If no simple surrogate fits well, show that limitation rather than presenting a convenient equation as the model's logic.
Practice case 13: Causal diagram
Setup: Place load, vibration, maintenance and failure in time order.
Task: Identify confounder, mediator and consequence roles.
Checking target: Predictive attribution does not select an adjustment set.
Draw several credible arrows and mark when each variable is measured. A large vibration attribution may reflect damage already present, while maintenance may change both vibration and failure. Use the graph to formulate a target trial or experiment, not to retrofit causal meaning onto SHAP values. Compare the predictive explanation with the causal question in separate panels. The discipline prevents feature importance from becoming a promise that changing the feature will improve reliability.
Practice case 14: Communication test
Setup: Show the same plot to three audiences.
Task: Ask each reader to explain sign and baseline.
Checking target: A misunderstood explanation is not usable.
Record whether technicians, engineers and managers can state the output, horizon, reference and uncertainty. Common errors include reading survival contributions as failure risk, treating baseline as average physics and interpreting colour as causality. Revise labels and narrative, then retest. Human comprehension is an empirical property, just like calibration. Avoid adding visual complexity that has not been shown to help a decision.
Practice case 15: Explanation monitoring
Setup: Compare attribution distributions across model versions.
Task: Detect shifts in proxy use.
Checking target: Stable performance can hide changed reasoning.
Track global contribution distributions, feature groups and time profiles after retraining. A new model may preserve AUC while switching from physical sensors to site identifiers or missingness proxies. Investigate changes with data lineage and validation, not by assuming old explanations remain valid. Version thresholds and background samples. Monitoring explanations does not certify causality, but it can reveal an unexpected dependency before it becomes an operational failure.