Did you know? A model can predict that an item is likely to sell next week without giving you a useful reason. Its numerical score may be accurate in many cases, yet a shop owner will still ask, “What information was this prediction based on?”
That is the problem addressed by explainable AI, often shortened to XAI. It studies ways to understand a model’s behaviour, inspect which inputs contributed to predictions and communicate limitations clearly.
Quick answer: identify the question, select a suitable explanation method, examine input contributions or model behaviour, check the explanation against examples, and avoid treating an explanation of a prediction as proof of a real-world cause.
The 30-second explanation
- Feature: an input variable used by a model, such as a price, date or measured quantity.
- Local explanation: an account of one particular prediction.
- Global explanation: a view of patterns in the model’s behaviour across many examples.
- Feature importance: a measure of how much an input contributes to model behaviour under a chosen method.
- Counterfactual explanation: a hypothetical comparison of model outputs when certain inputs change.
- Causation: a real-world claim that changing one factor produces a change in another—something ordinary prediction explanations do not automatically establish.
Explanations can reveal surprising shortcuts, but they also simplify complex mathematics. The goal is to make the model more inspectable, not to pretend every decision has one perfectly knowable cause.
Start with a fictional library forecast
Imagine a school librarian wants to predict the number of books borrowed tomorrow. A model uses past borrowing totals, the day of the week and whether a school event is scheduled. It predicts 55 books.
A deliberately simplified explanation might say: start from a reference value of 50, add 15 for a busy weekday pattern and subtract 10 for the recorded event-related pattern. The arithmetic gives 50 + 15 − 10 = 55.
These contributions are made-up numbers for teaching; they are not a measured explanation from a real model. But they show what a feature-attribution display tries to communicate: a baseline and contributions associated with the result.
The important warning is that “minus 10 for the event” is not proof that cancelling the event would produce ten extra loans. There may be other connected circumstances, and the model may only have discovered a correlation.
Step 1: Ask what kind of model is making the decision
Some models are easier to inspect directly. A small linear regression has coefficients that show how the equation combines inputs, subject to its assumptions. A shallow decision tree can be read as a sequence of split conditions.
A large ensemble of trees or a deep neural network has more complex internal behaviour. Developers may use post-hoc explanation tools to examine how it responds to inputs. These tools provide useful approximations or attributions, not necessarily the same sort of clarity as a simple equation.
Explainability is only one design objective. Reliability, robustness, privacy, fairness and usability also matter. A simple model that is badly trained can still make unjustified predictions despite being easy to inspect.
Step 2: Distinguish a single prediction from the overall pattern
A local explanation addresses one input, such as “Why did the model forecast 55 books for Tuesday?” A global explanation asks what the model tends to rely on across many days.
These answers can differ. A day-of-week feature might be important across an entire dataset but have little influence on one particular forecast. Conversely, a rare holiday feature might dominate one prediction while affecting very few others.
When reviewing a dashboard, ask whether its feature ranking summarises the whole model or the individual result you are inspecting. Confusing the two can produce a misleading story.
Step 3: See what happens when a feature is shuffled
One model-agnostic technique is permutation feature importance. First, developers measure performance on a suitable held-out dataset. Then they shuffle one feature’s values while keeping other features unchanged and measure how much the model’s error changes.
Suppose our fictional library forecast has a mean absolute error of 4 books on a test set. After shuffling its day-of-week feature, the error rises to 7 books. The increase of 3 books suggests that feature carried useful predictive information under this test.
Those numbers are illustrative. Shuffling a feature can create unrealistic combinations, particularly when two features are correlated. A result can also vary depending on the chosen dataset and error measure. This test does not prove that weekdays cause a particular borrowing total.
Step 4: Understand SHAP without the jargon
SHAP, short for Shapley Additive exPlanations, is a family of methods inspired by a game-theory approach to allocating contributions among participants. Here, the participants are features, and the quantity being explained is a model output.
An explanation may distribute a prediction’s difference from a chosen reference value among features under the method’s assumptions. The result can help a person inspect why a particular output differs from that reference. But the assignments depend on background data, how missing features are represented and other modelling choices.
SHAP values are not automatically causal effects. When features correlate, the allocation of contribution can be sensitive to the chosen method. Explaining what a predictive model relied on is not the same as proving what a business or person should change in the real world.
A useful rule: An explanation can show a model’s mathematical behaviour. It cannot by itself certify that the model made a correct or fair decision.
Step 5: Ask a careful “what if?” question
A counterfactual explanation can explore how a model’s output would differ for a hypothetical input. For example, “If the forecast’s planned opening hours were two hours longer, what would this model predict?”
The answer may be helpful for exploring sensitivity, but it is still a statement about the model. It does not establish that changing opening hours will actually cause the predicted number of books to be borrowed.
If a hypothetical change would be impossible, unethical or inconsistent with the underlying data, it is not a useful real-world recommendation. Developers should distinguish feasible interventions from mathematical input changes.
Step 6: Use explanations to find suspicious shortcuts
Imagine an AI photo classifier predicts whether a picture shows a bicycle. A feature-analysis tool suggests that the background colour influences results far more than the wheels or frame. That may be a clue that the model learned a shortcut from its training photographs.
Explanations can direct investigators towards such issues, but they do not diagnose every cause by themselves. A good follow-up is to test the model with new examples that vary the background while keeping the bicycle visible.
If the prediction changes dramatically, the evaluation provides stronger evidence of sensitivity to that background. The model may need better training data or a change in the intended task.
Step 7: Explain the result to the person who needs it
A developer may need feature attributions and error analyses. A teacher may need to know whether a model treats different writing styles consistently. A customer may need a plain-language reason for a recommendation and a way to challenge a mistake.
One explanation format will not suit every audience. For a meaningful decision, communication should identify what the model predicts, the relevant evidence, uncertainty and how to obtain human review. A complicated chart without context may create an illusion of transparency.
In consequential applications, a useful explanation should not replace a review process or applicable rights. People must have practical ways to correct wrong records and question harmful outcomes.
Try it: calculate a model’s toy explanation
Draw a box labelled “Forecast: 55 books.” Under it write “Reference: 50,” “Weekday contribution: +15” and “Event-associated contribution: −10.” Verify that the contributions add up to the forecast.
Now answer two different questions. First: “Which component raised this model’s forecast?” The positive contribution indicates the weekday-related component. Second: “Would cancelling the school event definitely increase real borrowing by ten books?” No—the numerical explanation alone cannot establish that.
The distinction between predicting and causing is the most valuable lesson in this exercise.
Five mistakes worth avoiding
- Confusing local with global: one prediction’s explanation is not a universal rule.
- Confusing correlation with causation: predictive feature contributions are not automatic policy recommendations.
- Ignoring correlated inputs: contributions can depend on how related features are handled.
- Trusting an attractive plot: readable charts can still summarise a model that performs poorly.
- Forgetting the person affected: meaningful transparency includes uncertainty, evidence and an avenue for review.
Frequently asked questions
Does SHAP tell why something happened in real life?
Not automatically. It explains aspects of a model’s prediction under assumptions. Identifying real-world causes requires suitable causal evidence.
What is the difference between local and global explanations?
A local explanation concerns a single prediction. A global explanation describes behaviour across many examples.
Can I explain any neural network perfectly?
No. Methods can offer useful insights, but complex networks are difficult to interpret and explanation tools have assumptions and limitations.
Does feature importance mean a feature should be changed?
No. A feature may predict an outcome without causing it. Interventions need separate reasoning and evidence.
Can explanations help find model errors?
Yes. They can reveal unexpected dependencies or shortcuts that investigators can test on new examples.
Your one-minute takeaway
- Explainable AI helps people inspect how model predictions relate to input features.
- Local, global and hypothetical explanations answer different questions.
- Check accuracy, causal assumptions and consequences before turning an explanation into an action.
Reliable sources and connected articles
- SHAP Official Documentation — feature attribution and examples.
- SHAP: Be Careful When Interpreting Predictive Models for Causal Insights — why explanation is not proof of cause.
- Interpretable Machine Learning: Permutation Feature Importance — a worked introduction.
- NIST AI Risk Management Framework — a broader approach to trustworthy AI.
- How Neural Networks Turn Numbers Into Predictions — understand model calculations.
- How AI Detects Bias in Data and Decisions — go beyond individual explanations to group outcomes.
- How AI Forecasts Sales, Traffic and Demand — a related prediction task.