Integrated Gradients explains how a differentiable model’s selected output changes as an input moves from a chosen baseline to the actual example. Instead of reading the gradient only at the final input, the method accumulates gradients along a path from reference to input and allocates the resulting output difference among input features. It is an attribution method: its values describe contribution to a particular model output relative to a particular baseline, not why the real world caused an event.
The practical workflow is to select a defensible baseline, specify the exact output to explain, integrate gradients along the interpolation path, check that feature attributions approximately sum to the model-output difference, and repeat under meaningful alternative baselines or changed examples. A colourful attribution map without a baseline definition and numerical convergence check can be visually persuasive yet misleading. Integrated Gradients has useful mathematical properties, but cannot prove causality, factual correctness or model safety.
This guide develops the method from first principles, solves an exact two-feature example, explains numerical convergence and saturation, tests a changed-context interaction and shows how educators and engineers can evaluate its limitations. All numerical models and case studies are invented for teaching; no neural network has been trained or executed on an external dataset in this article.
Contents
- The precise question: what changed relative to which reference?
- The mathematical construction in ordinary language
- Worked example: exact attributions of a two-feature function
- A baseline is a modelling assumption, not an empty decoration
- Why endpoint gradients can vanish while inputs still mattered
- Approximating the integral in an actual model
- Completeness residual: a test of arithmetic consistency
- Positive and negative attributions, and the meaning of units
- The original axioms: sensitivity and implementation invariance
- Interactions and the limits of a single feature score
- Images, language and tabular models need different baselines
- Integrated Gradients versus permutation importance and counterfactuals
- Failures that can make convincing attribution maps wrong
- Changed-context exercise: explain a study-risk score without causal claims
- What the method guarantees—and what it does not
- Frequently asked questions, practical routes and sources
The precise question: what changed relative to which reference?
Imagine a model gives a higher score for a photograph when a red object appears. The engineer asks which pixels contributed to that score. A raw gradient measured at the final photograph tells how the output would respond to a very small local change in each pixel, assuming differentiability. That sensitivity is related to, but not identical with, how much the whole difference from a reference photograph contributed to the final output.
Integrated Gradients begins by explicitly choosing a reference input x-prime, often called the baseline. It evaluates the model along a path between that reference and the actual input x. At every point it measures how sensitive the chosen output is to each feature, and then integrates those sensitivities. Each feature’s attribution is scaled by the feature’s own displacement from baseline to input.
This changes how the answer should be phrased. Rather than ‘the model thinks these are the causes’, a more defensible statement is ‘relative to this baseline and this selected model output, these input features account for the reported difference under Integrated Gradients’. The distinction matters because a different baseline can change what the method is asked to explain.
The output itself must be identified. A classification model may expose logits, post-softmax probabilities or a class-specific score. Attributions to the chosen logit are not automatically identical to attributions to its probability, because the probability depends on competing classes and a nonlinear normalisation. Explain which scalar function is being differentiated before interpreting signs or magnitudes.
The original study Sundararajan, Taly and Yan (ICML 2017), Axiomatic Attribution for Deep Networks, developed Integrated Gradients from desired attribution axioms including Sensitivity and Implementation Invariance. The mathematical properties clarify the method’s intended scope; the explanation still needs evidence about the user-facing decision.
The mathematical construction in ordinary language
Suppose the model’s selected scalar output is F(x), with input vector x and baseline x-prime. Define a straight path x(α) = x-prime + α(x−x-prime) for α between zero and one. At α=0 the path starts at baseline; at α=1 it reaches the real example. The method follows the model’s output along this path, rather than only asking what happens at the endpoint.
For feature i, Integrated Gradients assigns (x_i−x’_i) times the integral from zero to one of the partial derivative of F with respect to feature i evaluated at x(α). The derivative measures local sensitivity along the path. The integral accumulates those sensitivities. Multiplying by the feature displacement converts average sensitivity into an attribution measured in the output’s units.
If a feature has exactly the same input value as its baseline value, its displacement is zero and standard straight-path Integrated Gradients gives it zero direct attribution, even if the model is locally sensitive to that feature elsewhere. That is not a universal declaration that the feature is irrelevant; it is a statement about this particular contrast and path.
For a differentiable model, the fundamental theorem of calculus supports an important completeness relation: the sum of feature attributions equals F(x)−F(x-prime), subject to the assumptions needed for the path integral. In numerical applications, finite integration steps create an approximation, so a convergence residual is useful to check. If the residual is large, attributions may be unreliable as a numerical result even before assessing whether the baseline is meaningful.
The line integral is attractive because it is relatively straightforward to approximate with repeated gradient evaluations. No retraining or model-architecture modification is required in the original formulation. But the cost rises with the number of integration points, and choices about batching, interpolation and target output remain real engineering decisions.
Worked example: exact attributions of a two-feature function
Take a fictional differentiable scalar model F(x1,x2)=x1²+3×2. Choose baseline (0,0) and actual input (2,1). The baseline output is F(0,0)=0. The actual output is F(2,1)=2²+3×1=7. The difference we want to explain is 7 output units, not a probability of seven or a claim about a real neural system.
The straight path is x(α)=(2α,α). The partial derivative with respect to x1 is 2×1, which along this path becomes 4α. The displacement in x1 is 2. Therefore attribution IG1 = 2×∫ from 0 to 1 of 4α dα = 2×2 = 4. The partial derivative with respect to x2 is 3 everywhere; its displacement is 1, so IG2=1×∫ from 0 to 1 of 3 dα=3.
Completeness is exact here: IG1+IG2=4+3=7, equal to F(2,1)−F(0,0). The attributions describe how the model function changes between these two inputs: four output units are associated with the x1 component and three with the x2 component along the specified path. If a report says the first input contributed 57.14% of the total difference, that is 4/7≈57.14% under this particular contrast; the second contributes 3/7≈42.86%.
Now compare a raw endpoint gradient. At (2,1), gradient F is (4,3). Those numbers happen to share a superficial resemblance to the integrated attributions, but for x1 that is coincidence of this particular endpoint and function. The integrated x1 contribution is four after averaging a changing derivative and scaling by displacement; endpoint gradient four is a local derivative. They have different conceptual meanings and generally different numerical values.
All these calculations can be checked by differentiating the polynomial. They do not require a neural network run. The example makes the key idea transparent: an attribution should account for a chosen output difference, not simply report the steepness of the model at one point.
A baseline is a modelling assumption, not an empty decoration
Integrated Gradients cannot answer its standard contrast question without a baseline. For images, a black image may be computationally convenient but may be far outside the natural image distribution. For text models, a zero embedding or padding-token baseline may not correspond to meaningful language. For tabular records, an all-zero row may represent an impossible combination of age, income and categorical values.
The baseline influences both the output difference and the route through the model. If an engineer chooses a baseline representing ‘absence of a signal’, the result describes the difference from that representation. If the same engineer chooses a population reference or domain-specific neutral state, the allocation may change. Neither choice becomes objectively true merely because it produces a visually appealing explanation.
Revisit the toy function F(x1,x2)=x1²+3×2 with the same actual input (2,1), but change baseline to (1,0). The new baseline output is 1, so the target difference is 7−1=6. The contribution of x1 along this path is 2²−1²=3, while the contribution of x2 remains 3. The attributions now are (3,3), summing to six. The model and the actual input did not change; the reference question did.
This shows why two explanations of the same prediction can disagree without either containing an arithmetic error. One explains a change from (0,0), the other explains a change from (1,0). Report the baseline explicitly and, for consequential decisions, test several relevant baselines instead of cherry-picking the one that confirms a favoured story.
Baseline selection needs domain review. A hospital model might require a clinically meaningful reference distribution rather than an impossible zero patient. A school-performance model might need the actual prior knowledge of a learner rather than a fictitious pupil with every feature set to zero. These are examples of why attribution interpretation is inseparable from the context in which the model output is used.
Why endpoint gradients can vanish while inputs still mattered
A common reason to look beyond raw endpoint gradients is saturation. In a saturating nonlinear model, the output may change markedly while moving from baseline to actual input, even though the derivative at the endpoint has become very small. Measuring only that final derivative can give the impression that a feature does not matter, when the full path shows a substantial accumulated change.
Take a toy logistic function F(x)=1/(1+exp(−8x)), with baseline x=0 and actual x=1. The baseline output is 0.5; at the actual input F(1)≈0.999665. The model-output difference is approximately 0.499665. The derivative at x=1 is small because the sigmoid is almost flat near its upper bound, but the integrated gradient for the sole feature is the entire difference under completeness.
The single-feature attribution must be approximately 0.499665 when the integral is evaluated accurately. That does not require us to claim a specific number of gradient samples or invent an executed experiment. It follows mathematically from the fundamental theorem of calculus for this one-dimensional smooth function. The endpoint derivative and accumulated path effect are answering different questions.
Saturation is not the only reason for local gradients to mislead. Piecewise nonlinearities, interactions and numerical precision can complicate sensitivity analysis. Integrated Gradients offers a principled path-based account under its assumptions, but it still depends on the route between reference and input and on the selected scalar output.
Readers should avoid the opposite mistake: an attribution with a large integrated magnitude does not necessarily indicate a feature is causally responsible for the real-world outcome. It says the trained function changed along the specified input path. The model may rely on a spurious feature or on data correlations that do not survive intervention.
Approximating the integral in an actual model
For a practical differentiable network, the integral is often approximated with a finite collection of points along the path. One can evaluate gradients at evenly spaced interpolation values and average them with a numerical integration rule. The number of steps and quadrature method affect accuracy and computation cost, especially when the function changes rapidly along the path.
Captum’s Integrated Gradients documentation describes implementation options including approximation methods and a convergence delta. The PyTorch Captum tutorial presents how to use the method with model interpretation. The actual API behaviour should be checked against the installed library version rather than assumed from a generic illustration.
Suppose the toy first derivative along the earlier path is 4α. Using four midpoint positions α=0.125, 0.375, 0.625 and 0.875 gives derivative values 0.5, 1.5, 2.5 and 3.5. Their average is 2.0. Multiplying by displacement 2 gives x1 attribution four, exactly matching the analytic integral because the derivative is linear and midpoint integration is exact here.
For a more nonlinear function, four points may be insufficient. A developer can increase the step count and check whether attributions and the completeness residual stabilise. A decreasing residual is reassuring about numerical approximation, but it does not validate the semantic choice of baseline or the model’s real-world behaviour. Numerical correctness and explanatory usefulness are separate quality gates.
Performance must be considered. An image with many pixels does not mean each pixel necessarily needs a separate forward pass; implementations can batch interpolation points. Still, running many gradient evaluations can be expensive for large models. Explain the method, record its selected target and provide a practical budget for diagnostics rather than implying exact attribution is free.
Completeness residual: a test of arithmetic consistency
The completeness property gives a convenient internal test. Compute the sum of the feature attributions and compare it with the selected model-output difference F(x)−F(x-prime). Their difference, often reported as a convergence delta or approximation error, reveals whether the numerical integration reconstructs the expected total under the chosen function and inputs.
Imagine a fictional calculation where the model output changes by 2.40 units. An approximate attribution vector sums to 2.34 units. The absolute discrepancy is |2.40−2.34|=0.06 output units. Relative to the nonzero output difference, that is 0.06/2.40=0.025, or 2.5%. Whether this discrepancy is acceptable depends on the use case, numerical method and scale. The example is arithmetic, not a universal threshold.
Now assume a second integration setting yields attributions totalling 2.395 units. The residual is 0.005, about 0.21% of the 2.40-unit change. This is a closer numerical reconstruction. But if the baseline is clinically impossible or the model’s target is wrong, the smaller delta cannot rescue the meaning of the explanation.
When the output difference is very close to zero, dividing the residual by that tiny difference can create unstable or exaggerated relative-error percentages. In such cases report absolute error and appropriate scales, and do not pretend a one-number percentage describes explanation quality. Two large positive and negative attributions may cancel, leaving little total output difference despite important feature interactions.
An engineering audit should store the approximation method, step count, baseline, target and resulting delta. Recomputing on a different model version can change all of these. A one-off heatmap without numerical provenance is not a reproducible explanation.
Positive and negative attributions, and the meaning of units
An Integrated Gradients attribution can be positive or negative. A positive value means the feature’s movement from baseline contributes positively to the selected scalar model-output difference along the chosen path. A negative value contributes in the other direction. This sign is relative to the target output and baseline; it is not inherently a moral judgement that a feature is ‘good’ or ‘bad’.
Suppose an educational recommendation model has a selected output score that increases with practice completion and decreases with an absence indicator. An attribution map may assign positive value to one feature and negative to another. Before interpreting it, check whether the output is a logit, risk score or probability and whether a higher value actually corresponds to the stated recommendation. Labels such as ‘positive influence’ can be misleading if the score is a risk of failure.
Units also matter. If one input is measured in hours and another in dollars, their raw derivative magnitudes are not directly comparable without accounting for feature displacement and preprocessing. Integrated Gradients multiplies local sensitivity by the input difference, but the result still describes the scaled model function and its chosen encoding. If preprocessing changes, the explanation must trace back to meaningful original features.
Categorical variables and token embeddings introduce special difficulties. A one-hot category interpolated continuously can pass through inputs that do not correspond to any valid real category. An embedding baseline has mathematical convenience but does not automatically represent the semantic absence of a word. Report such interpretive limits whenever an explanation is likely to influence a human decision.
Where a model uses group interactions, a feature’s attribution is not the same as a standalone effect of toggling only that feature in the real world. Attribution divides a joint change according to the path and mathematical rule. A careful report should avoid claiming more than that rule establishes.
The original axioms: sensitivity and implementation invariance
The original Integrated Gradients authors argued that attribution methods should satisfy principled requirements. Sensitivity, in a relevant formal setting, asks that a feature receive nonzero attribution when a difference in that feature is responsible for a difference in output under the stated conditions. Implementation Invariance asks that functionally equivalent networks have the same attributions even if their internal implementations differ.
These properties are useful because neural networks can represent the same function through different parameterisations. An explanation that changes merely because one implementation uses a different internal computational arrangement, while the input-output function is identical, may reflect internals rather than the prediction being explained. Integrated Gradients was designed with such concerns in mind.
The axioms do not mean every feature attribution corresponds to a uniquely correct causal story. Mathematical requirements constrain the explanation method, but they do not establish that the model learned the right task or that its features capture real mechanisms. Different baselines and paths can still define different valid attribution questions.
Nor do all methods labelled ‘integrated gradients’ behave identically under every implementation mistake. Disconnected computational graphs, nondifferentiable operations, improper input scaling or inadvertently selecting a wrong target can break the intended calculation. Mathematical properties assume the method was applied to the function and path it claims to explain.
The primary paper Axiomatic Attribution for Deep Networks should be read for exact definitions and proofs. A digest can communicate intuition, but a production assurance argument needs the original formal conditions and a test showing that the library call corresponds to the intended scalar output.
Interactions and the limits of a single feature score
Many models use interacting features. A recommendation may depend on both prior attainment and practice availability, or an image classifier on a shape appearing within a context. If a model contains an interaction term, dividing the output change into individual feature attributions requires a rule. Straight-path Integrated Gradients supplies one such rule relative to the baseline.
Take F(x1,x2)=2×1+x1x2 with baseline (0,0) and actual input (2,3). The actual output is 2×2+2×3=10; the baseline output is zero. Along path (2α,3α), the partial derivative with respect to x1 is 2+x2=2+3α. Multiply its average over α from zero to one by x1 displacement 2: IG1=2×(2+1.5)=7.
The derivative with respect to x2 is x1=2α; multiply its average one by the x2 displacement 3: IG2=3×1=3. Completeness checks: 7+3=10. The interaction term x1x2 contributes partly to each feature under this straight path. No observation of a real dataset is required to perform this algebra.
Change the baseline to (1,1). Its output is 2×1+1×1=3, so the output difference to (2,3) is seven. Along the new path, x1=1+α and x2=1+2α. The x1 derivative is 2+x2=3+2α, averaged to four, with displacement one; IG1=4. The x2 derivative is x1=1+α, averaged to 1.5, with displacement two; IG2=3. The total is 4+3=7.
The example teaches a precise boundary: even when attributions sum correctly, allocations are conditional on a path and baseline. With interacting features, changing the reference shifts which portion of the interaction is counted as the difference being explained. It is a mathematical contrast, not an unambiguous causal ranking of human characteristics.
Images, language and tabular models need different baselines
Image explanations often display attribution heatmaps. A simple black-image baseline creates a clear numerical path but can introduce unrealistic intermediate colours or textures. A blurred image, background distribution or a set of baselines may produce different insights. Domain expertise should decide what absence or neutral reference means for the task, and reports should identify the actual choice.
Language models operate on discrete text and often continuous embeddings internally. Interpolating embeddings between an input token and a padding or reference embedding creates mathematical points that may not correspond to valid words. Integrated Gradients can still analyse the model function along that path, but the semantics of the interpolation require caution. Aggregating subword attributions into visible words adds another reporting choice.
Tabular models mix continuous, binary and categorical inputs. A straight line between a healthy patient and an all-zero baseline can pass through physiologically impossible combinations. A risk model explanation may then integrate through regions it never encountered during training. That does not necessarily invalidate the formal computation, but it limits how directly the numbers should be interpreted as real-world feature effects.
One practical diagnostic is to repeat the explanation under a few defensible baselines and ask which conclusions remain stable. If a feature changes sign depending on a plausible baseline, the correct answer is not to hide that disagreement. Explain the dependence and investigate whether the decision can rely on more direct evidence.
Use attribution to inspect a model, not to delegate factual responsibility to coloured cells. In education, a teacher should still review actual learner work; in medicine, a qualified decision maker still needs evidence and domain standards. Interpretability does not replace independent validation.
Integrated Gradients versus permutation importance and counterfactuals
Permutation feature importance asks how a model’s held-out evaluation score changes when a feature is permuted under a defined procedure. Integrated Gradients instead attributes a single selected output difference along a baseline path for a differentiable model. The first is typically evaluated over a dataset and performance measure; the second can explain one input. These methods answer different questions and have different failure modes.
Permutation methods can be misleading when features are highly correlated, because shuffled combinations may be unrealistic. Integrated Gradients can also traverse unrealistic combinations through its interpolation path. Neither automatically solves correlated-feature interpretation. The published permutation feature importance guide explains why the evaluation design and feature dependence matter.
Counterfactual explanations ask what feasible input change would alter a model’s output under specified constraints. A counterfactual may be useful for action, but must respect which inputs can actually be changed and whether changes are causally plausible. Integrated Gradients does not automatically answer how a person should act; an attribution to age or a correlated proxy is not a legitimate instruction to alter that property.
Gradient-based maps are often visually detailed, which can encourage overinterpretation. A sharp image does not necessarily correspond to a robust explanation. Check sensitivity to baseline choice, model version, target output and input perturbations. Where a concrete operational decision depends on explanations, consider corroborating them with different methods and independent task evidence.
A model reviewer should write down the question before choosing a tool: ‘Which pixels contributed to this logit relative to a black reference?’, ‘Which features matter to held-out accuracy?’ and ‘What feasible intervention could change this outcome?’ cannot all be answered by a single attribution score.
Failures that can make convincing attribution maps wrong
First, selecting the wrong output target yields perfectly computed numbers for the wrong question. A classifier may have several output indices; check the class mapping. Second, passing probabilities when the intended explanation concerns logits changes the function, sometimes making gradients saturate. Third, an unexamined baseline can dominate the explanation through irrelevant contrasts.
Fourth, using too few integration steps can produce a large completeness residual, especially across sharply changing activations. Fifth, forgetting preprocessing can make explanations refer to scaled or encoded features while the report labels them as original real-world units. Sixth, detaching tensors or breaking differentiability may silently remove a path through the computation graph.
Seventh, grouping token or pixel attributions can change signs and totals if aggregation is performed incorrectly. Eighth, selecting only examples where an attribution picture looks intuitive creates a biased presentation. A useful evaluation includes surprising failures, negative cases and representative samples rather than a showcase of selected images.
The repair sequence is straightforward: inspect the exact selected function, make a hand-checkable toy example, verify the baseline, compare the attribution sum with the output difference, increase integration resolution where necessary, and test plausible alternate baselines. Then examine whether the interpretation supports the actual decision. Computational convergence is necessary for trust in the numbers but not sufficient for trust in the story.
When a suspicious explanation persists, look for data leakage, spurious correlations, shortcut features, unstable preprocessing and output calibration. An attribution map may be revealing a genuine model flaw rather than a flaw in the explanation algorithm. The diagnostic task is to tell those possibilities apart with evidence.
Changed-context exercise: explain a study-risk score without causal claims
Imagine a fictional study-support model producing a risk score from two input features: number of uncompleted revision sessions and days until an assessment. A teacher asks which feature ’caused’ a student’s learning difficulty. That is already a risky framing: a predictive score may reflect correlations and incomplete data, while learning difficulties have many contextual causes.
For a transparent mathematical exercise, let F(x1,x2)=x1²+3×2 as before, with x1 representing a transformed missed-session indicator and x2 a transformed timing indicator. For an invented input (2,1) relative to baseline (0,0), attributions are four and three. The correct statement is ‘relative to this baseline, the selected model score differs by seven units, allocated four to the first transformed input and three to the second’. It is not ‘missing sessions caused four units of harm’.
Change one constraint. Suppose the model was trained with a standardised feature representation, so actual hours and days must be converted before inference. Attributing effects in the transformed coordinates without explaining the transformation can mislead parents. Any user-facing account must trace the features, units and baseline back to understandable facts, and should avoid converting attribution magnitude into a guarantee of future improvement.
Now suppose a student was absent for an approved medical reason. A model may assign high risk to missed sessions, yet the teacher’s response should consider context, accessible learning support and an independent assessment of understanding. Integrated Gradients can help inspect which recorded features affected the model score; it cannot determine a fair educational intervention by itself.
The changed-context lesson is the distinction between a numerical model explanation and an ethical real-world decision. Maintain task-specific human review and direct evidence of learning. A transparent score can assist diagnosis without becoming an unquestionable judgement about a child.
What the method guarantees—and what it does not
Integrated Gradients has a completeness property for suitable differentiable functions and well-defined paths, and the original paper motivates it through formal axioms. That is not a guarantee that the model is accurate, the baseline is meaningful, the inputs are causally relevant or the deployment is safe. Mathematical attribution quality and prediction validity are separate concerns.
Numerical approximations may introduce error, particularly for complex models and poorly resolved paths. Reporting the convergence delta helps quantify one aspect of that error. But a very small delta cannot detect every implementation choice, such as explaining an unintended output layer or interpolating through unrealistic feature combinations.
Interactions make attributions conditional on the chosen baseline and path. Correlated inputs make real-world interpretation harder still. A feature may appear important because the model exploits a proxy rather than a meaningful mechanism. A heatmap is not a substitute for studying the data collection process, representation and performance across relevant groups.
For consequential decisions, combine explanations with representative performance testing, subgroup analyses where appropriate, security and privacy controls, and domain expert review. If an explanation remains unstable under defensible baselines, communicate that uncertainty instead of choosing the most convenient picture.
Every model and quantity in the worked examples here is invented and analytically calculated. No claim is made to have trained, inspected or certified a real external model. The primary technical references provide a basis for implementation, while independent evidence must support the eventual application.
Frequently asked questions, practical routes and sources
What is Integrated Gradients? It is a method that attributes the difference in a selected differentiable model output between a baseline input and the actual input by integrating gradients along a connecting path.
Why choose a baseline? The question is comparative. Without a reference, ‘what contributed to this prediction?’ is underspecified. Different defensible baselines may answer different questions.
Why not just take the gradient at the input? Endpoint gradients measure local sensitivity and can be small after saturation, while Integrated Gradients accumulates sensitivity across the baseline-to-input path.
What is completeness? Under suitable conditions, the sum of feature attributions equals the selected model-output difference F(x)−F(baseline). Numerical approximation may leave a residual.
Does a high attribution mean a feature caused the real-world outcome? No. It means the feature received a contribution to a model-output contrast under the method and baseline. Causal claims require separate evidence and assumptions.
Can the method explain text? It can be applied to differentiable input representations such as embeddings, but the baseline, interpolation path and aggregation back to words require careful interpretation.
Does a zero attribution prove irrelevance? Not globally. It is conditional on the particular input, baseline, target and path. A feature can be important in other contexts or interactions.
What primary material should I read? The original Sundararajan, Taly and Yan (2017) paper, Captum’s Integrated Gradients documentation and the PyTorch model-understanding tutorial. These describe the method and its supported implementations.
For a student, reproduce the x1²+3×2 integral and change the baseline to see the answer shift. For an ML engineer, test numerical completeness and evaluate attribution stability before showing a plot to users. For a reviewer, ask which selected output, baseline and intended decision the explanation supports. Continue with How AI Works and education that transfers into life.