Backpropagation and gradient descent are the mechanisms that turn prediction errors into parameter updates. A neural network makes a prediction, a loss function measures how wrong it is, backpropagation computes how each parameter contributed to that loss, and an optimiser changes the parameters in a direction intended to reduce future loss.
This loop is the engine behind much of modern model training. Without it, a network would have architecture and random parameters but no practical way to learn from examples at scale.
This article explains backpropagation and gradient descent from first principles: derivatives, gradients, chain rule, forward pass, loss, backward pass, learning rate, batches, optimisers, vanishing gradients, exploding gradients, gradient clipping, automatic differentiation and convergence. Worked examples keep the mathematics connected to observable training behaviour.
Google’s current Neural Networks: Training using backpropagation describes backpropagation as the primary training algorithm that makes gradient descent feasible for multi-layer neural networks. Its learning-rate material also explains why updates that are too small train slowly and updates that are too large can fail to converge.
Previous: 023 — Self-Supervised Learning. This article shows how the learning signal actually changes the model.
The Hidden Transition: From “Wrong” to “Which Weight Should Move?”
Suppose a neural network predicts 0.30 when the target is 1.00. The statement “the prediction is wrong” is not enough to train the model. The network may contain millions or billions of parameters. Training needs to know which parameters should change and in which direction.
Backpropagation answers that credit-assignment problem by applying the chain rule through the computational graph. It computes the derivative of the loss with respect to each trainable parameter.
Gradient descent then uses those derivatives to update the parameters.
Start With One Parameter
Consider the simplest model: prediction y_hat = w × x. Let x = 2, target y = 6, and initial weight w = 1. The model predicts y_hat = 2.
Use squared error loss L = (y_hat − y)². The loss is (2 − 6)² = 16.
We want to know how the loss changes when w changes. Because y_hat = 2w, the loss is (2w − 6)². The derivative dL/dw = 2(2w − 6) × 2.
At w = 1, dL/dw = 2(−4) × 2 = −16. The negative gradient means increasing w will reduce the loss locally.
Gradient Descent Update
Choose learning rate η = 0.1. The update rule is w_new = w − η × gradient. With w = 1 and gradient −16, w_new = 1 − 0.1(−16) = 2.6.
Now prediction = 2.6 × 2 = 5.2. Loss = (5.2 − 6)² = 0.64. One update reduced the loss from 16 to 0.64.
The model has not “understood multiplication” in a human sense. It followed a numerical optimisation rule.
Why the Gradient Points Uphill
A derivative measures local slope. If dL/dw is positive, increasing w increases the loss, so gradient descent moves w downward. If the derivative is negative, increasing w lowers loss, so subtracting the negative gradient moves w upward.
The gradient therefore points toward increasing loss. Gradient descent moves in the opposite direction.
Many Parameters Create a Gradient Vector
A real model has many parameters. The gradient is not one number but a vector containing the derivative of the loss with respect to every parameter.
If parameters are w1, w2 and b, the gradient is [dL/dw1, dL/dw2, dL/db]. The optimiser updates all of them.
This is how one scalar loss can send learning signals throughout a large network.
The Forward Pass
Training begins with a forward pass. Inputs move through the network layer by layer. Weighted sums, activation functions and attention operations produce intermediate representations until the model generates an output.
The loss function compares that output with the training target. All intermediate operations form a computational graph that records how the output depended on parameters.
The Backward Pass
Backpropagation traverses the computational graph in reverse. Starting from the loss, it applies derivatives through each operation to compute parameter gradients.
If an operation’s output has little influence on the loss, its gradient may be small. If the loss is highly sensitive to a parameter, the gradient magnitude may be larger.
The Chain Rule
The chain rule connects nested functions. If loss L depends on prediction z, and z depends on weight w, then dL/dw = dL/dz × dz/dw.
Deep networks are long chains of functions. Backpropagation repeatedly applies this principle so the loss at the output can influence parameters in earlier layers.
Worked Two-Step Chain Rule Example
Let z = wx, a = z², and L = (a − y)². Suppose x = 2, w = 1, y = 9. Then z = 2, a = 4, L = 25.
dL/da = 2(a − y) = −10. da/dz = 2z = 4. dz/dw = x = 2.
By the chain rule, dL/dw = −10 × 4 × 2 = −80. This large negative gradient says increasing w locally reduces loss sharply.
Automatic Differentiation
Modern frameworks such as PyTorch, TensorFlow and JAX build or trace computation graphs and compute gradients automatically. Developers usually do not hand-derive every derivative in a billion-parameter model.
Automatic differentiation does not remove the mathematics. It automates repeated application of derivative rules.
Why Backpropagation Is Efficient
Naively perturbing every parameter one at a time to estimate its effect on loss would require enormous numbers of forward passes. Backpropagation reuses intermediate computations to obtain gradients for all parameters efficiently.
This efficiency is what makes gradient-based training practical for deep networks.
Finite Differences as a Gradient Check
For a small model, developers can verify automatic gradients numerically. Perturb one parameter by a tiny ε and estimate slope from the change in loss.
If analytic backpropagation and finite-difference estimates disagree strongly, there may be a bug in the implementation.
Finite differences are too expensive for ordinary large-scale training but useful for debugging tiny components.
Loss Functions Define What Training Optimises
Backpropagation computes gradients of the chosen loss. If the loss rewards the wrong thing, optimisation efficiently learns the wrong objective.
For language modeling, cross-entropy loss rewards probability on observed target tokens. For regression, squared error may be appropriate. Contrastive learning uses similarity-based objectives.
The optimiser cannot supply values or goals the loss never encoded.
Gradient Descent Versus Stochastic Gradient Descent
Full-batch gradient descent computes the gradient using the entire training dataset before one update. This is impractical for huge datasets.
Stochastic gradient descent uses one example or, more commonly, mini-batches. The gradient is noisy but much cheaper and can be updated continuously.
Most deep learning uses mini-batch optimisation.
Why Gradient Noise Can Help
Mini-batch gradients are imperfect estimates of the full-dataset gradient. The noise can help training move through complex loss landscapes rather than following one exact path.
Too much noise can destabilise training; too little can increase compute or lead to different optimisation behaviour. Batch size is therefore a hyperparameter.
Learning Rate: The Size of the Step
The learning rate scales the gradient update. Google’s Machine Learning Crash Course explains that a rate that is too low can make convergence extremely slow, while a rate that is too high can cause loss to fluctuate or diverge.
The correct rate depends on model, optimiser, batch size, training stage and other settings.
Worked Learning-Rate Comparison
Return to the one-weight model with gradient −16. At learning rate 0.001, the update changes w from 1 to 1.016—safe but slow. At 0.1, it changes to 2.6—large but helpful for this example. At 1.0, it jumps to 17, producing a prediction of 34 and enormous loss.
The same gradient can therefore produce progress or instability depending on learning rate.
Learning-Rate Schedules
Large models often change learning rate over time. A warmup period starts small; a peak rate accelerates learning; a decay phase reduces update sizes as training approaches a good region.
Schedules are hyperparameters and part of the experimental design.
Momentum
Plain stochastic gradient descent follows the current gradient. Momentum keeps a running direction from previous updates, helping smooth noisy gradients and accelerate movement through consistent slopes.
The intuition is a ball gaining velocity down a valley rather than responding independently to every small bump.
Adam and Adaptive Optimisers
Adam and related optimisers track moving averages of gradients and squared gradients to adapt update sizes for individual parameters.
These methods are widely used because they work well across many deep-learning problems, although optimiser choice and settings still require empirical evaluation.
Weight Decay
Weight decay penalises large parameter values or applies shrinkage during optimisation, acting as a form of regularisation.
It can improve generalisation in many settings, but the optimal strength depends on the training regime.
Vanishing Gradients
In deep networks, gradients can become extremely small as they propagate through many layers. Early layers then learn very slowly.
Google’s backpropagation guide identifies vanishing gradients as a common training problem. Activation functions, normalisation, residual connections and architecture design can mitigate it.
Exploding Gradients
Gradients can also grow extremely large. Updates then become unstable, loss can spike and numerical values can overflow.
Lower learning rates, normalisation, careful initialisation and gradient clipping are common tools for managing this failure mode.
Gradient Clipping
Gradient clipping limits gradient magnitude before the optimiser applies an update. It can prevent one unusually large batch from destabilising training.
Clipping treats the symptom of excessive gradient magnitude; developers should still investigate why gradients are exploding.
Dead ReLU Units
A ReLU activation outputs zero for negative inputs. If a unit’s input remains negative and gradients no longer move it back into the active region, the unit can effectively die.
Google’s backpropagation guide discusses dead ReLU units and notes that lower learning rates or ReLU variants can help.
Residual Connections
Residual connections let later layers add transformations to an earlier representation instead of replacing it entirely. They also provide shorter gradient paths through deep networks.
This architectural idea helped make very deep networks easier to optimise and is central to modern transformers.
Normalisation
Normalisation methods stabilise activation distributions and can improve gradient flow. Batch normalization is common in some networks; transformers commonly use layer normalization variants.
Normalisation affects optimisation dynamics, not the semantic goal of training.
Parameter Initialisation
Training begins from initial parameter values. If weights start too large or too small, activations and gradients can explode or vanish.
Modern initialisation schemes choose scale based on layer dimensions and activation functions to create a trainable starting point.
Loss Landscapes
With many parameters, the loss function forms a high-dimensional landscape. Gradient descent follows local slope rather than searching every possible parameter configuration.
Deep-network landscapes contain flat regions, saddle points and many directions. The simple picture of one bowl with one minimum is useful for intuition but incomplete.
Local Minima Are Not the Whole Story
In huge overparameterised networks, many parameter configurations can achieve low training loss. Generalisation depends on which solutions optimisation finds and how they behave on unseen data.
Training research therefore studies curvature, flatness, implicit regularisation and optimiser bias in addition to raw convergence.
Saddle Points
A saddle point slopes upward in some directions and downward in others. Gradients can become small even though the point is not a useful minimum.
Stochasticity and modern optimisers can help move through these regions.
Batch Size and Generalisation
Larger batches produce more stable gradient estimates and efficient accelerator utilisation. Smaller batches introduce more noise and may generalise differently.
There is no universal best batch size. It interacts with learning rate, hardware and data mixture.
Gradient Accumulation
When the desired effective batch does not fit in memory, gradients can be accumulated across several micro-batches before the optimiser step.
This changes scheduling while approximating a larger batch.
Mixed Precision
Training with lower-precision numerical formats reduces memory and increases throughput. Some operations remain in higher precision to preserve stability.
Loss scaling and hardware support help prevent tiny gradients from underflowing.
Distributed Backpropagation
Large training runs split model computation and data across many accelerators. Each device computes partial gradients; communication combines them for parameter updates.
Network bandwidth, synchronisation and fault tolerance become part of optimisation performance.
Gradient Checkpointing
Backpropagation normally needs intermediate activations from the forward pass. Storing all of them consumes large memory.
Gradient checkpointing saves only selected activations and recomputes others during the backward pass, trading extra computation for lower memory use.
Training Instability Is a Systems Problem Too
A loss spike may come from a pathological batch, numerical overflow, corrupted data, failed communication or a bad optimiser setting.
Monitoring needs both model metrics and infrastructure telemetry.
Backpropagation Does Not Explain the Model’s Reasoning
Gradients explain how training loss changes with parameters. They do not provide a human-readable chain of why the trained model later answers one question the way it does.
Training mechanics and interpretability are related but different research problems.
Backpropagation Does Not Mean the Model “Feels Error”
Loss is a numerical objective. The model does not need subjective awareness to have parameters updated.
Anthropomorphic language can obscure the actual mechanism: computation, derivative, update.
One Training Step Versus Long-Term Learning
A single update changes parameters slightly. Capability emerges from huge numbers of updates across varied data.
This is why one corrected user message at inference should not be casually described as “retraining the AI”.
Catastrophic Updates
An excessively large update can damage previously useful behaviour. Optimisers, learning-rate schedules and gradient clipping reduce this risk.
Fine-tuning on narrow data can also shift parameters enough to harm general capabilities, requiring regression tests.
Backpropagation Through Time
Recurrent neural networks historically used backpropagation through time, unrolling the sequence across time steps before applying gradient calculations.
Transformers process sequences differently, but gradients still propagate through the computational graph that produced the loss.
Attention Gradients
In a transformer, gradients flow through attention projections, softmax operations, value combinations, feed-forward layers, normalisation and residual paths.
Backpropagation is architecture-general: as long as operations are differentiable or handled by appropriate estimators, gradients can train the parameters.
Non-Differentiable Operations
Some decisions such as discrete sampling or external tool execution are not straightforwardly differentiable. Training may use surrogate objectives, reinforcement learning, straight-through estimators or separate supervised signals.
Not every part of an SI system is trained end-to-end with ordinary backpropagation.
Gradient Descent and Reinforcement Learning
Reinforcement learning can still use gradient-based optimisation. Policy-gradient methods estimate how actions influence expected reward and update parameters accordingly.
The learning signal is reward rather than a direct token target, but gradient-based parameter updates remain central.
Worked Example: Binary Classifier
Model output probability for “spam” is 0.30, but target label is 1. Cross-entropy loss is high because the model assigned low probability to the positive class.
Backpropagation computes gradients through the sigmoid or softmax and preceding layers. Updates increase or decrease weights depending on how features contributed to the error.
Across many emails, the model learns a decision surface that separates classes statistically.
Worked Example: Language Model Token
Context: “The capital of France is”. Target token: Paris. Suppose the model assigns Paris probability 0.20 and Lyon 0.30.
Cross-entropy loss penalises the low probability on Paris. Gradients change attention, embeddings and feed-forward parameters so future similar contexts can assign more probability to the correct target.
The update is distributed across many parameters; there is no single “Paris weight”.
Worked Example: Contrastive Embedding
A query and relevant document should have similar embeddings; an irrelevant document should be farther away. Contrastive loss measures these relationships.
Backpropagation changes encoder parameters so the query representation moves closer to the positive document and away from negatives under the objective.
A Backpropagation Failure Map
Level 1: wrong target or loss. Level 2: broken forward computation. Level 3: incorrect gradient implementation. Level 4: vanishing gradient. Level 5: exploding gradient. Level 6: poor learning rate. Level 7: unstable optimiser. Level 8: bad batch or data distribution. Level 9: converges on training but fails to generalise.
This eduKateSG map separates optimisation failures from data, architecture and downstream system failures.
Worked Diagnosis: Loss Does Not Move
Possible causes include learning rate near zero, frozen parameters, disconnected gradients, saturated activations, wrong labels or a constant-output bug.
Check gradients and parameter changes directly before scaling the model.
Worked Diagnosis: Loss Becomes NaN
Possible causes include numerical overflow, exploding gradients, invalid operations or corrupted data.
Inspect the first step where values become non-finite, lower the learning rate, check precision and validate inputs.
Worked Diagnosis: Training Loss Good, Validation Poor
Optimisation succeeded on the training data, but generalisation failed. The repair may involve data diversity, regularisation, model capacity or evaluation—not backpropagation itself.
A Practical Optimisation Checklist
Confirm the loss matches the objective. Verify forward outputs. Inspect gradient norms. Check that trainable parameters actually change. Plot training and validation loss. Monitor learning rate. Test a tiny batch that the model should be able to overfit.
The “overfit a tiny batch” test is useful: if a powerful network cannot fit a handful of examples, the training pipeline may be broken.
Independent Exercise 1: Direction
Gradient dL/dw = +5 and learning rate = 0.1. What direction does gradient descent move w?
Answer
w_new = w − 0.1×5, so w decreases by 0.5.
Independent Exercise 2: Negative Gradient
Gradient = −3 and learning rate = 0.2. What happens?
Answer
Subtracting a negative adds 0.6, so the parameter increases.
Independent Exercise 3: Learning Rate
Training loss oscillates wildly and grows. Which hyperparameter should you inspect first?
Answer
Learning rate is a primary suspect, although exploding gradients, bad data or numerical problems can also cause the pattern.
Independent Exercise 4: Vanishing Gradient
Early layers receive gradients near zero while later layers train. What problem does this suggest?
Answer
Vanishing gradients. Inspect activations, initialisation, normalisation and architecture.
Independent Exercise 5: Generalisation
Training loss is nearly zero but test loss is high. Did gradient descent fail?
Answer
Not necessarily. Optimisation succeeded on training data; the problem is overfitting or distribution mismatch.
From Scalar Calculus to Tensors
The one-weight example used ordinary derivatives. Real neural networks use vectors, matrices and higher-dimensional tensors. A weight matrix can contain millions of entries, and the loss depends on all of them.
Backpropagation computes partial derivatives for each element. The result has the same shape as the parameter tensor, so the optimiser can update every entry.
Linear algebra makes these operations efficient on GPUs and other accelerators. Matrix multiplications process many examples and parameters in parallel.
Jacobian Intuition
When a vector-valued function depends on a vector input, its local derivatives can be represented by a Jacobian matrix. Explicitly constructing huge Jacobians would be expensive.
Reverse-mode automatic differentiation avoids materialising every intermediate Jacobian. It efficiently computes vector-Jacobian products backward from the scalar loss.
This is why reverse-mode differentiation is well suited to neural networks with many parameters and one scalar objective.
Computational Graphs
A computational graph represents operations as nodes and dependencies as edges. For example: input → matrix multiply → add bias → activation → second layer → output → loss.
The forward pass computes values. The backward pass traverses dependencies in reverse, multiplying local derivatives by upstream gradients.
Frameworks record enough information during the forward pass to perform this reverse computation automatically.
Worked Graph Example
Let x = 3, w = 2, b = 1. First compute z = wx + b = 7. Then a = ReLU(z) = 7. Let target y = 5 and loss L = (a − y)² = 4.
Backward: dL/da = 2(a − y) = 4. Because z > 0, da/dz = 1. dz/dw = x = 3 and dz/db = 1. Therefore dL/dw = 4×1×3 = 12 and dL/db = 4.
A learning rate of 0.01 updates w to 1.88 and b to 0.96. The next prediction moves closer to the target.
Vectorised Batches
Instead of one x, the model receives a matrix of examples. One matrix multiplication produces activations for the entire batch. Loss is averaged or summed across examples.
Backpropagation then computes gradients reflecting the batch as a whole. This is why one unusually difficult example does not necessarily dominate the update unless the loss or weighting makes it dominant.
Per-Example Loss and Aggregate Loss
A batch can contain easy and hard examples. Per-example losses reveal which cases are driving the average. Monitoring only the mean can hide rare extreme failures.
Training systems sometimes clip, weight or resample examples based on loss, but these interventions change the effective objective and need evaluation.
Class Weighting
In imbalanced classification, positive examples may be rare. A weighted loss can increase their contribution so the model does not minimise loss by ignoring them.
The weighting encodes a training preference. It should reflect the evaluation objective rather than arbitrary tuning.
Label Smoothing
Instead of assigning probability target 1.0 to the correct class and 0 to every other class, label smoothing distributes a small amount of target probability across alternatives.
This can reduce overconfidence and improve generalisation in some classification settings. It deliberately changes the gradient signal.
Cross-Entropy Gradient Intuition
For a softmax classifier, the gradient with respect to the logits has a convenient form related to predicted probability minus target probability. If the correct class has probability too low, its logit receives pressure upward; incorrect classes receive pressure downward.
This local signal then propagates into earlier layers through the chain rule.
Sequence Loss Across Tokens
A language-model batch contains many token positions. Cross-entropy loss is computed at each valid target position and aggregated.
Padding tokens can be masked so they do not contribute to loss. If the mask is wrong, the optimiser can waste effort learning padding artefacts.
Loss masking is therefore part of training correctness.
Gradient Norms as a Diagnostic
The norm of a gradient vector summarises its magnitude. Tracking gradient norms across layers can reveal vanishing or exploding behaviour.
If one layer consistently has near-zero gradients, it may not be learning. If norms spike suddenly, the batch or optimisation state may be unstable.
Gradient Histograms
Histograms show the distribution of gradient values rather than only one norm. They can reveal saturation, dead units or unusually heavy tails.
Deep-learning dashboards often combine loss curves, learning rate, parameter norms and gradient statistics to make training observable.
Parameter Norms
Weights can grow during training. Monitoring parameter norms helps detect runaway updates or unexpected collapse.
Weight decay, normalisation and optimiser settings influence these trajectories.
Gradient Clipping by Norm
One common method rescales the entire gradient vector when its norm exceeds a threshold. The direction is preserved while magnitude is reduced.
Example: gradient norm 100, threshold 10. The vector is scaled by 0.1 so the new norm is 10.
This can stabilise recurrent networks and large language-model training where rare batches produce huge gradients.
Gradient Clipping by Value
Another method clips each gradient element to a fixed range, such as [−1, 1]. This changes direction more than norm clipping and has different behaviour.
The method and threshold are hyperparameters, not universal constants.
Accumulated Gradients and Zeroing
Frameworks often accumulate gradients by default. After an optimiser step, training code must clear or reset gradients before the next independent update unless accumulation is intentional.
Forgetting to zero gradients can silently change the effective batch and destabilise training.
Frozen Parameters
Fine-tuning can freeze some layers so gradients are not used to update them. Only selected parameters remain trainable.
This reduces compute and preserves parts of the pretrained representation. If the wrong layers are frozen, adaptation may be weak.
Adapters and Low-Rank Updates
Parameter-efficient fine-tuning methods can add small trainable modules or low-rank parameter updates while keeping most base weights fixed.
Backpropagation still computes gradients, but only a small subset of parameters changes. This lowers memory and storage costs for task-specific adaptation.
Gradient Flow Through Residual Blocks
Residual connections create direct additive paths from earlier to later representations. During backpropagation, gradients can flow through both the transformation and the identity path.
This helps very deep networks avoid losing all training signal in early layers.
Attention Softmax Gradients
Attention converts query-key scores into weights through softmax. Backpropagation computes how the loss changes with those weights, then with the query and key projections that produced the scores.
The result is that training can adjust which positions influence one another under different contexts.
Embedding Gradients
Token embedding tables are parameters too. When a token appears in training, gradients update the rows of the embedding matrix involved in that computation.
Frequent tokens therefore receive many updates. Rare tokens can receive fewer direct updates, although shared subwords and contextual layers still support them.
Tied Input and Output Embeddings
Some language models share weights between the input embedding table and the output token projection. Gradients from both roles affect the same parameters.
Weight tying reduces parameter count and can improve learning, but it also couples input and output representations.
Gradient Descent on Non-Convex Objectives
Deep-network loss functions are non-convex. Gradient methods are not guaranteed to find a unique global minimum.
Yet empirical optimisation works surprisingly well because networks are overparameterised and the landscape contains many useful low-loss solutions.
Training quality is therefore judged by validation and downstream behaviour rather than proof of a global optimum.
Convergence Is Task-Dependent
Loss flattening does not necessarily mean the model is finished. Some downstream capabilities continue improving slowly; others plateau or regress.
Training stops based on compute budget, validation trends, checkpoint comparisons and risk rather than one universal convergence threshold.
Early Stopping
If validation performance stops improving or begins worsening, training can stop before completing a planned number of epochs.
Early stopping reduces overfitting and compute waste in many supervised settings.
Overfitting Is Not a Backpropagation Bug
Backpropagation can perfectly perform its job—minimise training loss—while the model memorises the training set and generalises poorly.
The repair lies in data, regularisation, model capacity, validation and objective design.
Underfitting
If both training and validation loss remain high, the model may lack capacity, train too briefly, use poor features or optimise badly.
Diagnosis distinguishes inability to fit training data from inability to generalise.
Loss Plateaus
A plateau can arise when learning rate is too low, gradients are tiny, the optimiser is in a flat region or the model has reached the capacity allowed by the objective.
Learning-rate schedules, architecture changes or better data may help depending on the cause.
Warm Restarts and Cyclical Learning Rates
Some training schedules periodically raise the learning rate to explore new regions of the loss landscape. These techniques can escape plateaus or improve generalisation in certain tasks.
They are optimisation strategies, not replacements for correct objectives.
Second-Order Information
Gradient descent uses first derivatives. Second-order methods use curvature information from Hessians or approximations.
Exact second-order optimisation is expensive for large networks, but curvature approximations inspire optimisers and analysis tools.
Why We Do Not Use Exact Newton’s Method for Giant Models
Newton’s method would require storing or manipulating enormous second-derivative matrices. The memory and computation are prohibitive at modern model scale.
First-order methods offer a practical balance between information and cost.
Gradient Descent as an Iterative Repair Loop
Each batch exposes error, gradients allocate responsibility and the optimiser makes a correction. The next batch tests the revised parameters.
This resembles the broader Civilisation OS repair idea in structure, but here the mechanism is mathematical: loss gradient drives parameter updates.
Training Through Noisy Labels
If labels are wrong, gradients push the model toward wrong targets. Large datasets can tolerate some noise, but systematic label errors become systematic learning pressure.
Robust losses, filtering and human review can reduce the damage.
Gradient Contribution From Repeated Data
A duplicated example contributes its gradient repeatedly. This effectively weights the example more heavily.
Deduplication therefore changes optimisation, not just storage size.
Data Order and Optimiser State
Adaptive optimisers maintain running statistics. Presenting examples in a different order can lead to a different parameter trajectory even with the same dataset.
Reproducibility requires seeds, data order and optimiser state to be recorded where practical.
Checkpoint Resume
A checkpoint should save model parameters and often optimiser state, learning-rate schedule and random-number-generator state.
Resuming only the weights can produce a different training trajectory because momentum or adaptive statistics were lost.
Distributed Gradient Synchronisation
In data-parallel training, each accelerator processes a different mini-batch and computes local gradients. Collective communication averages or sums them before the update.
If one worker fails or contributes corrupted gradients, training can be affected. Distributed systems need validation and fault handling.
Gradient Compression
Communication can become a bottleneck. Research explores quantising or sparsifying gradients before transfer.
Compression saves bandwidth but can alter optimisation accuracy. It must be evaluated.
Model Parallelism
When one model does not fit on one device, layers or tensor operations can be split across devices. Backpropagation then sends gradients across those boundaries.
Pipeline bubbles and communication overhead become training-system concerns.
A Full One-Step Training Checklist
1. Load a batch. 2. Tokenise or encode inputs. 3. Run forward pass. 4. Compute loss. 5. Clear old gradients. 6. Backpropagate. 7. Inspect or clip gradients if required. 8. Apply optimiser step. 9. Update learning-rate schedule. 10. Log metrics.
Repeat while periodically validating and saving checkpoints.
Independent Exercise 6: Chain Rule
If dL/da = 3, da/dz = 4 and dz/dw = 2, what is dL/dw?
Answer
3 × 4 × 2 = 24.
Independent Exercise 7: Gradient Clipping
Gradient norm is 50 and clipping threshold is 5. Under norm clipping, by what factor is the vector scaled?
Answer
By 5/50 = 0.1, assuming standard norm clipping.
Independent Exercise 8: Frozen Layer
A layer has nonzero gradients but its parameters never change. What configuration might explain this?
Answer
The layer may be frozen or excluded from the optimiser’s parameter list.
Independent Exercise 9: Resume Training
You restore model weights but not optimiser state. Will the resumed run be exactly identical?
Answer
Not necessarily. Momentum or adaptive optimiser statistics may differ, changing subsequent updates.
A Full Training Iteration With Numbers
Take a tiny linear classifier with two inputs x1 = 1 and x2 = 2, weights w1 = 0.2 and w2 = −0.1, and bias b = 0. The logit is z = 0.2(1) + (−0.1)(2) = 0. Under a sigmoid, the predicted probability is 0.5.
Suppose the target label is 1. Binary cross-entropy loss is −log(0.5), about 0.693. For logistic regression, the derivative of the loss with respect to the logit is prediction minus target: 0.5 − 1 = −0.5.
Therefore dL/dw1 = −0.5×1 = −0.5, dL/dw2 = −0.5×2 = −1.0, and dL/db = −0.5. With learning rate 0.1, the new parameters become w1 = 0.25, w2 = 0.0, b = 0.05.
The next logit is 0.25(1) + 0(2) + 0.05 = 0.30. Sigmoid(0.30) is about 0.574, closer to the target 1. One gradient step moved the model in the useful direction.
Why Backpropagation Scales to Deep Networks
The same principle applies when the computation contains embeddings, attention, normalization and many feed-forward layers. Each operation contributes a local derivative. Reverse-mode automatic differentiation composes those local derivatives backward from the scalar loss.
The engineering challenge is memory, numerical stability and communication—not a different fundamental learning rule. Deep learning is large-scale repeated calculus organised by software and hardware.
Optimisation Completion Test
A reader has understood optimisation when they can distinguish four failures. If targets are wrong, fix the data or objective. If gradients are wrong, fix the computational graph or implementation. If gradients are correct but updates destabilise training, fix learning rate or optimiser settings. If training loss is low but validation is poor, fix generalisation rather than blaming backpropagation.
This diagnostic discipline is more useful than saying “the model did not learn”. The training loop has identifiable stages, and each stage produces evidence that can be inspected.
Optimisation Repair, Stabilise and Extend
Repair: begin by proving the training loop works on a tiny dataset. Inspect forward outputs, loss, gradients and actual parameter changes. If the model cannot fit a handful of examples, do not launch a massive training run.
Stabilise: after the loop works, monitor learning rate, gradient norms, loss curves and validation metrics across representative batches. Add clipping, schedules or normalisation only when evidence shows they solve a real instability.
Extend: scale batch size, model depth and distributed hardware gradually. Re-run the same tiny-batch and validation checks after infrastructure changes so a systems optimisation does not silently break mathematical training.
One More Numerical Exercise
Suppose a parameter w = 4, gradient = 2.5 and learning rate = 0.04. Gradient descent gives w_new = 4 − 0.04×2.5 = 3.9. If the next loss decreases, the local step was useful. If loss increases repeatedly, inspect learning rate, gradient correctness and batch variability.
Now suppose momentum or Adam is used. The update may not equal the raw current gradient times learning rate because optimiser state contributes. This is why debugging should inspect the actual optimiser configuration rather than assuming plain SGD.
The general training law still holds: gradients encode local sensitivity, the optimiser converts them into parameter updates, and validation decides whether those updates created useful generalisation.
Backpropagation Reality Check
Backpropagation can optimise almost any differentiable objective, including a badly specified one. The existence of stable gradients therefore proves only that the training mechanism is functioning, not that the model is learning the right behaviour.
A complete training system needs the right data, target, loss, optimiser, architecture and validation. Backpropagation is the credit-assignment engine inside that larger loop.
From Gradient to Capability: The Missing Middle
A gradient step is tiny and local. Capability emerges only after many such steps reshape representations across the network. One update can increase the probability of one target, but millions of related updates can create reusable structures for syntax, code, mathematics or retrieval.
This is why inspecting a single parameter rarely explains a high-level ability. Training distributes credit across many interacting weights. The right evidence for capability is behavioural evaluation after optimisation, not a story about one “knowledge neuron”.
Optimisation and Reproducibility
Two runs with the same architecture and dataset can diverge because of random initialisation, data order, dropout, distributed timing and numerical precision. Reproducibility therefore requires recording seeds, software versions, optimiser state, data version and hardware-relevant settings where practical.
Exact bit-for-bit repetition is not always feasible at large scale, but experiment records should be detailed enough to explain material differences and reproduce conclusions.
Optimisation Design Worksheet
Before a training run, define the loss, trainable parameters, optimiser, learning-rate schedule, batch size, gradient clipping policy, precision format and validation cadence. Then define failure thresholds: maximum gradient norm, non-finite loss, validation regression and checkpoint recovery procedure.
This converts “train the model” into an inspectable engineering process with clear sensors and stopping conditions.
A Final Training-Step Reality Check
When a training run behaves strangely, inspect the actual update loop before reaching for a larger model. Confirm that the batch contains the intended targets, the forward pass produces finite values, the loss is attached to the correct outputs, gradients exist for trainable parameters and the optimiser changes those parameters after each intended step.
Then inspect scale: gradient norms, learning rate, batch size, precision format and validation behaviour. This sequence distinguishes a broken training pipeline from a model that is simply difficult to optimise.
Backpropagation is powerful because it assigns numerical credit efficiently across enormous networks. It is not a guarantee that the chosen data, objective or deployment goal is correct. Optimisation can only become as useful as the target it is asked to minimise.
Frequently Asked Questions About Backpropagation and Gradient Descent
What is backpropagation?
An efficient algorithm that applies the chain rule backward through a computational graph to compute gradients of loss with respect to trainable parameters.
What is gradient descent?
A family of optimisation methods that update parameters in directions intended to reduce loss.
Are backpropagation and gradient descent the same?
No. Backpropagation computes gradients; gradient-based optimisers use those gradients to update parameters.
What is a gradient?
A vector of partial derivatives showing how the loss changes locally with respect to each parameter.
What is learning rate?
A hyperparameter that scales update size. Too small can train slowly; too large can destabilise training.
Why use mini-batches?
They make training computationally practical and provide stochastic gradient estimates.
Why do gradients vanish or explode?
Repeated multiplication through deep networks can shrink or enlarge gradient values depending on weights, activations and architecture.
Does backpropagation happen during normal chat inference?
Not usually. Standard inference performs a forward pass without updating model parameters. Training or fine-tuning adds the backward pass and optimiser step.
Backpropagation Is the Credit-Assignment Engine of Training
Pretraining provides examples and objectives. Backpropagation computes how the error depends on the model’s parameters. Gradient-based optimisation changes those parameters. Repeating this loop across vast datasets turns random initialisation into useful capability.
The mathematics is systematic rather than mysterious: forward computation, loss, derivatives, parameter update. The engineering challenge comes from doing it stably across deep networks, huge datasets and distributed hardware.
Continue through the How Super Intelligence Works hub. Previous: 023 — Self-Supervised Learning. Next: 025 — Scaling.
How Super Intelligence Works Series Navigation
Previous: 023 — Self-Supervised Learning · Series Hub · Next: 025 — Scaling