Did you know? An AI model can improve after getting an answer wrong without anyone typing the correct weights into it. Much of that improvement comes from a mathematical method called gradient descent. It repeatedly changes model parameters in directions that are expected to reduce a chosen error.
The quick answer: make a prediction → calculate a loss → compute gradients → update parameters → check again. Repeat until the model meets a suitable stopping criterion, then evaluate it on separate examples. The process is optimisation, not proof that every new answer will be true.
Imagine a student trying to find the lowest point in a valley while standing in fog. The slope under their feet provides a hint about which direction leads downhill. Gradient descent uses a mathematical slope to make a similar local decision, although real training happens in spaces with far more dimensions.
Gradient descent in 30 seconds
- Parameter: a weight or bias the model is allowed to adjust during training.
- Loss function: a mathematical measure of how poorly predictions meet the training objective.
- Gradient: information about how the loss changes when parameters change.
- Learning rate: the step-size setting used in parameter updates.
- Optimiser: the algorithm that decides how to update parameters using gradient information.
- Convergence: a state where further training changes produce little useful reduction in the chosen loss.
An important distinction: parameters are learned by training, while the learning rate is usually chosen as a hyperparameter by the developer or an automated tuning procedure. Both affect the result, but in different ways.
Step 1: Tell the model what counts as an error
Suppose an imaginary prediction system estimates a journey will take 20 minutes, but the recorded journey lasted 25. The difference is five minutes. A training objective might penalise the squared difference, the absolute difference or another suitable measure.
Different objectives treat mistakes differently. Squared error gives particularly large errors stronger weight. For a classification task, the loss may measure how much probability the model assigned to the intended category. The correct choice depends on the task and the consequences of error.
Developers should not assume that a lower loss means the model is more useful in every situation. A loss can be optimised using incomplete labels or misleading examples. Honest evaluation must ask whether improvement transfers to unfamiliar, relevant cases.
Step 2: Find the direction of change
If a model has one trainable weight, the loss can sometimes be drawn as a curve. Its derivative describes the local slope. A positive slope suggests that moving the weight slightly lower will reduce the loss; a negative slope suggests the opposite.
With many weights, the gradient is a collection of partial derivatives, one for each parameter. It indicates how the chosen loss is locally sensitive to changes in different directions. For neural networks, backpropagation efficiently computes gradient information using the chain rule.
Gradients are not explanations of the real-world cause of a prediction. They help a training algorithm change its numerical parameters.
Step 3: Take a step in the negative-gradient direction
A familiar gradient-descent update is new parameter = old parameter − learning rate × gradient. The minus sign matters: for a loss we want to minimise, the update ordinarily moves against the local gradient.
Think of the learning rate as a step-size knob. Small steps may make progress slowly. Very large steps can jump past a useful region and cause the loss to fluctuate or increase. There is no universally best setting for every model and dataset.
A practical training system may also use scheduled learning-rate changes or a specialised optimiser. Those techniques change the update procedure, but the basic purpose remains to improve a defined training objective.
Try a tiny worked example with real arithmetic
Here is a toy mathematical problem, not a real AI benchmark. Suppose the loss for one model parameter is L(w) = (w − 4)². The minimum is at w = 4 because that makes the squared error zero.
Its gradient is 2(w − 4). Start at w = 0, so the gradient is −8. Choose a learning rate of 0.25. The next weight is 0 − 0.25 × (−8) = 2.
The loss falls from L(0) = 16 to L(2) = 4. Repeat: the gradient at w = 2 is −4, so the next update gives 2 − 0.25 × (−4) = 3, and the loss falls again to L(3) = 1.
The model is moving closer to four: 0, then 2, then 3. You can see the idea without any specialised hardware. Real neural networks adjust many parameters at once, often using mini-batches of training data.
What changed? The parameter moved. What did not happen? The mathematics did not independently confirm that the dataset, target or application was appropriate.
Step 4: See what happens when the learning rate is too large
Return to the same toy loss. With w = 0 and learning rate 1.1, the update would be 0 − 1.1 × (−8) = 8.8. The new loss is (8.8 − 4)² = 23.04, higher than the starting loss of 16.
This demonstrates overshooting: a large step moves past the minimum and lands farther away. On this particular quadratic, that step size can lead to oscillations that grow. Many real objectives are more complicated, with plateaus and curved regions that make the appropriate rate less obvious.
A learning rate that is too small is not automatically safer for a practical training job: it can waste time or fail to make enough progress within available resources. Developers tune it using suitable development information.
Step 5: Learn from full datasets or mini-batches
The phrase batch gradient descent commonly refers to an update based on a full training dataset. Stochastic gradient descent can refer to updates using individual randomly sampled examples, while mini-batch methods use selected groups.
Mini-batches can make training more computationally convenient, especially on accelerators, while introducing variation into the gradient estimate. Small batches may produce noisy updates; large batches may require substantial memory and exhibit different optimisation behaviour.
The best approach depends on model architecture, hardware, dataset size and training objectives. For a reader, the central point is that a training gradient may be estimated from part of the dataset rather than recomputed from every example after every small update.
Step 6: Recognise that not every optimiser is plain gradient descent
Common training tools use methods such as SGD with momentum or Adam. Momentum can incorporate information from earlier gradients so updates do not respond only to the latest step. Adam uses adaptive estimates of gradient statistics to scale updates by parameter.
These methods can improve optimisation in suitable circumstances, but their names do not guarantee convergence or generalisation. Different optimisers, schedules and regularisation settings can produce different results on the same model.
The safe way to compare them is to hold the task and evaluation process consistent, report relevant settings and test the final model rather than only its training speed.
Step 7: Know when to stop
A training system may monitor its training loss and a separate validation measure. If training loss keeps falling but validation performance gets worse, the model may be overfitting the training examples.
Stopping early at a suitable checkpoint can help avoid needless optimisation of training details that do not carry over to new cases. This is one reason an independent development set is useful.
The final test set should not be used repeatedly to choose the learning rate, stopping step and model architecture. Doing so would weaken the independence of that final evaluation.
When gradient descent looks busy but nothing useful improves
- Wrong target: the optimiser faithfully minimises a loss that does not capture the user’s real goal.
- Misleading labels: incorrect examples can steer parameters in the wrong direction.
- Excessive learning rate: updates can overshoot and become unstable.
- Tiny learning rate: training may progress far too slowly.
- Overfitting: training error falls while performance on unfamiliar examples deteriorates.
- Data leakage: the apparent test success is exaggerated because future or repeated examples crossed the evaluation boundary.
A practical way to explain this to a student
Ask a student to solve a new algebra question and compare the solution with a checked example. The student identifies which step caused the error and corrects the method. This is a helpful analogy to iterative improvement, but humans reason in ways that do not reduce to numerical gradient updates.
For an AI model, the “correction” is a change to many weights produced by an optimiser. A good trainer still needs examples that represent the intended task, as well as a way to verify that the new model behaves better outside the training set.
Frequently asked questions
Is gradient descent the same as backpropagation?
No. Backpropagation is a method of computing gradients through a computational network. An optimiser such as gradient descent uses gradient information to update parameters.
Does a smaller training loss mean the model is more accurate?
Not necessarily. It means the chosen training objective has improved on those examples. Accuracy and usefulness on unseen real tasks require separate evaluation.
Why can a high learning rate be a problem?
Updates may move too far in parameter space, making the loss unstable or preventing convergence.
Do all AI algorithms use gradient descent?
No. Decision trees, some search methods and other algorithms may use different training or optimisation procedures.
Can gradient descent make a model know the truth?
It adjusts parameters to reduce a mathematical loss. Trustworthiness depends on data, task design, sources and evaluation.
Your one-minute takeaway
- Gradient descent uses loss gradients to adjust learnable parameters.
- Learning rate controls the step size and can make updates too slow or unstable.
- Success means useful performance on independent tasks, not only a low training loss.
Trusted learning and related articles
- Google Machine Learning Crash Course: Gradient Descent — an official introduction with a worked example.
- Google: Learning Rate and Hyperparameters — why step size matters.
- PyTorch: Optimising Model Parameters — loss, gradients and optimisers in a practical training loop.
- How Neural Networks Turn Numbers Into Predictions — understand the weights being updated.
- How AI Learns From Examples — training, validation and independent testing.
- How AI Is Tested Before Release — verifying that optimisation actually helps.
- How AI Works: Complete Overview — explore more explanations.