Chapter map

1. 1. Gradients
2. 2. Mean
3. 3. Deviations
4. 4. Outer products
5. 5. Population covariance
6. 6. Cross covariance
7. 7. Anisotropy
8. 8. Batch rule
9. 9. Batch four
10. 10. Learning rate
11. 11. Update covariance
12. 12. Standard deviations
13. 13. Curvature comparison
14. 14. Stationarity check
15. 15. Non-Gaussian check
16. 16. Claim

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Gradients

Here the geometry becomes operational. Use (3,0),(1,0),(2,2),(2,−2).

A concrete situation

Four equally likely two-dimensional gradients define the population.

What the mathematics does not license

Keep coordinate units fixed. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Record vectors. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Mean

This step deserves a careful pause. Average coordinate-wise.

A concrete situation

g=(2,0).

What the mathematics does not license

Use training sampling weights. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Record mean. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. Deviations

The useful question is not only how to calculate, but what the calculation assumes. Subtract the mean.

A concrete situation

(1,0),(−1,0),(0,2),(0,−2).

What the mathematics does not license

Centre once. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Record vectors. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Outer products

A small example makes the mechanism visible. Form ddᵀ for each deviation.

A concrete situation

First pair contributes x variance; second pair y variance.

What the mathematics does not license

Outer products are matrices. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Record terms. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Population covariance

This is where modelling choices begin to affect evidence. Average four outer products.

A concrete situation

C=diag(.5,2).

What the mathematics does not license

This uses the 1/n population convention. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Record C. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Cross covariance

The central mathematical idea is concrete. Inspect off-diagonal entries.

A concrete situation

Both are zero.

What the mathematics does not license

Zero here follows the designed symmetry. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Record values. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. Anisotropy

Here the geometry becomes operational. Divide 2 by .5.

A concrete situation

Y-direction variance is four times x-direction variance.

What the mathematics does not license

A scalar trace would hide direction. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Record ratio. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Batch rule

This step deserves a careful pause. For replacement sampling use C_B=C/B.

A concrete situation

Covariance shrinks with batch size.

What the mathematics does not license

Without replacement needs finite-population correction. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

State rule. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Batch four

The useful question is not only how to calculate, but what the calculation assumes. Set B=4.

A concrete situation

C_B=diag(.125,.5).

What the mathematics does not license

Batch examples are independent in the model. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Record matrix. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Learning rate

A small example makes the mechanism visible. Set η=.1.

A concrete situation

Update noise scales by η.

What the mathematics does not license

The covariance scales by η². The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Record η. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Update covariance

This is where modelling choices begin to affect evidence. Compute η²C_B.

A concrete situation

Matrix=diag(.00125,.005).

What the mathematics does not license

Momentum changes the temporal covariance. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Record result. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Standard deviations

The central mathematical idea is concrete. Take square roots.

A concrete situation

Values≈.035355 and .070711.

What the mathematics does not license

Standard deviation is in update units. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Record values. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Curvature comparison

Here the geometry becomes operational. Pair C with a local Hessian.

A concrete situation

Diffusion and restoring dynamics can be compared.

What the mathematics does not license

Matrices may not share eigenvectors. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Report alignment. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Stationarity check

This step deserves a careful pause. Run constant-step dynamics after apparent equilibration.

A concrete situation

Empirical moments can be compared with a local model.

What the mathematics does not license

Training may never be stationary. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Test windows. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. Non-Gaussian check

The useful question is not only how to calculate, but what the calculation assumes. Inspect tails and autocorrelation.

A concrete situation

Diffusion assumptions can be challenged.

What the mathematics does not license

Do not rely on covariance alone. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Plot diagnostics. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Claim

A small example makes the mechanism visible. State sampling, covariance convention, batch and step.

A concrete situation

The teaching update has fourfold directional variance anisotropy.

What the mathematics does not license

Do not call it a posterior automatically. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Bound conclusion. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

SGD noise covariance records the direction and scale of minibatch-gradient uncertainty. A scalar variance says how much noise exists overall; a covariance matrix also shows where it acts. Near a local optimum, constant-step stochastic gradient dynamics can sometimes be approximated by a diffusion or Ornstein–Uhlenbeck process under strong assumptions, linking optimization noise, curvature and a stationary distribution.

Use four equally likely two-dimensional per-example gradients (3,0), (1,0), (2,2) and (2,−2). Their mean is g=(2,0). Deviations are (1,0), (−1,0), (0,2) and (0,−2). Using the population 1/n convention, the covariance is C=diag(.5,2).

For an independent batch mean of size B sampled with replacement, covariance becomes C/B. At B=4 it is diag(.125,.5). With learning rate η=.1, update-noise covariance is η²C/B=diag(.00125,.005). The second direction carries four times the variance of the first even though its mean gradient is zero.

Real SGD noise is state-dependent, non-Gaussian, correlated across examples and altered by augmentation, momentum, clipping, adaptive preconditioning and without-replacement sampling. A local diffusion model is an approximation, not automatic Bayesian inference. Report covariance convention, sampling, batch, checkpoint, learning rate, preconditioner, curvature assumptions and validation diagnostics.

---

Frequently asked questions

What is SGD noise covariance?

The covariance of stochastic gradient error around its mean under a declared data-sampling rule.

Why use a matrix rather than one variance?

A matrix shows which parameter directions are noisy and how directions co-vary.

How does batch size affect it?

For independent replacement sampling, batch-mean covariance is the per-example covariance divided by batch size.

When does a diffusion approximation help?

Locally, with small constant steps and regularity assumptions that must be checked.

Is constant-step SGD automatically Bayesian inference?

No. Posterior interpretations require model, temperature, curvature and stationarity conditions.

What must be reported?

Gradient definition, sampling, covariance denominator, batch, learning rate, momentum, clipping, preconditioner, checkpoint and diagnostics.

---

Primary source and fact-check note

Stephan Mandt, Matthew D. Hoffman and David M. Blei, “Stochastic Gradient Descent as Approximate Bayesian Inference,” Journal of Machine Learning Research 18(134), 1–35 (2017). Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• Stochastic Gradient Langevin Dynamics
• Natural gradient
• Hessian eigenvalue density
• K-FAC Fisher blocks
• How Mathematical Stability Works

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: stochastic gradients, convergence and stability evidence

Identity audit

Preserve example IDs, sampling weights and augmentation states when forming gradients.

Covariance audit

State centring vector, denominator, shrinkage and whether gradients are clipped or preconditioned.

Dynamics audit

Combine covariance with local curvature and learning rate without assuming shared eigenvectors.

Approximation audit

Test Gaussianity, autocorrelation, stationarity and finite-step error before a diffusion or posterior story.

Mechanism workshop: calculate, challenge and transfer

Manual reconstruction

Rebuild the smallest example for anisotropic stochastic-gradient diffusion. Label gradients, matrices, conventions and every unit.

Batch experiment

Change batch size under one sampling rule and predict mean, covariance and normalized error before calculating.

Step experiment

Change the learning rate while holding the objective and checkpoint fixed. Record stability factors and update covariance.

Direction experiment

Compare scalar summaries with eigenvalues, coordinate variances or singular directions.

Trajectory experiment

Measure early, middle and late checkpoints with identical estimators and preserve raw sequences.

Estimator experiment

Compare exact small-population values with minibatch or stochastic approximations and quantify bias.

Student transfer

Apply the mechanism to a tiny regression, classifier or repeated-measurement learning problem.

Parent explanation

Explain why more examples in one update reduce noise yet eventually give smaller time savings.

Limit statement

Write one supported claim, one overclaim and one repaired claim naming objective, sample and assumptions.

Next measurement

Choose a complementary curvature, prediction, calibration or robustness check and explain its separate evidence.

Evidence notebook: reconstruct every mathematical step

Evidence notebook 1: Gradients

Step 1 isolates gradients. Use (3,0),(1,0),(2,2),(2,−2). The reproducible result is: Four equally likely two-dimensional gradients define the population. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Keep coordinate units fixed. Record vectors. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 2: Mean

Step 2 isolates mean. Average coordinate-wise. The reproducible result is: g=(2,0). This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Use training sampling weights. Record mean. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 3: Deviations

Step 3 isolates deviations. Subtract the mean. The reproducible result is: (1,0),(−1,0),(0,2),(0,−2). This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Centre once. Record vectors. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 4: Outer products

Step 4 isolates outer products. Form ddᵀ for each deviation. The reproducible result is: First pair contributes x variance; second pair y variance. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Outer products are matrices. Record terms. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 5: Population covariance

Step 5 isolates population covariance. Average four outer products. The reproducible result is: C=diag(.5,2). This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: This uses the 1/n population convention. Record C. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 6: Cross covariance

Step 6 isolates cross covariance. Inspect off-diagonal entries. The reproducible result is: Both are zero. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Zero here follows the designed symmetry. Record values. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 7: Anisotropy

Step 7 isolates anisotropy. Divide 2 by .5. The reproducible result is: Y-direction variance is four times x-direction variance. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: A scalar trace would hide direction. Record ratio. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 8: Batch rule

Step 8 isolates batch rule. For replacement sampling use C_B=C/B. The reproducible result is: Covariance shrinks with batch size. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Without replacement needs finite-population correction. State rule. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 9: Batch four

Step 9 isolates batch four. Set B=4. The reproducible result is: C_B=diag(.125,.5). This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Batch examples are independent in the model. Record matrix. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 10: Learning rate

Step 10 isolates learning rate. Set η=.1. The reproducible result is: Update noise scales by η. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: The covariance scales by η². Record η. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 11: Update covariance

Step 11 isolates update covariance. Compute η²C_B. The reproducible result is: Matrix=diag(.00125,.005). This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Momentum changes the temporal covariance. Record result. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 12: Standard deviations

Step 12 isolates standard deviations. Take square roots. The reproducible result is: Values≈.035355 and .070711. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Standard deviation is in update units. Record values. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 13: Curvature comparison

Step 13 isolates curvature comparison. Pair C with a local Hessian. The reproducible result is: Diffusion and restoring dynamics can be compared. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Matrices may not share eigenvectors. Report alignment. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 14: Stationarity check

Step 14 isolates stationarity check. Run constant-step dynamics after apparent equilibration. The reproducible result is: Empirical moments can be compared with a local model. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Training may never be stationary. Test windows. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 15: Non-Gaussian check

Step 15 isolates non-gaussian check. Inspect tails and autocorrelation. The reproducible result is: Diffusion assumptions can be challenged. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Do not rely on covariance alone. Plot diagnostics. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 16: Claim

Step 16 isolates claim. State sampling, covariance convention, batch and step. The reproducible result is: The teaching update has fourfold directional variance anisotropy. This is evidence about directional stochastic-update uncertainty. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Do not call it a posterior automatically. Bound conclusion. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Failure cases and responsible recovery

Scalar collapse

Trace alone hides noisy directions and cross-covariance. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Mean error

Covariance must centre around the correct expected gradient. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Denominator opacity

Population and unbiased sample covariances differ. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Replacement shortcut

Finite datasets sampled without replacement need correction. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Momentum omission

Temporal filtering changes effective noise. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Gaussian assumption

Matching covariance does not make noise normal. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Stationarity story

Training checkpoints may not define a stationary process. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Posterior claim

Approximate diffusion is not automatic Bayesian sampling. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.