Chapter map

1. 1. Per-example gradients
2. 2. Mean signal
3. 3. Deviations
4. 4. Squared deviations
5. 5. Noise variance
6. 6. Signal power
7. 7. Noise scale
8. 8. Batch variance
9. 9. Batch one
10. 10. Batch two
11. 11. Batch eight
12. 12. Batch sixteen
13. 13. Checkpoint sweep
14. 14. Vector form
15. 15. Parallel test
16. 16. Claim

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Per-example gradients

A small example makes the mechanism visible. Use scalar values [−2,2,6,2].

A concrete situation

Four equally likely examples define the teaching population.

What the mathematics does not license

Gradient units and loss scaling matter. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record values. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Mean signal

This is where modelling choices begin to affect evidence. Average all four values.

A concrete situation

g=2.

What the mathematics does not license

Use the same sampling weights as training. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record mean. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. Deviations

The central mathematical idea is concrete. Subtract two from each gradient.

A concrete situation

Values are [−4,0,4,0].

What the mathematics does not license

Do not centre around a minibatch mean for this population example. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record deviations. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Squared deviations

Here the geometry becomes operational. Square each deviation.

A concrete situation

Values are [16,0,16,0].

What the mathematics does not license

Retain every example. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record terms. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Noise variance

This step deserves a careful pause. Average squared deviations.

A concrete situation

Variance=8 under the 1/n convention.

What the mathematics does not license

An unbiased sample estimator uses another denominator. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Name convention. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Signal power

The useful question is not only how to calculate, but what the calculation assumes. Square the mean gradient.

A concrete situation

Signal²=4.

What the mathematics does not license

A near-zero signal makes the ratio unstable. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record value. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. Noise scale

A small example makes the mechanism visible. Divide eight by four.

A concrete situation

B_noise=2.

What the mathematics does not license

This is a scalar one-dimensional example. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record statistic. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Batch variance

This is where modelling choices begin to affect evidence. Use Var(mean)=8/B for replacement sampling.

A concrete situation

Noise falls inversely with B.

What the mathematics does not license

Without-replacement correction differs. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

State sampling. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Batch one

The central mathematical idea is concrete. Set B=1.

A concrete situation

Normalized MSE=2.

What the mathematics does not license

A single example is noisier than the signal. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record ratio. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Batch two

Here the geometry becomes operational. Set B=2.

A concrete situation

Normalized MSE=1.

What the mathematics does not license

Noise and squared signal are comparable. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record ratio. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Batch eight

This step deserves a careful pause. Set B=8.

A concrete situation

Normalized MSE=.25.

What the mathematics does not license

This assumes repeated sampling from the population distribution. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record ratio. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Batch sixteen

The useful question is not only how to calculate, but what the calculation assumes. Set B=16.

A concrete situation

Normalized MSE=.125.

What the mathematics does not license

Doubling still halves noise but may not halve wall time. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record ratio. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Checkpoint sweep

A small example makes the mechanism visible. Estimate during training.

A concrete situation

Noise scale can change as loss changes.

What the mathematics does not license

One initial value is not a schedule. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Plot trajectory. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Vector form

This is where modelling choices begin to affect evidence. Use trace covariance over squared mean norm.

A concrete situation

Multiple parameter directions contribute.

What the mathematics does not license

Parameter scaling changes the scalar summary. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Report formula. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. Parallel test

The central mathematical idea is concrete. Run several batch sizes with tuned learning rates.

A concrete situation

Predicted and measured speedups can be compared.

What the mathematics does not license

Hardware and optimizer confound timing. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record compute. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Claim

Here the geometry becomes operational. State model, loss, checkpoint and sampling.

A concrete situation

The teaching population has gradient noise scale two.

What the mathematics does not license

Do not declare a universal optimal batch. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Bound conclusion. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

Gradient noise scale compares the variation of per-example gradients with the squared size of their mean. A useful scalar form is B_noise=E||g_i−g||²/||g||², where g is the population or full-data gradient under the stated sampling distribution. The statistic helps explain when increasing batch size still reduces gradient uncertainty and when extra parallel examples become increasingly redundant.

Take four equally likely scalar per-example gradients [−2,2,6,2]. Their mean is two. Deviations from the mean are [−4,0,4,0], so the mean squared deviation is (16+0+16+0)/4=8. The squared signal is 2²=4, giving B_noise=8/4=2.

For independent sampling with replacement, the variance of a batch-mean gradient is 8/B. Relative to the squared true gradient, the normalized mean-squared error is 2/B. It is two at batch size one, one at batch size two, .25 at batch size eight and .125 at batch size sixteen.

This teaching calculation is exact under its sampling rule, but real estimates depend on model checkpoint, data distribution, loss scaling, gradient clipping, augmentation, sequence length, parameter grouping and estimator bias. Gradient noise scale can guide batch experiments; it does not name a universally optimal batch or guarantee speed, cost, convergence or generalization.

---

Frequently asked questions

What is gradient noise scale?

A ratio comparing per-example gradient variation with squared mean-gradient signal under a declared sampling rule.

Why does it relate to batch size?

Independent batch averaging divides gradient covariance by batch size, creating diminishing noise reduction.

Is it one fixed number for a model?

No. It can change with checkpoint, loss, data, augmentation and parameterization.

Does it determine the best batch exactly?

No. It provides guidance that must be tested with learning-rate tuning, hardware and performance targets.

Does a larger batch always improve generalization?

No. Optimization, compute and statistical outcomes require separate evidence.

What must be reported?

Model, loss, checkpoint, sampling, estimator, clipping, batch, learning-rate tuning, compute metric and uncertainty.

---

Primary source and fact-check note

Sam McCandlish, Jared Kaplan, Dario Amodei and the OpenAI Dota Team, “An Empirical Model of Large-Batch Training,” arXiv:1812.06162 (2018); official OpenAI research summary. Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• Natural gradient
• Hessian eigenvalue density
• Dynamical isometry
• How Mathematical Stability Works
• eduKateSingapore learning routes

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: stochastic gradients, convergence and stability evidence

Sampling audit

Match the per-example distribution, weights, augmentation and replacement rule to actual training.

Signal audit

Report mean-gradient norm and warn when it is near zero.

Estimator audit

Separate population variance, unbiased sample estimates and paired small-versus-large batch estimators.

Systems audit

Test predicted batch gains against learning-rate tuning, hardware throughput, communication and target performance.

Mechanism workshop: calculate, challenge and transfer

Manual reconstruction

Rebuild the smallest example for gradient signal-to-noise and batch returns. Label gradients, matrices, conventions and every unit.

Batch experiment

Change batch size under one sampling rule and predict mean, covariance and normalized error before calculating.

Step experiment

Change the learning rate while holding the objective and checkpoint fixed. Record stability factors and update covariance.

Direction experiment

Compare scalar summaries with eigenvalues, coordinate variances or singular directions.

Trajectory experiment

Measure early, middle and late checkpoints with identical estimators and preserve raw sequences.

Estimator experiment

Compare exact small-population values with minibatch or stochastic approximations and quantify bias.

Student transfer

Apply the mechanism to a tiny regression, classifier or repeated-measurement learning problem.

Parent explanation

Explain why more examples in one update reduce noise yet eventually give smaller time savings.

Limit statement

Write one supported claim, one overclaim and one repaired claim naming objective, sample and assumptions.

Next measurement

Choose a complementary curvature, prediction, calibration or robustness check and explain its separate evidence.

Evidence notebook: reconstruct every mathematical step

Evidence notebook 1: Per-example gradients

Step 1 isolates per-example gradients. Use scalar values [−2,2,6,2]. The reproducible result is: Four equally likely examples define the teaching population. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Gradient units and loss scaling matter. Record values. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 2: Mean signal

Step 2 isolates mean signal. Average all four values. The reproducible result is: g=2. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Use the same sampling weights as training. Record mean. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 3: Deviations

Step 3 isolates deviations. Subtract two from each gradient. The reproducible result is: Values are [−4,0,4,0]. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Do not centre around a minibatch mean for this population example. Record deviations. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 4: Squared deviations

Step 4 isolates squared deviations. Square each deviation. The reproducible result is: Values are [16,0,16,0]. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Retain every example. Record terms. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 5: Noise variance

Step 5 isolates noise variance. Average squared deviations. The reproducible result is: Variance=8 under the 1/n convention. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: An unbiased sample estimator uses another denominator. Name convention. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 6: Signal power

Step 6 isolates signal power. Square the mean gradient. The reproducible result is: Signal²=4. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: A near-zero signal makes the ratio unstable. Record value. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 7: Noise scale

Step 7 isolates noise scale. Divide eight by four. The reproducible result is: B_noise=2. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: This is a scalar one-dimensional example. Record statistic. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 8: Batch variance

Step 8 isolates batch variance. Use Var(mean)=8/B for replacement sampling. The reproducible result is: Noise falls inversely with B. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Without-replacement correction differs. State sampling. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 9: Batch one

Step 9 isolates batch one. Set B=1. The reproducible result is: Normalized MSE=2. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: A single example is noisier than the signal. Record ratio. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 10: Batch two

Step 10 isolates batch two. Set B=2. The reproducible result is: Normalized MSE=1. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Noise and squared signal are comparable. Record ratio. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 11: Batch eight

Step 11 isolates batch eight. Set B=8. The reproducible result is: Normalized MSE=.25. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: This assumes repeated sampling from the population distribution. Record ratio. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 12: Batch sixteen

Step 12 isolates batch sixteen. Set B=16. The reproducible result is: Normalized MSE=.125. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Doubling still halves noise but may not halve wall time. Record ratio. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 13: Checkpoint sweep

Step 13 isolates checkpoint sweep. Estimate during training. The reproducible result is: Noise scale can change as loss changes. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: One initial value is not a schedule. Plot trajectory. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 14: Vector form

Step 14 isolates vector form. Use trace covariance over squared mean norm. The reproducible result is: Multiple parameter directions contribute. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Parameter scaling changes the scalar summary. Report formula. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 15: Parallel test

Step 15 isolates parallel test. Run several batch sizes with tuned learning rates. The reproducible result is: Predicted and measured speedups can be compared. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Hardware and optimizer confound timing. Record compute. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Evidence notebook 16: Claim

Step 16 isolates claim. State model, loss, checkpoint and sampling. The reproducible result is: The teaching population has gradient noise scale two. This is evidence about signal-normalized gradient variation. Preserve example, parameter and checkpoint identities beside every value. The safeguard is: Do not declare a universal optimal batch. Bound conclusion. Change one input while holding the objective, sampling rule, batch, step and estimator fixed. Recalculate every mean, covariance, eigenvalue, contraction factor or bound that must move. Explain whether the difference came from data, curvature, noise, approximation or systems behavior. Finish with the narrow conclusion and the additional evidence still needed.

Failure cases and responsible recovery

Batch-mean centring

Centring each batch around itself erases the population signal needed by the ratio. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Near-zero division

Noise scale explodes when the mean gradient is almost zero. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Loss-scale omission

Rescaling the loss and preprocessing can change reported components. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Sampling mismatch

Without-replacement and weighted sampling need different variance formulas. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Checkpoint freezing

One value cannot describe an entire training run. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Hardware shortcut

Statistical batch efficiency is not wall-clock throughput. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Learning-rate neglect

Batch comparisons require defensible optimizer tuning. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.

Generalization promise

Noise scale does not guarantee held-out quality. Rebuild a counterexample, repeat under another defensible setting and state the extra evidence needed to repair the conclusion.