Batch Normalisation Explained: Running Stats and Train-Eval Mode

Did you know that the same neural network can return different predictions for the same input depending on whether it is in training or evaluation mode? Batch normalisation is one reason. During ordinary training it uses statistics of the current minibatch and updates running estimates; during evaluation, when those estimates are enabled, it uses the stored values instead. Forgetting that distinction can make an apparently accurate model unreliable after deployment.

Batch normalisation transforms intermediate activations by subtracting a mean, dividing by a standard deviation and then applying learned scale and shift parameters. It can help optimisation in appropriate architectures, but its behaviour depends on batch composition, model mode, statistics and implementation. The dependable method is to know which values are used at each stage, solve the arithmetic, test small-batch edge cases and validate the model in the same mode in which real users will receive predictions.

What batch normalisation changes

An artificial neural network carries numbers through layers. Those intermediate numbers are called activations. Their magnitude and distribution can change while training adjusts earlier weights. Batch normalisation introduces an explicit transformation at a chosen layer, usually treating each feature or channel according to statistics calculated across a designated set of batch elements.

At its simplest, take one activation x, subtract the appropriate mean μ, and divide by the square root of variance σ² plus a small positive epsilon. That creates a normalised value. The layer may then multiply the result by a learned parameter gamma and add a learned parameter beta. Gamma and beta allow the network to choose a useful scale and offset rather than being forced to keep every channel at mean zero and unit variance in its final output.

This is not the same as fitting a preprocessing scaler to input columns once using the training dataset. Batch normalisation acts within the neural network, can use minibatch statistics during training and maintains its own inference-time behaviour. Input feature scaling addresses raw data geometry; batch normalisation addresses layer activations under a particular model design. The related feature scaling guide covers a distinct problem.

Do not interpret the word “normalisation” as a guarantee of a Gaussian distribution. Subtracting a mean and dividing by a standard deviation standardises moments; it does not force an arbitrary distribution to become bell-shaped. Heavy tails, outliers and multimodal activations can remain.

The exact forward-pass equation

For a channel with observations x₁ through xₘ, the batch mean is the sum of those observations divided by m. The training forward-pass variance commonly divides the sum of squared deviations by m. The normalised activation is (x minus mean) divided by the square root of variance plus epsilon. An affine-enabled batch norm layer returns gamma times that normalised value plus beta.

The epsilon term prevents division by zero when variation is tiny, though it does not solve every floating-point issue. The values of epsilon and the scope of the observations must be known before comparing implementations. For a two-dimensional convolutional activation tensor, a channel’s statistics typically aggregate over batch and spatial positions rather than just across images. A vector-valued feature layer uses a different shape but the same basic idea.

Gamma and beta are learned parameters and ordinarily receive gradients. Running means and running variances are buffers updated by a specified rule, not the same kind of trainable parameters. Confusing these categories leads to incomplete checkpoints and mistaken assumptions about which values an optimiser updates.

A useful first-principles question is: if the input activation doubles, what changes in the normalised output? The answer depends on whether the entire distribution changes, whether a single observation changes relative to its batch companions, and whether evaluation statistics are fixed. There is no general rule that simply doubles or halves the output.

Fully solved four-value calculation

Consider a fictional layer whose one-channel minibatch activations are 2, 4, 6 and 8. The mean is (2+4+6+8)/4 = 5. The deviations from the mean are −3, −1, 1 and 3. Their squares are 9, 1, 1 and 9, which sum to 20. The training forward-pass population-form variance is 20/4 = 5. Ignoring epsilon only for this hand calculation, the standard deviation is square root of 5, approximately 2.23607.

The normalised values are −3/√5 ≈ −1.34164, −1/√5 ≈ −0.44721, 1/√5 ≈ 0.44721 and 3/√5 ≈ 1.34164. They sum to zero up to rounding. Their mean squared value is one under the chosen population denominator.

Now suppose learned gamma equals 2 and beta equals 1. The transformed values are approximately −1.68328, 0.10558, 1.89442 and 3.68328. For the original activation 2, the result is 2 × (−1.34164) + 1 = −1.68328. This illustrates that the trainable affine stage deliberately allows a mean different from zero.

A real implementation uses epsilon inside the square root and follows its own specified arithmetic, so small numerical deviations from these illustrative values are expected. The exercise is not a claim that any model was trained; it establishes a checkable test fixture. If a program yields drastically different outputs, investigate its axes, mode, epsilon, gamma, beta and input shape before blaming the optimiser.

The subtle variance denominator

There are two common variance estimators. One divides the sum of squared deviations by m; the other divides by m−1 when m is greater than one. The second is often called the unbiased sample variance under familiar assumptions. These are not numerically identical on a small batch.

For the four observations 2, 4, 6 and 8, the squared-deviation sum is 20. Dividing by 4 gives 5; dividing by 3 gives 20/3, approximately 6.66667. The difference is large enough to matter in hand tests, especially for small batches.

PyTorch’s BatchNorm2d documentation specifies an important distinction: its training forward pass calculates variance with the biased estimator, equivalent to correction zero, while the variance value contributed to the moving average uses the unbiased estimator, equivalent to correction one. A test that incorrectly assumes the same denominator in both places can flag a correct implementation as inconsistent.

This is one reason to cite actual framework documentation rather than explain “batch norm takes the standard deviation” and leave the crucial denominator unstated. Implementations and numerical kernels evolve, so tests must be tied to the library version used. The reader should be able to reproduce the exact variance calculation in the context of their own chosen API.

Training mode and evaluation mode

In typical batch norm training mode with running statistics enabled, the current minibatch determines the normalisation statistics for that forward pass. At the same time, the layer updates running estimates that will later serve evaluation. In typical evaluation mode, it uses those stored running estimates rather than recomputing them from the particular inference minibatch.

This difference is intentional. During training, batches provide the statistics available at that moment and regularise the computation in a certain sense through batch dependence. At deployment, users may send one input at a time, and predictions should generally not depend on which unrelated examples happen to share the inference batch. Fixed running estimates provide a more stable reference.

Calling model.eval() in PyTorch changes the mode-sensitive behaviour of modules such as batch norm and dropout. It is not identical to disabling gradient recording with a no-gradient context. One controls module behaviour, the other controls autograd bookkeeping. A model can be in training mode inside a no-gradient evaluation loop and still update its running batch-norm statistics if the layer is configured that way.

This matters for validation. Running a validation set in training mode can make scores depend on how examples were batched and can accidentally alter statistics with data that were meant only for evaluation. Proper evaluation needs an explicit, tested mode policy and should not quietly rewrite model state.

The running mean is an exponential average

PyTorch documents a running-statistic update rule of new estimate = (1−momentum) times old estimate + momentum times the current batch statistic. With momentum 0.1, each new batch contributes one-tenth of the update while nine-tenths comes from the previous running estimate. This “momentum” is a smoothing parameter and should not be confused with optimiser momentum, which refers to a different process.

Take an illustrative running mean initialised to zero. The first minibatch has mean 3, so the updated running mean becomes 0.9×0 + 0.1×3 = 0.3. The next minibatch has mean 7, giving 0.9×0.3 + 0.1×7 = 0.27+0.7 = 0.97. The running mean after these two batches is 0.97, not the arithmetic mean of the two batch means, which would be 5.

This is not a paradox. An exponential moving average retains its initialisation and gives recent observations particular weights. After many representative batches, the effect of the starting value diminishes. But at the beginning of training, after a dataset shift, or with a tiny number of batches, stored estimates can lag behind the current activation distribution.

The documented PyTorch argument momentum=None instead selects a cumulative moving average for the running statistics. This is a materially different update rule. Choose and record the configuration according to the model’s data stream and validation behaviour rather than interpreting “momentum” as an optimisation switch.

A solved running-variance calculation

Use two illustrative minibatches with one-channel values (2, 4) and (6, 8). The first mean is 3 and the second mean is 7. Each batch has deviations −1 and 1, so its population-form forward variance is (1+1)/2 = 1. Its unbiased sample variance is (1+1)/(2−1) = 2.

Suppose the framework follows the PyTorch convention just described, with running variance initialised to 1 and momentum 0.1. After the first batch, the running variance becomes 0.9×1 + 0.1×2 = 1.1. After the second batch it becomes 0.9×1.1 + 0.1×2 = 0.99 + 0.2 = 1.19.

Meanwhile, the running mean progresses from zero to 0.3 and then to 0.97. In evaluation mode, using illustrative gamma 1, beta 0 and epsilon negligible for hand arithmetic, an activation of 6 is normalised as (6−0.97)/√1.19, about 4.61. In training mode on the batch (6,8), the same activation is normalised as (6−7)/√1 = −1. The outputs are drastically different at this early stage.

This is not evidence of a defective layer. It shows why immature running statistics and the choice of mode can dominate small numerical examples. With enough representative training batches, the running buffers may become more useful; whether they do so for a specific application is an empirical question requiring validation.

What does batch size mean here?

A batch of 32 images is not always a sample size of 32 for every batch-norm statistic. In a two-dimensional convolutional layer with tensor shape (N,C,H,W), each channel aggregates over N×H×W positions. If N=4, H=8 and W=8, there are 4×8×8 = 256 spatial-batch observations per channel under the usual operator semantics. The observations are correlated in real images, so 256 is a count of values, not necessarily 256 independent examples.

For a dense feature layer using BatchNorm1d on an (N,C) tensor, the corresponding count may be the number of examples N. With N=1 in training, that feature-wise variance cannot be estimated in the same useful way, and common implementations may reject the input. The exact supported shapes and constraints must be checked in the framework.

This distinction matters when someone says “batch size one makes batch norm impossible”. For convolutional batch norm, an N of one can still leave many spatial positions contributing to a channel statistic, depending on shape. Other modes or operations may still fail when the available aggregation dimension has too few values. Avoid absolute rules that erase the operator’s shape semantics.

A practical diagnostic prints tensor shapes immediately before each normalisation layer in a controlled test. Many apparent learning failures are simply mistaken expectations about which axis is normalised.

Gradient accumulation does not create one batch

Suppose a trainer processes four microbatches of eight examples and accumulates gradients before one optimiser step. Its effective gradient batch may contain 32 examples, but a conventional batch-normalisation layer calculates training statistics separately inside each forward pass of eight examples. It does not automatically pool the activations across all four passes simply because the gradients are accumulated.

That means changing a training setup from one forward pass of 32 examples to four forwards of eight can change batch-norm outputs and running-statistic updates even when the loss aggregation and effective optimiser step are designed to match. The two training procedures are not mathematically identical.

For a toy illustration, the first microbatch contains low-valued activations while the fourth contains high-valued ones. Each forward normalises relative to its own local mean. A single combined forward might use a much broader mean and variance. Repeating four running-buffer updates also differs from one update using combined statistics.

The adjacent gradient clipping guide explains another accumulation-sensitive boundary. It is important not to conflate the two: gradient clipping should ordinarily act after accumulating intended gradients, whereas standard batch norm operates during each forward pass. If your model requires different statistical semantics, choose an appropriate architecture or supported synchronisation strategy.

Small batches and unstable statistics

A small minibatch provides fewer values from which to estimate a mean and variance, at least in feature-wise normalisation. If those values differ strongly from one batch to the next, training outputs can fluctuate in ways that are unhelpful for the particular task. Running estimates may also converge slowly or be sensitive to ordering.

Consider two minibatches with means 0 and 100. With momentum 0.1 and initial running mean zero, the first leaves the running mean at zero and the second moves it to 10. If evaluation data are centred nearer 50, that early buffer is a poor reference. This problem is not repaired by setting a smaller learning rate for ordinary trainable weights alone; the running statistics have their own update rule.

Possible remedies include using a larger batch when feasible, changing how data are batched, collecting representative running statistics, freezing appropriate layers, adopting synchronised batch statistics where supported or replacing batch norm with group or layer normalisation when suitable. None is a universal improvement. Every change alters the model and should be judged by controlled held-out results.

Outliers deserve special attention. A single extreme activation can shift mean and variance, affecting other members of the batch. Robust data handling and architecture checks may be more appropriate than arbitrary manipulation of running buffers. First diagnose whether the strange activation represents real task variation or a preprocessing or numerical error.

Per-device batches in distributed training

Data-parallel training commonly gives each accelerator a portion of the overall data batch. Standard batch normalisation usually computes statistics from the examples local to the forward pass on that device, unless a synchronising variant is deliberately used. A global nominal batch of 128 across eight workers can therefore produce eight separate local groups of 16 for normalisation purposes.

A synchronised batch-normalisation operator can collect relevant statistics across a specified process group during training. PyTorch documents this behaviour in SyncBatchNorm, including mode and distributed-environment conditions. It introduces communication costs and deployment constraints, so it is not a free switch.

For example, imagine worker A sees values centred near zero while worker B sees values centred near ten. Local batch norm centres each group around its own mean. Synchronized statistics would use information from both groups and yield different normalised values. Which representation helps depends on task, data assignment and architecture.

Reproducible comparisons must record the number of devices, local batch size, synchronisation policy and relevant process groups. A training report saying only “batch size 64” may omit exactly the detail responsible for different behaviour across runs. In deployment, check the chosen variant’s evaluation semantics rather than assuming the distributed training arrangement persists.

Evaluation should not mutate hidden buffers

A holdout set should inform model assessment without silently modifying the model under evaluation. If a validation loop leaves the network in training mode, ordinary batch norm may update running buffers as validation examples pass through. The experiment has then allowed validation data to change the model state, contaminating the intended independence of the assessment.

Use an explicit mode transition before evaluation and restore the desired training mode afterwards. Verify which submodules are affected, especially in composite models where some components were intentionally frozen. Measure prediction invariance across different inference batch compositions when the application expects single-input stability.

There is an important exception to blanket claims: when track_running_stats is false, PyTorch documents that its batch-normalisation layer uses batch statistics during evaluation as well. In that configuration, merely calling eval does not create fixed running statistics that do not exist. This design can make the output depend on inference companions and should be a conscious decision.

A strong test compares an input scored alone, where supported, and alongside different valid companions. With fixed inference-time running stats and no other batch-dependent operations, the batch norm contribution should not change solely because a neighbour changed. If it does, inspect the mode, tracking configuration and other model components. Do not assign all batch sensitivity to batch norm without isolating it.

Freezing weights is not freezing statistics

Practitioners sometimes disable gradients for part of a pretrained model by setting a parameter’s requires_grad flag to false. That prevents ordinary gradient updates to that parameter but does not by itself stop mode-sensitive running statistics from being updated during a training-mode forward pass. The buffers belong to the module’s behaviour, not just its optimiser parameter list.

This distinction matters in transfer learning. A frozen image encoder might have carefully calibrated batch-norm buffers from a large pretraining dataset. If fine-tuning uses a tiny, unusual dataset and leaves those buffers updating, their values may drift even though the encoder’s trainable weights are held fixed. The model’s evaluation quality can change for a reason not visible in the optimiser’s update summary.

The correct policy depends on whether the new data are sufficiently representative, how far the task has shifted and whether re-estimating statistics is desirable. One may freeze batch norm behaviour in evaluation mode while fine-tuning other layers, or deliberately update the buffers under a planned calibration regime. Test the choice rather than copying a universal recipe.

Document separately which weights require gradients, which modules use training mode, whether running stats are tracked and how checkpoint buffers are handled. The phrase “freeze the encoder” is otherwise ambiguous.

Checkpoints need buffers as well as weights

A batch-norm layer commonly has learned affine weights and non-trainable running buffers. A checkpoint that saves only selected optimiser parameters but omits running means and variances may reconstruct a network that looks structurally correct while producing different inference outputs. The model’s state is larger than the subset of tensors a particular optimiser updates.

A faithful checkpoint should capture the full model state appropriate to the framework, plus training state needed for resumption. Record configuration such as epsilon, momentum, tracking behaviour and architecture version. If a model is exported to another runtime, verify that its inference operation implements compatible arithmetic and uses the intended stored values.

A basic recovery test saves the model, reloads it into a fresh process, sets the correct mode and compares predictions on fixed inputs under suitable tolerances. Do not rely solely on an equality check of trainable parameter tensors. Running-buffer mismatches can be invisible to that comparison.

It is also possible for a model to be intentionally recalibrated after training, using representative authorised data to update statistics. That process must be separated from untouched test evaluation and carefully documented. Otherwise the apparent performance of a “saved” model may depend on a hidden sequence of extra forward passes.

Error diagnosis: predictions depend on neighbours

Suppose a user submits the same image twice. Alone it receives score 0.8, but placed beside unrelated images it receives 0.55. First confirm that preprocessing, random augmentation, dropout and other model operations are controlled. Then inspect batch-norm tracking and model mode. If the model uses current-batch statistics at inference, the neighbours can change the mean and variance used for normalisation.

One possible repair is to use evaluation mode with valid stored running statistics where the architecture supports it. Another is to change the normalisation design to one less sensitive to batch composition. But simply hard-coding the first score or averaging multiple arbitrary batch arrangements is not a principled correction.

If a model that formerly worked begins to fail after a checkpoint change, compare saved running buffers and module configuration. If it fails only when batch size becomes one, examine the exact layer type and aggregation dimension. If it fails after moving from one device to multiple devices, examine local versus synchronised batch statistics.

Debug by creating the smallest case that reproduces the discrepancy: a fixed input, controlled companion set, identical weights and logged mode configuration. This is more informative than vaguely attributing inconsistency to “AI randomness”. Batch normalisation can be deterministic for given state and inputs while still being batch-dependent under specific settings.

A changed-context transfer problem

Imagine a model trained on photographs of documents where a BatchNorm2d layer saw 32 images per worker. The team now wants to deploy the same architecture to single-frame microscope images with different brightness distributions, and some normalisation layers receive tiny spatial maps.

Can the old training recipe be copied unchanged? Not responsibly. The source distribution and inference batch differ, and deeper layers may have too few values for stable batch estimates. The engineer must identify which channels and dimensions each layer normalises, inspect running statistics, test one-image inference and decide whether to retain, recalibrate or replace the normalisation design.

Consider a toy channel with training running mean 20 and variance 25, gamma 1 and beta 0. An inference activation of 30 is mapped, ignoring epsilon, to (30−20)/5 = 2. If a new imaging system adds a constant brightness offset of 10 to that activation while the stored statistics stay unchanged, the value 40 maps to (40−20)/5 = 4. Batch normalisation with fixed running stats does not automatically cancel an unseen deployment shift.

The correct transfer solution includes representative validation on the new domain, controlled normalisation alternatives and independent final testing. It may also require revisiting input preprocessing or lighting controls. A useful conceptual rule survives the change: know which population your statistics describe and do not assume the deployment population is identical.

An evaluation plan that can fail honestly

A good experiment compares a baseline model and the candidate batch-normalised model under matched data splits, budgets and evaluation criteria. It records model architecture, normalisation placement, batch size per forward, optimiser settings, training duration, random seeds where applicable and the selection of validation versus test data.

Test ordinary deployment scenarios, not only high-throughput laboratory batches. If users send one input, measure one-input predictions. If the model serves groups of documents, vary companion composition and order to detect unexpected cross-example coupling. Include small and outlier-heavy batches and check performance across important subgroups or operating conditions.

Measure accuracy or task-specific loss, latency, memory, calibration and stability rather than assuming that internal normalisation always improves outcomes. A layer can make optimisation easier for one architecture and reduce quality or complicate deployment for another. Report unsuccessful comparisons too when they constrain where the method is appropriate.

Keep test sets independent of model selection. Tuning normalisation settings repeatedly against the final test set creates an optimistic assessment even if the network itself is mathematically correct. The data leakage and validation guide explains the broader evidence issue. Correct normalisation is necessary for certain architectures, but it is not a substitute for honest evaluation.

Practical checks and numerical guardrails

For unit testing, reproduce the (2,4,6,8) example and assert the batch mean is 5, population-form variance is 5 and affine-transformed output agrees within a justified tolerance. Check epsilon is added inside the square root. Confirm the selected channel and spatial axes are correct. A wrong axis can yield numerically valid but semantically incorrect outputs.

Next verify that running means and variances update according to the framework’s documented convention. The two-batch example should distinguish the training forward variance from the variance contributed to a moving average. Test that a training-mode forward changes buffers when tracking is enabled and an evaluation-mode forward does not, except where the chosen configuration explicitly behaves differently.

Test checkpoint round trips, input shapes, small-batch failures, and models containing both batch norm and dropout. When using distributed training, test a controlled two-worker situation and compare local and synchronised policies. Before deployment, inspect the exported runtime’s operators and precision rules.

Monitor for unusually fast buffer changes or discrepancies between training and evaluation outputs. Avoid logging sensitive raw activations from individual users; aggregate safe diagnostics where possible. A good alarm describes a concrete numerical invariant that failed rather than declaring the model mysteriously unstable.

Frequently asked questions

Does batch normalisation make every model more accurate? No. It can help optimisation in appropriate networks, but the benefit depends on architecture, data, batch size and evaluation design.

Is batch normalisation the same as scaling raw features? No. A fitted input scaler uses statistics of training data columns, while batch norm usually operates on intermediate layer activations with mode-dependent behaviour.

Why can training and evaluation predictions differ? Ordinary batch norm uses current minibatch statistics in training and stored running statistics in evaluation when tracking is enabled. Dropout and other mode-sensitive operations can also contribute.

Does gradient accumulation increase batch norm’s statistical batch size? Not automatically. Separate forward passes generally calculate separate batch statistics, even when their gradients are combined before one optimiser step.

Can I call no_grad instead of eval? No. These control different aspects of a framework. No-gradient execution does not by itself change a module’s training or evaluation mode.

Should I always freeze running statistics for transfer learning? No universal rule applies. Compare frozen and carefully updated statistics on representative validation data, then test deployment behaviour independently.

Primary references and reader pathways

For exact implementation details, see PyTorch BatchNorm2d for the forward equation, variance conventions and running estimates; PyTorch SyncBatchNorm for distributed statistics; and PyTorch functional batch_norm for the lower-level operator. Consult the version actually deployed.

Readers working through model optimisation can connect this to gradient clipping and stable updates, feature scaling and outliers and the broader How AI Works guide. These tackle separate but interacting mechanisms.

Students can use the solved examples to understand mean, variance, standard deviation, weighted averages and the difference between sample statistics and estimates used later. The education learning-transfer article links mathematical reasoning to applied problems beyond an examination.

The central safeguard is to ask three questions whenever predictions look inconsistent: which observations defined the statistics, which mode selected those statistics, and which saved state was used? Answer those precisely before changing an optimiser or blaming the data. Small, independently checkable arithmetic is often the fastest route to diagnosing a surprisingly complex AI failure.

How AI Works | Batch Normalisation — Running Statistics, Train-Eval Mode and Small Batches

Contents