Sharpness-Aware Minimization (SAM) is a training method that asks a neural network to perform acceptably not only at its current weights, but also after a small adverse change to those weights. Instead of following the ordinary gradient immediately, it first looks uphill in a specified parameter neighbourhood, measures the gradient at that perturbed location and then updates the original weights. In its common first-order form this takes two gradient evaluations per optimiser update.
This matters because two networks can fit the same training examples and still behave differently on unseen data. SAM seeks a region of parameter space whose loss does not rise sharply under a small weight perturbation; it is an optimisation preference, not a mathematical guarantee of higher test accuracy. The radius, perturbation norm, mini-batch choices, parameter scaling, stochastic model state and evaluation budget determine what the method actually does.
This guide derives the algorithm, checks a complete two-parameter example, explains exactly what the two passes must preserve, separates SAM from gradient clipping and adversarial input training, and shows how to evaluate genuine held-out improvements. Figures describing invented training runs are clearly labelled hypothetical, not claimed measurements of an eduKate model. The foundational primary account is Foret and colleagues’ 2021 ICLR paper, available from Google Research and the authors’ public implementation.
Contents: choose your reading route
Begin with the generalisation problem, parameter perturbations, the minimax objective, the derivation and the exact update order.
For a worked lesson, follow the original two-weight example, its verified update, zero-gradient branches and an independent changed-radius question with its answer.
For an implementation, continue with same-batch replay, optimiser state, mutable model buffers, rho selection, coordinate dependence, ASAM, clipping and AMP.
For an experiment or decision, read distributed training, compute denominators, held-out evaluation, evidence limits, diagnosis and repairs, when to use SAM, the reproduction ledger, FAQs, primary sources and verified site routes.
Why a low training loss is an incomplete description of learning
Imagine two hypothetical classifiers that both label all 2,000 training examples correctly. One uses weights near a narrow valley: moving one weight by a small amount makes many predicted labels change. The other occupies a region where modest changes leave the predictions mostly stable. Their training accuracy is identical, but the loss landscape around the weights differs. That observation motivates a separate question about local stability; it does not yet prove which classifier will generalise better.
Ordinary empirical risk minimisation chooses parameters that make an average loss on the training sample small. This is a useful and often effective objective. It does not explicitly ask what would happen to the same training loss if model weights were perturbed within a radius. Very flexible networks can obtain low training loss in numerous parameter configurations, and those configurations need not perform alike on new cases.
SAM changes the search preference rather than the task label. It makes the optimiser attend to a small vicinity of each candidate parameter vector. A parameter setting with a slightly worse loss at its exact centre may be preferred if its nearby worst-case loss is smaller. The point is not that all broad regions are good, but that controlling nearby loss is one potentially valuable additional signal in selecting a trained model.
A reader should separate four objects at the beginning: the fixed training dataset, the per-example loss, the current model weights, and the geometry used to define a neighbourhood around those weights. Only when those are specified can one say what SAM is optimising. A map labelled ‘flat minimum’ without its parameter units, loss definition and radius is more of an illustration than a reproducible scientific claim.
The perturbation lives in parameter space, not in the input
An adversarial image example changes a pixel array while the model parameters remain fixed. SAM does almost the reverse: it changes candidate model parameters in an inner optimisation step while usually using the same mini-batch inputs and targets. Both are often written as min–max expressions, yet their threat models and objectives are different. Mixing them up leads to incorrect code and misleading descriptions of robustness.
For a weight vector w with d coordinates, define a perturbation vector e of the same dimension. A common SAM neighbourhood is all e satisfying the Euclidean norm ||e||₂ no greater than rho. Rho is a radius measured in the model’s weight-coordinate units, not pixels, percentage points of accuracy or seconds of training. A different norm or reparameterisation changes the exact neighbourhood and therefore changes the problem.
The method does not choose arbitrary bad predictions and call those the target. It searches for a nearby point producing relatively high training loss and then updates w to make that vicinity less damaging. The perturbation is temporary: it helps construct the next training gradient but is not simply accepted as the permanently updated model. This restoration step is essential in a correct implementation.
One practical diagnostic is to log the original and perturbed parameter norms separately. If the implementation’s parameter tensor remains permanently displaced before the base optimiser applies its step, it is not following the intended common SAM sequence. Likewise, if an engineer perturbs input pixels and computes a normal gradient, that may be an adversarial training variant, but it is not this parameter-neighbourhood algorithm.
The exact neighbourhood objective and what is only approximate
Write L_B(w) for average training loss on a selected mini-batch B. The local worst-case loss is F_B(w) = max over ||e||₂ ≤ rho of L_B(w + e). A full training ideal would minimise the corresponding sharpness-aware risk, with any separately specified regulariser. The quantity F_B(w) is a maximum, not an arithmetic mean across neighbourhood points, and it is not automatically the evaluation loss on unseen data.
Define local sharpness for this discussion as F_B(w) − L_B(w). That difference is nonnegative when e = 0 is permitted, because the maximisation includes the centre. A perfectly low sharpness difference does not guarantee low loss at the centre; a constant high-loss plateau is flat but still bad. SAM is usefully understood as searching for a low worst-case loss, rather than pursuing flatness while ignoring predictive performance.
Solving the inner maximisation exactly for a large neural network would usually be expensive. The common practical SAM approximation uses a local first-order ascent direction instead of repeatedly optimising e until the true maximum is found. The second gradient therefore corresponds to a deliberately chosen nearby point, not an independently certified solution to the inner max. A description that calls e the mathematically exact worst possible perturbation is too strong.
The neighbourhood also depends on which model parameters are included. A programmer might exclude frozen layers, buffers, nontrainable variables or parameters with no active gradient. Whatever convention is used, it belongs in the experiment definition. A radius applied separately to each layer is not the same feasible set as one Euclidean radius over all selected trainable parameters.
Deriving the first pass without a mysterious formula
Near w, a differentiable loss can be approximated by L_B(w + e) ≈ L_B(w) + g · e, where g = ∇L_B(w). For a fixed radius rho and an L2 constraint, the Cauchy–Schwarz inequality gives g · e ≤ ||g||₂ ||e||₂ ≤ rho ||g||₂. The linearised loss is therefore increased most by using a perturbation parallel to g with full permitted norm.
When ||g||₂ is positive, the resulting approximation is e_hat = rho g / ||g||₂. The formula explains why SAM first computes a regular gradient even though that gradient is not immediately used for the final parameter update. It supplies the direction of the deliberately unfavourable, nearby evaluation point. The division normalises direction; rho controls its magnitude.
The derivation is local. It ignores higher-order curvature while choosing e_hat, so for a large radius or a strongly nonlinear loss landscape it may not locate the true worst-case point. The subsequent gradient at w + e_hat samples the loss more realistically than simply reusing the original g, but neither operation is a proof that all neighbouring losses have been minimised.
A subtle mathematical qualification concerns differentiation through e_hat. In the ideal exact-max problem, envelope-type reasoning can justify differentiating loss at a maximiser under suitable conditions. In practical two-pass SAM, the perturbation is approximated; the usual implementation treats the chosen perturbation as fixed for the second gradient. Automatically differentiating through the perturbation construction would produce a different algorithm, often requiring higher-order computation.
What the second pass does and why the original weights return
First compute g₁ = ∇L_B(w). Build e_hat using the chosen radius. Temporarily evaluate the same model’s parameters at w + e_hat and compute g₂ = ∇L_B(w + e_hat). Restore the original w; then pass g₂ to a base optimiser, such as SGD, to update that original w. In plain SGD notation, w_new = w − eta g₂, where eta is the learning rate.
Two separate gradients are not two permanent training steps. The uphill move is a probe that asks how the mini-batch loss changes near the current position. The downhill move changes the actual model. Updating the optimiser’s momentum during the uphill probe, advancing its learning-rate scheduler twice, or applying weight decay twice can silently convert one intended update into something else.
Keep four states conceptually separate: the original parameters, temporary perturbed parameters, optimiser state and model buffers. Parameters need restoration before the base step. Momentum and adaptive moments should normally progress once for a logical SAM update. Buffers such as batch-normalisation running statistics may need their own documented handling because a second forward pass can otherwise mutate them twice.
An implementation trace should distinguish forward/backward pass one, perturbation, forward/backward pass two, restoration and final optimiser step. This trace is a stronger debugging tool than a single training-loss curve. If training seems unexpectedly unstable, a short trace of actual parameter hashes and optimiser-step counters can establish whether the intended order occurred.
Worked numerical example: two weights and an exact loss
Consider a completely synthetic two-parameter quadratic loss L(x,y) = 0.5 x² + 2 y². It has gradient (x, 4y). Start at w = (1,1), choose rho = 0.1 in Euclidean parameter units, and choose a plain-SGD learning rate eta = 0.05. This is a deterministic arithmetic exercise, not a trained model or a prediction about classification accuracy.
At (1,1), the unperturbed gradient is g₁ = (1,4). Its global L2 norm is sqrt(1² + 4²) = sqrt(17) ≈ 4.12310563. The ascent perturbation is 0.1 times (1,4) divided by sqrt(17): approximately (0.02425356, 0.09701425). The perturbation’s squared norm is 0.01 by construction, so its norm is exactly 0.1 apart from display rounding.
The temporary weights become approximately (1.02425356, 1.09701425). The centre loss equals 0.5 × 1² + 2 × 1² = 2.5 loss units. The perturbed loss is approximately 2.93143 loss units, a rise of about 0.43143. First-order prediction is g₁ · e_hat = 0.1 sqrt(17) ≈ 0.41231; the remaining difference comes from the quadratic curvature ignored by the linear approximation.
This example makes the units explicit: x and y are dimensionless parameters in the toy model, rho has the same parameter-coordinate scale, eta converts the chosen gradient to a parameter step, and L is an arbitrary loss value. There is no denominator of student scores or GPU hours hidden in these calculations. The example’s exact expression lets a reader verify all rounded decimals independently.
Finish the toy calculation and compare ordinary SGD
The gradient at the perturbed point is g₂ = (1.02425356, 4 × 1.09701425) ≈ (1.02425356, 4.38805700). Restore w = (1,1) before the final update. With eta = 0.05, the common SAM-plus-SGD step gives w_new ≈ (1 − 0.05121268, 1 − 0.21940285) = (0.94878732, 0.78059715).
Ordinary SGD from the same starting point would use g₁ directly: w_SGD = (1 − 0.05, 1 − 0.20) = (0.95, 0.80). Both are valid one-step arithmetic results for the specified algorithms. They are not the same step, and neither can be described simply as ‘SAM adds 0.1 to every weight’. The SAM perturbation is norm constrained and follows the chosen gradient direction.
For interest, the ordinary-SGD result has toy loss 0.5 × 0.95² + 2 × 0.80² = 1.73125. The SAM result’s toy loss is approximately 1.66876. That lower value for this one carefully chosen quadratic is not a theorem that SAM always reduces current training loss more than SGD. At another point, with different curvature, optimiser or radius, the immediate ordering can reverse.
The transfer question is why no test accuracy is reported. This toy has no training/test split, labels or prediction function, so generalisation cannot even be defined from the supplied data. The example verifies the mechanics of perturbation, restoration and a one-step update. Any claim about improved generalisation would require a real unseen evaluation set and controlled comparison.
The zero-gradient and missing-gradient branches matter
The normalisation formula divides by ||g||₂. If all selected gradient components vanish exactly, naive code divides by zero. A stable implementation specifies an explicit branch: commonly it uses a zero perturbation or a carefully defined stabiliser while preserving the intended scale. An arbitrary epsilon added to the denominator changes the tiny-gradient regime, so its value and effect should be documented.
Different parameters may have absent gradients rather than exactly zero-valued gradients. A frozen layer is not necessarily part of the perturbation’s coordinate set. Sparse gradients may need a specific implementation; blindly flattening a sparse tensor as though it were dense can be wrong or unsupported. Verify that every parameter group has a clear ownership rule for the global norm and temporary perturbation.
An extremely small finite gradient can also be dominated by numerical precision effects. The normalised vector may still have length rho, sending a weight probe far relative to the signal’s reliable direction. Checking gradient finiteness and the chosen precision before perturbation is more useful than assuming that a small training loss guarantees a meaningful gradient.
One safe practical test is a constant loss that produces zero gradients, a model with a frozen first layer and a model where one branch is unused by the current mini-batch. The expected behaviour is not necessarily identical across libraries, but it must be deterministic, documented and free of NaN parameters. A successful standard SGD step on these fixtures is not a substitute for testing the extra SAM branch.
Why the two passes normally use the same mini-batch
In the common two-pass approximation, the first gradient defines a perturbation that is adverse for a selected mini-batch B, and the second gradient evaluates the loss at that perturbation for the same B. If the second pass silently uses unrelated data, it no longer directly evaluates the same local objective that selected e_hat. Such a variant might be studied deliberately, but it should not be presented as the ordinary method.
The term ‘same mini-batch’ includes more than a batch identifier. If a training pipeline chooses a random crop, different dropout mask or online data augmentation independently on each pass, the effective loss sampled in the two computations can differ. Implementations choose different practical strategies for handling such randomness. The crucial editorial distinction is to say what was replayed, what was resampled and which stochastic objective the experiment actually optimised.
Suppose batch B contains 64 distinct examples and the augmentation pipeline samples a crop for each image. If the second pass uses 64 new crops, the raw example identities match but the tensors differ. This may reduce consistency with the literal mini-batch inner-max derivation. Recording augmentation seeds or explicitly choosing a two-view variant turns an accidental implementation detail into a measured research decision.
Correct bookkeeping also means clearing or replacing first-pass gradients before the second backward calculation. If g₁ and g₂ are accumulated into the same parameter .grad arrays without an intended correction, the base optimiser sees g₁ + g₂ rather than g₂. That is a different update even if the loss printed after the second pass appears normal.
Momentum, Adam and weight decay are separate choices
The two-pass mechanism supplies a modified gradient to a chosen base optimiser. With SGD, the outer step is easy to write. With momentum or Adam, the outer update also depends on running state from prior iterations. The gradient at perturbed weights should feed the one intended outer update; the temporary perturbation itself should not normally increase Adam’s step counter or update momentum buffers as if it were another training iteration.
Decoupled weight decay is also an outer-update policy. Applying weight decay during the temporary ascent probe makes the probe incorporate extra movement not present in the simple SAM derivation. A regularisation term may be part of the chosen objective, but then the experiment should state that explicitly. The SAM neighbourhood term and parameter shrinkage are conceptually different and should not be added twice through convenience wrappers.
For an adaptive optimiser, the magnitude of the final weight change is not simply eta multiplied componentwise by g₂. Adam rescales directions according to its moving moment estimates and hyperparameters. A teaching example using the plain-SGD formula must not be misapplied as an exact numerical forecast for an AdamW implementation with existing moment state.
Inspect step counters, scheduler advancement and weight-decay calls in a one-batch dry run. A logical SAM update has two loss/gradient evaluations but normally one base-optimiser step. If the logged optimiser step counter advances twice for one consumed mini-batch, investigate before comparing learning curves or claiming a training-speed result.
Batch normalisation and other stateful layers
Batch normalisation has running statistics that can change during a training forward pass. A two-pass optimiser introduces a practical question: should both passes update those running estimates, should the first or second be excluded, or should a documented alternative be used? There is no safe shortcut in saying that restoring weight tensors automatically restores mutable non-parameter buffers.
Consider a running mean updated with momentum 0.1 after observing a batch mean of 10, starting from zero. One update gives 1. A second identical update gives 0.9 × 1 + 0.1 × 10 = 1.9. If the intended bookkeeping was one observation per logical training update, mutating the buffer twice changes it substantially. These numbers illustrate state updates, not a universal required batch-normalisation policy for SAM.
Dropout also contributes randomness, though often without a persistent running mean. If different masks are used across passes, g₂ may reflect both the weight perturbation and a changed random network realisation. Whether that is acceptable depends on the training definition and reproducibility goal. Record seeds and state-handling conventions rather than treating the method’s name as a complete recipe.
Other stateful systems deserve the same scrutiny: stochastic-depth decisions, exponential moving-average teacher parameters, quantisation observers, mutable feature memories and gradient caches may be updated on forward passes or callbacks. Define one intentional place in the loop for each state transition. A green validation accuracy figure does not diagnose corrupted internal state if the run is not reproducible.
Choosing rho is a real modelling decision
Rho sets the radius of the neighbourhood in the chosen parameter norm. At rho = 0, the first-order SAM perturbation vanishes and the algorithm reduces toward the base optimiser, aside from implementation overhead and stochastic-state differences. Increasing rho exposes the model to larger local excursions. That can push training toward less sharp regions, but an excessively large radius may make the first-order approximation poor or prevent the model from fitting useful patterns.
A radius of 0.05 is not 5 percent error, five basis points of accuracy or an answer that transfers unchanged between every architecture. It is a number in a coordinate system over selected parameter tensors. Different architectures, loss scalings and parameter groups may make the same numerical rho represent very different relative moves. Selecting rho therefore belongs to validation-based experimentation, not blind copying from a benchmark command.
Suppose a model’s gradient norm is 20 and rho is 0.05. The first-order estimated increase along the ascent direction is rho × ||g||₂ = 1 loss unit, while the perturbation’s parameter L2 norm is 0.05. Double the radius to 0.10 with the same gradient and the linear estimate becomes 2 loss units. Curvature may change the actual increase; neither figure is a prediction of held-out test error.
Choose a small predeclared set of radii on a validation protocol and compare across repeated runs. Keep the batch definition, training budget and model selection rule clear. A search over dozens of radii evaluated on the test set is test leakage, even when the optimiser itself never consumes test examples during gradient calculation.
Why visual flatness can be a coordinate illusion
Raw weight-space sharpness is sensitive to how parameters are represented. In a simple two-layer linear chain, an output can depend on the product ab. Replacing a with 10a and b with b/10 leaves the product, and therefore the function, unchanged. But a fixed Euclidean-radius neighbourhood around the new parameter pair does not describe the same set of function changes as a radius around the original pair.
That example matters whenever an article equates a flatter-looking loss plot with an inherently better predictor. Two parameterisations can implement precisely the same predictions yet have different coordinate-based curvature and local sharpness. Normalisation layers and positively homogeneous activations create further rescaling relationships. The selection of norm and scale should be explicit.
For a concrete calculation, take a = 2 and b = 3, yielding product 6. Rescale to a’ = 20 and b’ = 0.3, also yielding 6. Perturb b by +0.01 at the first parameterisation and the product changes by 0.02; perturb b’ by +0.01 after rescaling and it changes by 0.20. The same coordinate perturbation has ten times the change in this illustrative output product.
The calculation does not refute every benefit reported for SAM; it limits what a raw geometric slogan can prove. A practical comparison should report whether its optimisation is based on a fixed unweighted norm, a layerwise norm, an adaptive scale or another neighbourhood. It should evaluate actual generalisation instead of declaring victory from a single unnormalised loss-surface picture.
How adaptive sharpness-aware minimisation changes the question
ASAM, introduced by Kwon and colleagues in a 2021 ICML paper, targets the parameter-rescaling weakness of a rigid fixed-radius neighbourhood. It defines an adaptive notion of sharpness and associated optimisation method rather than assuming that an identical absolute perturbation in every parameter dimension has an identical functional meaning. It is a related method, not merely another spelling of SAM.
The authors describe a scale-aware construction and theoretical analysis in their original paper. A practitioner should consult the precise implementation choices, including the normalisation transform, when attempting to reproduce ASAM. It would be incorrect to take the ordinary perturbation e_hat = rho g / ||g||₂ from this article, change the title to ASAM and declare the result equivalent.
An instructive decision is whether the model has substantial parameter-rescaling freedom or widely different magnitudes across layers. In that case a fixed global radius might be especially hard to interpret. ASAM is one researched alternative, though the real choice also depends on measured stability, software support, training budget and out-of-sample performance.
For reproducibility, report the actual optimiser variant, its coordinate weighting, epsilon choices, treatment of zero entries and whether a supplied library matches the cited algorithm. An implementation that is partly SAM and partly an adaptive rule should be named and specified as such, not credited with results from either original paper without running a controlled experiment.
SAM is not gradient clipping, and the order matters
Gradient clipping constrains a gradient’s magnitude before applying an update, often to stop an extremely large step. SAM deliberately uses a gradient direction to select a nearby point that has higher loss, then takes a gradient there. A global norm appears in both methods, but it serves different purposes. One controls update magnitude; the other controls the length of a temporary weight-space probe.
For example, take a gradient (3,4) with norm 5. Norm clipping to threshold 2 scales it to (1.2,1.6) for the outer update. SAM with radius 0.1 uses that same original direction to create a temporary perturbation of (0.06,0.08), whose norm is 0.1; it then computes a new gradient at the perturbed weights. Substituting the clipped vector as the final SAM gradient skips the second evaluation and changes the algorithm.
If both clipping and SAM are used, decide whether clipping applies to the first gradient before constructing the perturbation, to the final outer gradient, or according to another explicitly tested policy. Pure norm scaling before perturbation may leave the direction unchanged under simple conditions, yet zero thresholds, adaptive groups and nonlinear transforms can still matter. The first and final gradients have different roles and should be logged separately.
The existing eduKate explanation How AI Works | Gradient Clipping covers the clipping operator itself. The present article instead owns the SAM two-evaluation optimisation job. Read the two together when a real training loop contains both controls.
Mixed precision: avoid normalising a scaled gradient
Automatic mixed precision may scale a loss before backward computation so small floating-point gradients do not underflow. The gradient buffers then contain a scaled version of the intended values until they are unscaled. A perturbation based on the wrong scaled norm can interact badly with overflow handling, parameter-group logic and stabilisers. Inspect the mixed-precision library’s supported sequence rather than embedding the raw SAM formula in a generic callback.
PyTorch’s official AMP examples explain that gradients created by a scaled backward call should be unscaled before a user examines or alters their magnitude. They also warn that unscale operations have placement and repetition constraints. This applies directly to calculating a meaningful gradient norm and to final norm clipping, although a complete two-pass SAM loop requires additional care beyond the simple single-pass example.
Suppose a hypothetical gradient (3,4) is multiplied by scale 128, yielding (384,512) with norm 640. The physical gradient’s norm remains 5. If a routine merely normalises both vectors, the direction is identical in exact arithmetic; but adding an unscaled stabiliser, clipping before unscaling or encountering overflow changes behaviour. Computing the norm from the intended unscaled, finite gradient avoids that ambiguity.
Do not blindly call an AMP helper twice in one optimiser-step interval if the framework permits it only once. Design and test a library-compatible pair of backward passes, overflow detection, restoration and final update. When an overflow makes an update invalid, ensure temporary perturbations are restored even if the optimiser step is skipped. Unit tests should cover that recovery path deliberately.
Data parallelism and gradient accumulation change the bookkeeping
In distributed training, several workers may each hold a replica of the same model but receive different mini-batches. A global SAM perturbation based on the aggregated batch gradient can differ from local perturbations computed separately on each worker. These are distinct algorithmic choices with different approximate objectives, communication needs and potentially different results. The original SAM paper also discusses sharpness measured on subsets rather than only one full global sample.
Consider two workers, one computing gradient (1,0) and the other (0,3). Their mean is (0.5,1.5), whose norm is sqrt(2.5) ≈ 1.5811. For rho = 0.1, a perturbation from that mean is approximately (0.03162,0.09487). Independently normalised local perturbations would instead be (0.1,0) and (0,0.1). Averaging those local perturbations gives (0.05,0.05), not the globally normalised perturbation.
Gradient accumulation similarly changes what constitutes one logical batch. If four micro-batches of 16 examples form an effective batch of 64, the intended first-pass gradient may need accumulation over all 64 before the perturbation. The second pass must reflect the corresponding data definition. Applying the perturbation after each micro-batch produces four local probes instead of one effective-batch probe.
Record whether communication synchronises first-pass gradients, second-pass gradients or both; whether all workers share the same radius and effective parameter norm; and whether dropout and batch-normalisation states are controlled coherently. Two runs may both be labelled SAM while implementing different distributed variants. Without those details, a performance comparison is difficult to reproduce or interpret.
Count actual work rather than saying twice as slow
The common SAM algorithm uses roughly two forward/backward gradient evaluations per base-optimiser update instead of one. This establishes an arithmetic relationship in gradient passes, not a universal twofold wall-clock slowdown. Data loading, all-reduce communication, activation recomputation, checkpointing, accelerator utilisation and caching also contribute to elapsed time and energy.
For a deliberately hypothetical budget, take 12,000 optimiser updates. Ordinary single-pass training evaluates 12,000 batch gradients. A straightforward two-pass SAM run evaluates approximately 24,000 gradients for those same 12,000 logical updates. If each pass takes 0.4 seconds and overhead is zero by assumption, the gradient-pass time would be 4,800 seconds versus 9,600 seconds. Real overheads and saved operations invalidate a rigid elapsed-time prediction.
An alternative fair question holds total gradient evaluations fixed at 24,000: a baseline could make 24,000 single-pass updates while SAM makes 12,000 two-pass updates. Both comparisons matter, but they answer different questions. Equal epochs, equal optimiser steps, equal data exposures, equal wall time and equal GPU-hours are not synonyms. A study must name its matching rule.
Memory cost is likewise not automatically doubled. Sequential first and second passes may release intermediate activations between them, though certain implementations retain graphs, clone parameters or store buffers. Measure peak allocated memory under the actual framework. The authors’ public JAX implementation offers a starting point for reproducing their experimental environment, not an assurance that every custom implementation has identical throughput.
A fair generalisation comparison needs a real evaluation protocol
Because SAM modifies training, the key outcome is not whether its perturbed training loss looks impressively low. Compare predictive performance on held-out data that neither model used to compute gradients. Use the same dataset definition, architecture, preprocessing, labelling policy, evaluation metric and predeclared search budget wherever possible. Validate a radius and optimiser configuration without repeatedly tuning to the final test set.
Suppose a fictional hold-out set has 5,000 examples. A baseline correctly classifies 4,050, for accuracy 4,050/5,000 = 81.0 percent and 950 errors. A SAM variant correctly classifies 4,125, for 82.5 percent accuracy and 875 errors. The absolute accuracy difference is 1.5 percentage points, while the relative reduction in error count is (950 − 875)/950 ≈ 7.89 percent. Those are different denominators and should not be conflated.
The 75 extra correct outcomes alone do not establish causation or statistical reliability. Ideally the same held-out examples are assessed by both models so paired disagreements can be examined, and repeated seeds expose variability. More importantly, if the SAM run received double the compute budget and a more extensive hyperparameter search, attributing the entire difference to the algorithm would be misleading.
Accuracy may not be the only relevant target. A safety-sensitive classifier may need false-negative rates, calibration, subgroup performance or abstention behaviour. The site’s separate bilevel hyperparameter optimisation guide explains why validation choices and training choices are distinct optimisation levels. SAM does not eliminate those boundaries.
What the original research demonstrates—and what it does not
Foret, Kleiner, Mobahi and Neyshabur introduced SAM in work published at ICLR 2021. Their paper presents a training objective, a first-order practical procedure, theoretical arguments concerning generalisation and reported experiments across selected architectures and benchmarks. The public project provides experiment scripts and training configurations. This is substantive original evidence that the method can be useful, not proof of universal superiority.
The generalisation arguments require assumptions and a defined measure of neighbourhood sharpness. A bound on a population-versus-empirical quantity is not a statement that every finite training run, every data distribution and every radius must improve held-out accuracy. A loose or assumption-dependent bound can motivate the method without giving an operational guarantee for a new school dataset, medical classifier or language model.
One should also distinguish the full neighbourhood objective from a first-order approximation computed on mini-batches. Mini-batch gradients are noisy, the chosen perturbation may not maximise loss exactly, and finite optimisation may never reach a global minimum. The observed model is the result of a particular training procedure under a particular budget, not the guaranteed optimiser of the abstract minimax function.
Claims such as ‘always finds a flat minimum’, ‘cannot overfit’ or ‘automatically fixes label noise’ overstate the literature. Original experiments reported promising robustness behaviour under selected noise settings; a new application must test its own noise, class imbalance, architecture and data shifts. Keep mechanistic predictions, mathematical results and empirical benchmark observations in separate sentences.
Find and repair the failure rather than guessing a better radius
If SAM fails to improve validation results, start with correctness checks. Verify the two-pass trace, perturbed norm, weight restoration, single outer optimiser step, parent mini-batch choice, gradient clearing, stateful-layer handling and actual radius. Only after these pass should a hyperparameter search be interpreted. Many apparent optimisers’ defects are accounting errors in the training loop.
A too-large radius can make the temporary ascent point unrepresentative of the local first-order approximation or push it into a radically different part of the loss surface. A tiny radius may be indistinguishable from the base optimiser relative to noise. Compare a deliberately small set of radii with repeated seeds and a fixed, suitable validation metric. Report the selected radius without implying one universal setting.
If compute is the constraint, look at gradient evaluations per successful improvement, not only accuracy per update. A two-pass method that slightly improves validation accuracy but substantially reduces the number of experiments a team can run may be a poor operational choice. Conversely, in some settings extra training work can be justified by stronger generalisation. The right decision includes hardware, energy, deadline and service stakes.
If training loss behaves strangely while held-out metrics remain flat, compare the average centre loss with the perturbed loss; they are not supposed to be identical. Look for exploding gradients, badly handled normalisation buffers, AMP overflow, unchanged parameters after skipped updates or inconsistent augmentation seeds. Do not quietly weaken the evaluation protocol to make a new optimiser appear successful.
Independent transfer test with a changed radius and an answer
Now change the toy calculation rather than merely repeating it. Use L(x,y) = 0.5 x² + 2 y², start again from (1,1), use a larger rho = 0.2, and keep learning rate eta = 0.05. First find the perturbation from g = (1,4), then the second-pass gradient and final weights. State whether the change proves anything about generalisation.
The initial norm remains sqrt(17), so the new perturbation is (0.2/sqrt(17), 0.8/sqrt(17)) ≈ (0.04850713, 0.19402850). The perturbed point is approximately (1.04850713,1.19402850). The gradient there is (1.04850713, 4.77611400). The ordinary base step from restored (1,1) therefore gives w_new ≈ (0.94757464, 0.76119430).
The new radius changes the second-pass gradient even though the centre loss, initial gradient and learning rate are unchanged. It does not force a claim that the new iterate is better on unseen data. In fact this synthetic quadratic has no unseen examples. The important transfer is that SAM depends on a coordinate-scaled neighbourhood, then updates original—not perturbed—weights.
For an additional boundary, set rho = 0. The perturbation becomes (0,0) and the second gradient equals the ordinary one, yielding (0.95,0.80) for this plain-SGD example. Now consider a constant loss with gradient (0,0): define an explicit safe zero-gradient branch rather than dividing by zero. These two cases test the meaning of the algorithm more precisely than a colourful loss-contour picture.
When SAM is worth testing and when it may be the wrong tool
SAM is worth considering when a model already trains reliably but shows a concerning difference between training performance and properly measured validation performance, and a team can afford extra gradient evaluations. It should be compared with well-tuned ordinary regularisation, data quality work, augmentation, architecture adjustment and appropriate early stopping. A new optimiser does not replace correct labels or a relevant dataset.
In a data-limited image classifier, a controlled study might compare baseline SGD, data augmentation plus SGD, and SAM under both equal-step and equal-compute designs. In a large language-model setting, training cost and optimizer-state complexity can make the balance different. In a tiny online-learning service that must update within milliseconds, two passes may be impractical even if the method helps accuracy elsewhere.
Risks outside optimisation remain. Distribution shifts, missing groups in training data, incorrect ground truth, privacy leakage and adversarial manipulation are not resolved by preferring low neighbourhood loss. A model may become smoother in a defined weight-coordinate sense and still fail badly for rare users or real-world contexts that were never represented during development.
Close the decision loop with the reader’s original task: define a valid loss, test two-pass arithmetic, validate against a relevant held-out metric, state the compute budget and record exact implementation choices. If SAM helps under those controls, the evidence supports using it for that bounded application. If it does not, the negative result is useful engineering information rather than a reason to manufacture a new narrative.
A reproducible experiment record that survives handover
A second engineer should be able to reconstruct the model, dataset split, preprocessing, loss aggregation, frozen and trainable parameter set, base optimiser, learning-rate schedule, gradient accumulation, SAM radius and norm, random seeds, batch-normalisation handling, mixed-precision policy and distributed-aggregation choices. Merely writing ‘SAM, rho = 0.05’ is insufficient to reproduce an unfamiliar codebase.
Save a small collection of deterministic unit tests with the training code. The two-dimensional quadratic example checks global norm arithmetic; the rho-zero test checks equivalence to the base step; the zero-gradient fixture catches division errors; a stateful buffer fixture checks duplicate updates; and a deliberately invalid gradient verifies restoration when an optimiser step is aborted.
Log the number of base updates, number of gradient passes, epoch count, actual data exposures, wall-clock duration and peak memory. Tie metrics to the same validation split and report uncertainty across runs. Distinguish snapshots selected by validation from a final locked-test evaluation. Anyone can make one metric look better by changing denominators; reproducible records prevent that misleading comparison.
The hyperparameter optimisation guide is useful when designing the radius search. For gradient-norm control and AMP, use the gradient clipping specialist. Both are separate reader jobs, not interchangeable names for this article’s two-gradient-pass optimisation mechanism.
Frequently asked questions
Is SAM an optimiser or a training wrapper?
SAM is a sharpness-aware training method that can be coupled to a base optimiser. Its characteristic feature is the temporary ascent probe and second gradient. SGD, momentum or an adaptive optimiser then carries out the outer update under its own state and update rules.
Does SAM always double training time?
It commonly requires roughly two gradient evaluations per logical update rather than one, but exact wall time depends on data loading, network communication, memory behaviour, model structure and hardware. Compare elapsed GPU-hours and outcome metrics rather than claiming a universal time multiplier.
How does SAM differ from adversarial input training?
SAM perturbs selected model parameters during its inner probe. Adversarial input training perturbs input data under an input-specific constraint and threat model. They may both involve minimax language, but their neighbourhoods, guarantees and applications are different.
What does the radius measure?
Rho bounds the chosen norm of the temporary parameter perturbation. In ordinary L2 SAM it is the Euclidean distance across the selected parameter coordinates. It is not directly an accuracy percentage, input-noise magnitude or invariant property of a network’s represented function.
Why is the same batch usually used twice?
The first pass selects an adverse direction for a particular mini-batch loss; the second measures the gradient at that nearby point for the corresponding loss. Different data or stochastic augmentations can define a different procedure, which should be documented and evaluated rather than silently assumed to be identical.
Can SAM fix mislabeled training data?
No general guarantee follows. Original work reported encouraging selected label-noise experiments, but noisy, biased or systematically wrong labels still require data investigation and robust evaluation. A smoother parameter neighbourhood cannot create ground truth that a dataset lacks.
What should be done if the first gradient is zero?
Specify a safe handling branch rather than divide by zero. Many practical approaches use zero perturbation for that logical step. Check that the selected parameter set and gradient scaling are correct, then test the branch with known constant and partially frozen toy models.
How does SAM relate to adaptive SAM (ASAM)?
ASAM is a related algorithm motivated partly by parameter-rescaling sensitivity of fixed-radius sharpness. It changes the measure of neighbourhood sharpness and associated optimisation rule. Reproducing ASAM requires its own original method and code details, not merely renaming an ordinary SAM run.
Do lower loss-surface sharpness plots prove higher test accuracy?
No. Raw curvature and radius depend on parameterisation, and many training factors affect unseen performance. A valid comparison uses held-out metrics, controlled budgets, model selection without test leakage and repeated runs where appropriate.
What is the first bug to look for in a SAM implementation?
Trace whether it computes g1 at original weights, perturbs with the intended norm, recomputes g2, restores the original weights and then performs exactly one base-optimiser update. Also check gradient buffers and batch-normalisation state, since errors there can mimic an optimisation failure.
Primary research and developer sources
Original research from the authors
Foret, Kleiner, Mobahi and Neyshabur, Sharpness-aware Minimization for Efficiently Improving Generalization (ICLR 2021). This primary research introduces the loss-neighbourhood objective, efficient first-order training procedure and bounded experimental claims. The article uses none of its benchmark numbers as invented outcomes.
Research team’s released training code
google-research/sam. This repository is the authors’ own public research implementation, with JAX experiment commands and specific hyperparameter settings. A reproduced run still needs the exact code version, model, data pipeline and hardware details.
Adaptive variant primary source
Kwon, Kim, Park and Choi, ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks (ICML 2021). This primary paper explains why fixed-radius geometric sharpness is sensitive to reparameterisation and proposes a distinct adaptive approach.
Official implementation reference for gradient scaling
PyTorch official Automatic Mixed Precision examples. This developer document describes scaled and unscaled gradients, norm clipping, and GradScaler ordering. It is cited for AMP semantics, not as an officially supported complete SAM implementation.
Primary arXiv manuscript
Original SAM research manuscript (arXiv:2010.01412). Read it for the derivation, theoretical qualifications, definition of local sharpness and experiments; distinguish paper assumptions from an engineering guarantee for any new dataset.
Verified further reading on eduKateSG
For the mechanics of a different training-stability intervention, see Gradient Clipping — Global Norms, Mixed Precision and Stable Updates. Clipping controls gradient magnitude rather than the radius of an adverse weight perturbation.
For a different optimisation decision, read Bilevel Hyperparameter Optimisation, which explicitly separates training optimisation from validation-based settings such as rho. Return to the How AI Works overview to choose another machine-learning mechanism.
A useful independent task is to implement the quadratic two-pass example using plain arithmetic, verify every intermediate value, then compare a baseline and SAM on a properly separated dataset while reporting the same denominators and compute budget. If the final result is inconclusive, report it as inconclusive rather than as an optimisation breakthrough.