Did you know? A classifier trained to recognise four categories does not have to be rewarded for assigning exactly 100% probability to the correct training label. Label smoothing replaces the usual one-hot target, such as (1, 0, 0, 0), with a distribution that keeps most probability mass on the labelled class and assigns a small portion to the others. The model still learns to identify the observed category, but the loss no longer demands unlimited confidence in that single answer.
The practical value of label smoothing is not that the labels have secretly become uncertain facts. It is a controllable change to the training objective, sometimes useful as regularisation and sometimes useful for prediction calibration. It can also remove informative differences among alternative classes and make a smoothed teacher model less useful for knowledge distillation in some settings. Use it only after defining the class mapping and loss convention, compare held-out models, and examine probability reliability and minority errors rather than assuming lower training confidence is automatically better.
This guide gives the exact formula, two fully solved calculations, diagnoses implementation failures, explains the research limits and transfers the reasoning to a new classification problem. It is about present-day machine learning, not a claim that artificial superintelligence exists. Every classroom, service and numerical scenario below is an invented teaching illustration unless a primary research source is explicitly identified.
Contents
- What smoothing changes in a one-hot classification target
- Why ordinary cross-entropy can encourage sharp probabilities
- Worked example: four classes and epsilon equal to 0.10
- A gradient-level view: how the direction changes
- Smoothing conventions and class counts cannot be guessed
- Training confidence is not the same as calibrated probability
- Why smoothing can make a teacher less informative
- Smoothing does not automatically repair incorrect labels
- Compare models under a fair experiment, not a favourite chart
- Seven mistakes that turn a simple idea into a misleading result
- An implementation checklist for PyTorch and similar systems
- Transfer problem: three categories, a new epsilon and a changed decision
- What label smoothing does not promise
- Frequently asked questions
- Reader pathways, related guides and primary sources
What smoothing changes in a one-hot classification target
In ordinary supervised classification, a class label names one category. A one-hot training target represents it as a probability-like vector containing a one at the observed class and zeros elsewhere. For four classes ordered A, B, C and D, a record labelled A has target (1, 0, 0, 0). Under ordinary cross-entropy, a model can lower the loss for that record by raising its predicted probability of A toward one while pushing the others toward zero.
Label smoothing modifies the target that enters the loss. With K classes, original one-hot target y, smoothing parameter epsilon and a uniform distribution u with each entry 1/K, define the smoothed target q = (1 − epsilon) y + epsilon u. Under this convention the true-class target becomes 1 − epsilon + epsilon/K; each other class gets epsilon/K. The entries still add to one.
The wording ‘smoothing by ten percent’ can be ambiguous. Under the uniform-mixture definition, epsilon=0.10 does not mean the observed class gets exactly 0.90. With four classes it gets 0.90 + 0.025 = 0.925, while each of the three alternatives gets 0.025. Some other implementations distribute epsilon only across the incorrect classes, producing a different target. Both conventions exist, but they cannot be silently substituted in an experiment.
PyTorch’s official CrossEntropyLoss documentation describes label_smoothing as mixing the original target with the uniform distribution. That is the convention used throughout the first numerical example here. Framework documentation, rather than a remembered blog snippet, should decide which behaviour an actual codebase implements.
The distinction matters because a target vector is part of the optimisation problem. Even a small change in its entries changes the gradient direction and strength. The model is not merely told to ‘sound less confident’; the learning system receives a quantitatively different objective for every affected training example.
Why ordinary cross-entropy can encourage sharp probabilities
Cross-entropy compares a target distribution with predicted class probabilities. For one-hot targets, only the predicted probability of the observed label contributes directly to the loss. If A is correct, the per-example loss is −log(p_A), using a natural logarithm. Increasing p_A from 0.90 to 0.99 lowers that loss. Increasing it from 0.99 to 0.999 lowers it further. The objective rewards increasingly concentrated predictions even when the distinction among other classes carries useful information.
That does not mean every ordinarily trained classifier ends up unjustifiably overconfident. Data, regularisation, model capacity, architecture, sample size and optimisation all influence output reliability. But the incentive in a one-hot loss is clear: the individual training label is treated as the target mass of one. Smoothing changes that incentive without needing to change the model’s last-layer architecture.
Under smoothed cross-entropy, loss is −Σ q_c log(p_c) across the K classes. When q allocates nonzero mass to alternatives, the loss penalises a model that drives every alternative probability to an extremely small value. This matters most for predictions that are excessively sharp. The optimum for an individual target distribution, if the model can represent it without other constraints, is to match q rather than place all mass on one class.
The mechanism was described as part of the research on Rethinking the Inception Architecture for Computer Vision. It has since been studied across other multi-class model families. The findings are context-specific; the formula is general, but the effect on generalisation and downstream tasks requires evidence.
A useful intuition is that smoothing discourages certainty based on one training label, not that it supplies a new independent measurement of uncertainty. If the dataset’s label is actually wrong, a uniformly smoothed wrong label is still centred on the wrong class. Better labels, disagreement handling and careful dataset design remain separate responsibilities.
Worked example: four classes and epsilon equal to 0.10
Consider four classes A, B, C and D, with the record labelled A. The hard target is y=(1,0,0,0). Set epsilon=0.10 and K=4. The uniform component gives 0.10/4 = 0.025 to every class. The retained hard-label component gives 1−0.10 = 0.90 to A. The smoothed target is therefore q=(0.925,0.025,0.025,0.025). Check its denominator: 0.925 + 3 × 0.025 = 1.000.
Suppose the classifier predicts p=(0.70,0.20,0.08,0.02). Those probabilities also add to one. Ordinary one-hot cross-entropy is −log(0.70) ≈ 0.356675 nats. Smoothed cross-entropy is −0.925log(0.70) −0.025log(0.20) −0.025log(0.08) −0.025log(0.02) ≈ 0.531104 nats. Both numbers are correct for this invented example and this target convention.
Notice that the smoothed loss is higher for this particular prediction. That does not prove smoothing is worse or better. It reflects a changed objective: probability mass 0.02 on class D is relatively small when the smoothed target for D is 0.025, and the loss still wants a nonzero probability for other classes. Comparing raw training losses from different targets does not by itself tell us which model predicts independent cases more accurately.
Now imagine an extremely sharp prediction p=(0.98,0.01,0.005,0.005), still labelled A. Hard-label loss falls to −log(0.98) ≈ 0.020203. Smoothed loss with the same q is approximately 0.398733. The smoothed objective strongly discourages making every alternative almost impossible. Yet if an external use truly requires a near-certain probability, this regularisation could also impose unwanted limits. The application must decide what probability claims mean.
These are per-record loss calculations, not measured model performance. They cannot establish that a trained neural network will output exactly q on unfamiliar examples, or that its test-set calibration automatically improves. The purpose is to make the training signal visible before adding optimiser updates and many interacting observations.
A gradient-level view: how the direction changes
With softmax predictions and standard cross-entropy under suitable conventions, the derivative of loss with respect to each class logit is the predicted probability minus the target probability. For one-hot labels, this is p−y. For smoothed targets, it becomes p−q. This provides a concise explanation of what the model is encouraged to change after observing an example.
Use the four-class example p=(0.70,0.20,0.08,0.02). Against hard target y=(1,0,0,0), the logit gradient is (−0.30,0.20,0.08,0.02). Its entries sum to zero, consistent with a probability redistribution. Against q=(0.925,0.025,0.025,0.025), the gradient becomes (−0.225,0.175,0.055,−0.005). Those entries also sum to zero.
The negative final component is easy to miss. Since p_D=0.02 is below its smoothed target 0.025, the gradient encourages the model to increase D’s logit under gradient descent rather than reduce it further. Under hard labels, D’s contribution was positive 0.02, encouraging reduction. Smoothing changes the instruction about extremely small alternative probabilities, not merely the display of confidence at the end.
How much a parameter actually changes depends on learning rate, batch aggregation, class weights, momentum and other examples. A gradient that is smaller for one class does not imply the whole model becomes less accurate or universally less confident. Training dynamics are collective, and different records have conflicting demands on shared parameters.
An implementation should check tensor dimensions and target normalisation before interpreting gradients. With an unexpected class order, the smoothed mass can be allocated to the wrong category. With invalid soft targets whose entries do not sum to one, the loss may run yet no longer describe the intended probability distribution. PyTorch notes that when users pass arbitrary probability targets, responsibility for their validity remains with the caller.
Smoothing conventions and class counts cannot be guessed
Two common conventions deserve explicit separation. Uniform-mixture smoothing uses q=(1−epsilon)y + epsilon/K across all K classes, which gives the observed class 1−epsilon+epsilon/K. Another convention allocates the residual epsilon only across K−1 incorrect classes, leaving exactly 1−epsilon on the true class. At K=4 and epsilon=0.10, the first produces (0.925,0.025,0.025,0.025); the second produces (0.90,0.033333…,0.033333…,0.033333…). They do not generate identical training gradients.
The difference may look small, but experiments tuned near a useful setting can be sensitive to it. A paper’s reported epsilon cannot be imported safely unless its definition and implementation match. State the target vector explicitly when explaining a result. If an experimental reproduction produces unexpected values, inspect the conventions before blaming model architecture or random seeds.
With two classes, uniform smoothing distributes epsilon/2 to each category. That behaves differently from using a fixed incorrect-class amount when K is large. A classification task with thousands of classes spreads the uniform component very thinly across many alternatives. Conversely, a tiny class vocabulary can assign noticeable mass to each alternative.
Not every task has mutually exclusive classes. In multilabel classification, a record can belong to several labels at once. A single probability vector summing to one may be the wrong representation; independent binary targets and their corresponding losses may be more appropriate. Do not copy a multi-class label-smoothing formula into a multilabel system without checking what each output means.
Padding and ignored labels also matter for sequence prediction or segmentation. A padding position is not an uncertain class that should receive uniformly distributed probability. It is a position excluded from the relevant loss under the task’s declared rules. Smoothing changes valid target distributions, not which examples or positions count.
Training confidence is not the same as calibrated probability
A calibrated probabilistic classifier is one whose stated probabilities match observed frequencies under an appropriate evaluation protocol. If many truly independent predictions receive 0.80 probability for their selected class, then about 80% should be correct in an ideal calibrated sample, with uncertainty from finite observations. Lower probabilities are not intrinsically better. Predicting 0.55 for events that happen 90% of the time is underconfident rather than reliable.
Some published experiments have found that label smoothing can improve calibration as well as generalisation. The 2019 NeurIPS paper When Does Label Smoothing Help? reports such findings in its studied settings, alongside an important cost for knowledge distillation. An engineer should reproduce the relevant benefit on independent data rather than assume it transports unchanged to every model or deployment population.
Consider a fictional held-out group of 100 independent predictions with average stated top-class confidence 0.80. Suppose 65 are correct. The empirical accuracy is 65/100 = 65%, so the gap between mean stated confidence and observed accuracy in that illustrative group is 80%−65%=15 percentage points. That is a warning sign about overconfidence for that group, not a complete global calibration score.
A second invented model on the same 100 cases has mean top-class confidence 0.70 and gets 68 correct. Its group gap is |70%−68%|=2 percentage points, while its observed accuracy is 68%. In this narrow example, the second is both better aligned and slightly more accurate. Real evaluation would need repeated groups, a suitable binning or scoring method, and uncertainty intervals. The model identity and prevalence must be kept constant when making a causal claim about smoothing.
Temperature scaling is a different intervention: it changes logits or scores after training using calibration data. Label smoothing changes the target and thus the learned representation during training. The sibling probability calibration guide explains how to inspect reliability and why a single attractive confidence plot can mislead.
Why smoothing can make a teacher less informative
Knowledge distillation uses a teacher model to train a smaller student, often by matching a distribution of teacher outputs rather than just the top predicted category. The distribution over alternative classes can carry useful relationships: a picture labelled ‘wolf’ might also receive substantial probability for ‘dog’ and far less for ‘bus’. Such differences can help a student learn structure beyond the hard label.
Uniform label smoothing encourages the target to allocate equal small mass to all non-labelled classes. During training, this can reduce the teacher’s incentive to preserve fine distinctions among alternative categories in its final scores. The 2019 NeurIPS study observed settings in which smoothed teachers were less effective for distillation, even where smoothing helped other measures. The result is a meaningful trade-off, not a universal prohibition.
Do not overstate the mechanism. A smoothed teacher need not output exactly equal probabilities for all alternatives on every input. Shared features, architecture and training dynamics still produce differences. But the training objective has less reason to reward highly differentiated low-probability alternatives for each hard-labelled example. That can affect the soft information a student receives.
If the next task is model compression, evaluate the teacher as a teacher, not only as a classifier. Compare student results when learning from the hard-label teacher, smoothed teacher and other suitable baselines under matched student capacity and training resources. Do not claim a better standalone teacher necessarily produces a better compressed model.
Where a student must learn from teacher probabilities, keep the distinction between smoothing the original supervised targets and applying a temperature to the teacher’s scores for distillation. These are different operations. A temperature can redistribute an existing set of score differences; uniform-mixture smoothing changes the original learning target. The resulting information content cannot be assumed identical.
Smoothing does not automatically repair incorrect labels
A noisy label is an observation error or ambiguity in the training target. Smoothing modifies the loss associated with that label but does not reveal the correct category. If a specimen is wrongly labelled A when it is truly C, a smoothed target still places most of its mass on A. Increasing epsilon may weaken that wrong signal, yet it can also weaken reliable labels and obscure real distinctions.
Research on the interaction between label smoothing and noisy labels has produced nuanced findings rather than one universal rule. The study Does Label Smoothing Mitigate Label Noise? investigates this connection to noise-correction techniques. Later work To Smooth or Not? When Label Smoothing Meets Noisy Labels reports that advantages can diminish in high-noise regimes. These claims apply to the methods and data examined; they should not be used to diagnose a new dataset from its class distribution alone.
The first repair for suspicious labels is a transparent annotation policy, re-examination of high-impact cases, and measurement of disagreement. If the category scheme is ambiguous, revise definitions or admit uncertainty. If reviewers systematically confuse adjacent classes, a new label taxonomy or hierarchical classification may be more faithful than distributing uniform probability across unrelated classes.
Class imbalance is different again. If the positive category is rare, an unweighted objective may underemphasise it. Label smoothing alone does not correct prevalence or costs. The specialist class weighting and focal loss guide discusses separate ways of allocating training attention and why those methods also need held-out evaluation.
Never infer that smoothing is a substitute for a final label audit. Models trained on contaminated or ambiguous data can be calm and wrong. A lower maximum probability may make the error look less alarming without actually improving the decision the model makes.
Compare models under a fair experiment, not a favourite chart
Start with a baseline trained under the ordinary target convention. Keep data splits, preprocessing, architecture and evaluation protocol fixed while changing epsilon. Compare a small, defensible set of candidate values rather than tuning dozens of numbers on the final test set. Include epsilon zero, because the added complexity must earn its place.
Use a validation set to select the smoothing parameter and associated training controls, then test the selected model on an untouched representative set. Report both accuracy or task performance and probability-quality measures relevant to the application. If a system feeds probabilities into downstream decisions, inspect their reliability. If it only ranks alternatives for human review, discrimination, recall at capacity and error severity may matter more.
Because training changes with smoothing, apparent improvements may interact with learning rate, weight decay, regularisation, batch size and model capacity. The NeurIPS authors themselves caution about the role of such interactions when interpreting results. Maintain a comparison that makes a causal interpretation plausible rather than changing half the training recipe at once.
Where a model is used across demographic groups, sensors, languages or sites, inspect performance by relevant subgroup. A small aggregate improvement could hide deterioration for a minority population. A better-calibrated global average does not prove each subgroup is adequately calibrated. Give the reader actual counts and intervals rather than smooth curves unsupported by a large sample.
Evaluate eventual downstream use. A student classifier used to select extra practice has a different error cost from a model that recommends irreversible penalties. A system that declares ‘certain’ less often may be preferable in one setting and unhelpful in another. A suitable decision policy should be designed and tested alongside the score model, not inferred from a training hyperparameter.
Seven mistakes that turn a simple idea into a misleading result
Failure one is using the wrong smoothing convention. If one framework distributes epsilon uniformly across all K classes and another only among incorrect classes, the same label_smoothing parameter can yield different targets. Fix it by writing the vector and checking that its entries add to one.
Failure two is passing invalid probability targets. When a model library accepts soft targets, it may not enforce every probability constraint. Verify nonnegative values, shape, class order and sum-to-one for each valid example. Failure three is applying smoothing to padding or ignored labels. Define a mask and check which positions contribute to the denominator.
Failure four is mixing smoothing with aggressive class weights without analysing the combined objective. A weight on one class can affect soft-target mass on that class across many examples. The result may differ substantially from weighting only the observed class. Use a small numerical unit test of the actual implementation rather than assuming two options commute.
Failure five is interpreting lower confidence as calibration. Calibration means empirical agreement with stated probabilities, not a preference for numbers closer to 0.5. Failure six is choosing epsilon by repeated feedback from the final test set. That turns the test set into a hidden training resource and inflates the credibility of the reported improvement.
Failure seven is treating better teacher accuracy as a guarantee of better knowledge distillation. A smoothed teacher may lose useful relational information among alternative classes. Evaluate the compressed student directly and report both its utility and any probability changes.
Across all seven, the remedy is a transparent contract: target definition, library version, epsilon, masks, class mapping, loss reduction, independent split, primary outcome and secondary checks. When a result is surprising, inspect this chain from the earliest possible error rather than adding another unexplained technique.
An implementation checklist for PyTorch and similar systems
Begin with a tiny, manually checked target. Create one record for a four-class problem, set epsilon to 0.10 and compare the expected target (0.925, 0.025, 0.025, 0.025) with the framework’s described convention. Feed a known vector of predicted probabilities and verify that the resulting scalar loss is approximately 0.531104 under the exact formula used here. A small unit test is more persuasive than a confident statement about the parameter.
Next check that model inputs are logits when the loss expects logits. PyTorch CrossEntropyLoss combines the relevant transformation internally; passing already-normalised probabilities into a logits-taking interface can change the result. Conversely, if you implement cross-entropy directly from probabilities, you must handle numerical stability near zero. Follow the official interface rather than a copied pseudocode fragment.
Confirm dimensions and reduction. A batch of ten labelled examples with four categories should have a compatible logits tensor and label representation. For sequence tokens or pixel classes, there may be additional spatial dimensions. Verify that the returned mean divides by the intended number or effective weight. Seemingly small reduction differences can interact with the optimizer’s step size.
Then inspect class weighting and ignore masks where used. Do not assume every combination of soft probability targets, ignore_index and weight behaves exactly like the integer-label case. Read the installed framework version’s documentation and use tests with known expected values. If a flag is not supported for the chosen target type, redesign the computation explicitly rather than relying on silent defaults.
Save experimental metadata: dataset version, label policy, epsilon convention, training split, random seed, architecture, optimizer, selected validation criterion and final test results. Document whether smoothing was applied only to valid class positions. A reproducible outcome should not depend on remembering which notebook cell someone ran first.
Finally test the operational effect. Does a model predict reliably on unseen records, produce useful probabilities and preserve distinctions that downstream users need? Record failures honestly and prefer the simpler baseline when smoothing offers no clear benefit.
Transfer problem: three categories, a new epsilon and a changed decision
Imagine a different classroom example with three mutually exclusive categories: plant, animal and mineral. One training record is labelled plant. Set epsilon=0.12 using the uniform-mixture convention. The true-class target is 1 − 0.12 + 0.12/3 = 0.92; each other class has 0.12/3 = 0.04. The target q=(0.92, 0.04, 0.04), which sums to exactly one.
Suppose a model assigns p=(0.80, 0.15, 0.05). Ordinary one-hot cross-entropy is −log(0.80) ≈ 0.223144. Smoothed cross-entropy is −0.92log(0.80) −0.04log(0.15) −0.04log(0.05) ≈ 0.401006 nats. Check that the class count in the denominator is three, not four as in the previous case. Copying the four-class target would produce probabilities that do not match this task.
Now change the use case. Instead of offering optional study questions, the predictions decide whether a specimen is immediately excluded from a laboratory procedure. A smoother classifier might assign less extreme probabilities, but that is not a sufficient safety control. The laboratory needs a clearly defined threshold, checks against real specimens, error-cost analysis and human verification. A training regulariser cannot authorise an irreversible operational decision.
Change the data again: some specimens legitimately have both plant and mineral material. The three mutually exclusive classes no longer describe the facts. The problem is not solved by increasing epsilon from 0.12 to 0.30. The label ontology or task structure needs repair, perhaps by separating material composition from primary category. Smoothing cannot make an inconsistent target definition correct.
The transfer lesson is that the formula remains stable within its assumptions while the surrounding problem changes. Identify class count, exclusivity, label meaning and final user decision before importing a training trick. A correct numerical target is necessary for a valid experiment, but not sufficient for an intelligent application.
What label smoothing does not promise
Label smoothing is not a universal accuracy booster. It can help under some data and model conditions and harm under others. Its effect depends on architecture, task, optimizer, data quality and the outcome being measured. Source experiments describe particular settings, not an unconditional guarantee that every model should use epsilon above zero.
It does not create independent evidence of which label is true. It does not correct missing examples, solve class imbalance, repair leakage or prevent adversarial inputs. It does not by itself make predictions fair across groups or safe for consequential action. These problems need their own diagnostics and controls.
Lower stated confidence is not the same as humility, and a model cannot become trustworthy merely by replacing 0.99 with 0.85. Trustworthiness depends on validity under the relevant distribution, appropriate uncertainty, independent evaluation and a bounded decision process. Underconfident models can also mislead users.
Some applications need detailed soft relations among alternative classes. Uniform smoothing may erase part of this information, especially when the trained model later acts as a teacher for distillation. In such cases a different objective or no smoothing may be preferable. Test the downstream role rather than optimising an intermediate metric alone.
The invented arithmetic in this article intentionally uses small vectors, fixed distributions and labelled cases so readers can reproduce each calculation. It is not a performance report from a commercial classifier or a claim to have run PyTorch here.
Frequently asked questions
What is label smoothing in simple terms? It replaces a perfectly one-hot training target with a mostly correct-class distribution that gives a small amount of target probability to alternatives. It changes the loss that guides training, not necessarily the true uncertainty in the label.
Does smoothing mean the correct class is no longer correct? No. The observed training label remains the designated category. The softened vector is a regularisation choice, not a newly measured ground-truth distribution.
Why doesn’t epsilon=0.1 always produce 0.9 on the true class? Under uniform-mixture smoothing, epsilon is distributed across all K classes, including the correct one. With four classes the true target is 0.925. Other conventions allocate all epsilon to incorrect classes.
Does lower confidence guarantee calibration? No. Calibration requires that predicted probabilities agree with actual observed frequencies on representative independent data. Underconfidence and overconfidence are both possible.
Can label smoothing fix mislabeled training data? It can alter how strongly a noisy label influences the loss, but it does not identify or correct the true class. Label review and noise-aware methods remain necessary.
Should I use label smoothing with class weights? Possibly, but the combined loss must be understood and tested. Check whether weights affect all smoothed class terms and whether rare-class errors improve on independent data.
Why might smoothing hurt knowledge distillation? It can reduce informative differences among incorrect-class scores in the teacher model, limiting the soft relationships a student learns. Empirical results depend on the model and training scheme.
What happens when epsilon is zero? The uniform-mixture formula returns the original hard one-hot target. This is the natural baseline for comparisons.
Is it suitable for every language model or multilabel classifier? No. The target interpretation, loss, class structure and downstream goal differ across tasks. Choose a method appropriate to the actual output space.
Reader pathways, related guides and primary sources
If you are a student, calculate the four-class target from epsilon and K, check that probabilities sum to one and reproduce the 0.531104 loss. Then change the number of classes and explain why the uniform contribution changes. Treat the arithmetic as a model of the learning signal, not proof that a model has become wiser.
If you are a machine-learning engineer, write a unit test for the target convention and loss reduction, keep a zero-smoothing baseline, validate probabilities on a representative held-out set, and examine the effect on any downstream distillation pipeline. For related specialisms, consult class weights and focal loss and probability calibration.
If you are a reviewer or educator, ask which labels are reliable, what softening convention was used, whether a minority class lost recall, and which score justifies the final decision. A visibly cautious probability is still a prediction requiring evidence, not an authority.
Primary references: PyTorch CrossEntropyLoss documentation; Szegedy et al., Rethinking the Inception Architecture for Computer Vision (CVPR 2016); Müller, Kornblith and Hinton, When Does Label Smoothing Help? (NeurIPS 2019); and the original analyses Does Label Smoothing Mitigate Label Noise? and To Smooth or Not? When Label Smoothing Meets Noisy Labels.
All calculations and illustrative outcomes in this guide are explained rather than claimed as tool execution. Continue through the How AI Works guide or the wider learning-transfer framework.