eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Sampled Softmax — Large Vocabularies, Expected Counts and Honest Evaluation

—

by

eduKate Asia

Did you know? A classifier with fifty thousand possible output classes can require a score for every class whenever ordinary softmax is evaluated. Sampled softmax reduces part of that training expense by comparing the true classes against a selected sample of alternatives rather than calculating the full normalising sum for every update. The trick is useful for very large output vocabularies, but it changes the training loss approximation. A small sampled-class loss cannot be presented as if it were an exact full-vocabulary probability.

The correct workflow is to identify the true label, choose and document a candidate-sampling distribution, account for each candidate’s expected count according to the implemented loss, handle accidental sampling of true labels, train with the approximate objective, and evaluate on a representative task using the full logits or an otherwise explicitly justified measurement. This method is distinct from negative sampling and noise-contrastive estimation, which have different objectives even when they also train with a subset of classes.

This article explains that distinction through original numerical examples, the actual source documentation, implementation diagnostics and a changed-context exercise. The illustrations are not model experiments performed here. All computed quantities are explicitly defined; any claim about a trained model improving must come from independent observations, not from the fact that a sampled loss is cheaper.

Contents

The full-softmax calculation grows with the class vocabulary

Imagine a model predicting the next token from a vocabulary of fifty thousand tokens. Its final hidden state is compared with a weight vector for every possible class, creating one logit per token. The softmax function exponentiates those logits and divides each exponential by the sum over all classes. The true token’s negative log probability becomes a supervised training loss. This is a mathematically straightforward objective, but its final projection and normalisation can be expensive as the class count grows.

For one training item with logits z_1 through z_K, the ordinary softmax probability of the correct class y is exp(z_y) divided by the sum of exp(z_j) across all K classes. The negative log-likelihood is log(sum_j exp(z_j)) minus z_y. This calculation couples every class through one denominator. Omitting classes changes the denominator and therefore changes the loss, even when the true class’s logit remains the same.

The cost matters especially when a class weight matrix is large and many updates are needed. A full projection may require substantial arithmetic and memory movement, although hardware acceleration, batching, sharding and approximate kernels can change the practical cost. It would be wrong to promise one fixed speed-up from the count of classes alone. We can calculate the candidate arithmetic reduction, but only a measured implementation can establish wall-clock gains.

Sampled softmax aims to make the training update cheaper by evaluating the known positive label and a comparatively small set of candidate alternatives. The candidates are drawn under a particular sampling law, often using an efficient frequency-based or log-uniform distribution. A corrected score can take account of how likely each class was to enter the sample. The resulting loss is an approximation with its own assumptions and bias behaviour.

The official TensorFlow sampled_softmax_loss documentation explicitly describes it as a training method over a huge number of classes and says it is generally an underestimate of the full softmax loss. The page recommends calculating the full softmax loss when suitable for evaluation or inference. That warning is central to honest reporting.

The positive class and the sampled alternatives play different roles

A supervised example has one or more designated target classes. The positive class is present because the training record says it is the target, not because a random sampler happened to choose it. Negative candidates are alternatives selected to approximate a larger comparison set. The sampled loss must know which is the true class and which candidates were selected from the sampling distribution.

If the sampler independently happens to draw the true class as one of the candidate negatives, it creates an accidental hit. Some implementations remove or mask those hits because treating the true class as its own negative can corrupt the intended comparison. TensorFlow exposes a remove_accidental_hits argument and provides documented candidate-sampling utilities. The behaviour needs testing rather than an assumption that a data loader will never produce a duplicate.

Repeated candidate classes also require attention. Sampling with replacement can generate the same alternative more than once. Deduplicating candidates changes their inclusion probability and expected counts, so the corrective computation must match the actual sampling semantics. A simple Python set is not an innocuous substitute for a probabilistically specified candidate sampler.

Sampled softmax does not require that every excluded class is unimportant. An unsampled class could receive a very high logit from the model, in which case the approximate batch comparison may miss an error that full softmax would expose. Larger or better-designed samples can sometimes reduce this risk, but sampling every class would forfeit the intended computation advantage.

An implementation must keep the class index consistent among the label tensor, output matrix, candidate sampler and any sharded weights. A class ID is not a word string, and two systems can number the same vocabulary differently. A silent reindexing bug can make a sampled loss look numerical while training the wrong associations.

Fully solved example: exact softmax versus an uncorrected subset

Take a fictional classifier with four classes A, B, C and D. For one record labelled A, assume the logits are (2, 1, 0, −1). The ordinary denominator is exp(2)+exp(1)+exp(0)+exp(−1), approximately 7.38906+2.71828+1+0.36788=11.47522. The exact probability of the true class is exp(2)/11.47522≈0.643914, and the full negative log-likelihood is log(11.47522)−2≈0.440190 nats.

Now deliberately select only the true class A and two alternatives, C and D, ignoring class B. The naive subset denominator becomes exp(2)+exp(0)+exp(−1)≈7.38906+1+0.36788=8.75694. Its uncorrected subset loss is log(8.75694)−2≈0.169846 nats. This smaller value is not evidence that the model became better. We merely removed an alternative with logit 1 from the normalising comparison.

The difference is approximately 0.440190−0.169846=0.270344 nats. It arises from the change in the denominator, not any parameter update. If a student compares the two losses as if they were calculated on the same objective, the conclusion will be false. In training, candidate correction and sampling assumptions affect the approximation, so the naive-subset number is a useful demonstration of the problem rather than a complete sampled-softmax implementation.

Imagine instead selecting A, B and D. The excluded logit is then C=0, and the subset denominator is exp(2)+exp(1)+exp(−1)≈10.47522. That produces a different approximate loss even though the model and true class have not changed. Candidate selection is a source of stochastic variation that an experiment needs to document.

The central lesson is to keep three objects separate: the exact full-vocabulary log probability, the set of sampled candidates and the approximate training objective. A positive result on the third does not prove a positive result on the first. The proper comparison is ultimately the intended predictive performance on held-out examples.

Why expected sampling counts appear in corrected logits

Not all classes are sampled with equal probability. A frequency-based sampler might draw a common word more often than a rare technical term. If the training objective naively treats sampled alternatives as an ordinary uniform set, it can systematically over-represent common candidates. A sampled-softmax formulation can correct class scores using the probability or expected count of each candidate under the sampling scheme.

TensorFlow’s Candidate Sampling Algorithms Reference describes a sampled softmax score involving a correction of the form F(x,y)−log Q(y|x), where Q represents the expected count or sampling probability under its specified candidate process. This is an algorithmic detail, not a statement that the probabilities of all classes have become exact. The precise expected-count convention, handling of unique samples and true candidates must match the implementation.

Suppose two hypothetical negative classes have equal raw logit zero, but one has expected sampled count 0.20 and the other 0.05. Subtracting the natural log expected count changes their corrected scores by −log(0.20)≈1.609 and −log(0.05)≈2.996 respectively. The less frequently sampled class receives the larger correction in this illustrative calculation. It would be an error to conclude that the corrected scores are calibrated predictive probabilities by themselves.

The true class may also have an expected-count adjustment under the particular sampled-softmax estimator. Code that corrects only negatives while copying formulas from a different sampler can silently alter the objective. Avoid assembling a custom estimator from fragments without a derivation or checked reference. Use a documented implementation or verify its exact distribution and gradient in a small synthetic case.

Sampling correction improves the relationship to the intended class comparison under assumptions; it does not eliminate all approximation error. Finite candidate sets, nonuniform sampling, accidental hits and repeated negatives still influence training. The final loss is a stochastic approximation, not simply full softmax evaluated faster through an algebraic identity.

Worked scale calculation: fifty thousand classes versus 513 candidates

Consider a made-up model whose output hidden dimension is 256 and whose vocabulary contains 50,000 classes. A simple dense output projection for one example computes a 256-component dot product for every class, requiring approximately 50,000×256=12,800,000 component multiplications before counting additions and other operations. This is a rough arithmetic count, not a claim about actual device runtime.

Suppose a sampled training objective uses the one known target and 512 sampled alternatives, a total of 513 candidate class scores. Computing the same type of dot product only for those candidates would require about 513×256=131,328 component multiplications. The naive ratio is 12,800,000/131,328≈97.47. Under the simplified projection-only count, the sampled computation requires roughly one ninety-seventh as many component multiplications.

Real training includes many other costs: extracting embeddings, generating samples, correcting expected counts, other network layers, synchronising devices and possibly evaluating full logits for certain operations. Batching can also make full dense operations very efficient on GPUs. The factor 97.47 is not a prediction of ninety-seven-times faster training and must not be advertised that way.

Now suppose the training batch has 64 examples. It would be misleading simply to multiply the candidate calculation by 64 without checking whether some sampled candidates are shared across the batch or whether matrix multiplication can reuse work. Implementation details affect memory access and throughput. Measure the actual number of sampled and true classes, wall-clock time, device utilisation and quality on the specific architecture.

If the actual computational bottleneck is a very large transformer backbone rather than its output projection, reducing the classifier head’s arithmetic may have limited impact. A sound profiling decision identifies the expensive operation first and only then chooses whether sampled softmax addresses it.

Choosing a candidate sampler is a modelling decision

Uniform sampling gives every class the same candidate-selection probability before accounting for any rejection or uniqueness rule. Log-uniform or frequency-informed sampling can emphasise some classes over others. A good sampler may cover informative alternatives cheaply, but each option produces a different expected-count correction and gradient distribution. The sampling distribution belongs in the experimental record.

Popularity-based sampling can be attractive in a long-tailed vocabulary because common outputs occur often, but very rare labels may then be seen too infrequently as negatives. A model could leave their relative scores poorly learned. Conversely, deliberately hard negative sampling may focus training on confusing alternatives but alter the probability model if corrections do not reflect the new selection law.

Sampling must be linked to stable class identities. When a vocabulary expands or indices are remapped, cached candidate distributions and weights may become invalid. The system should version the class map and sampler state together. A saved model whose output row 412 once meant a different token than the current tokenizer can be fundamentally wrong even if the matrix dimensions match.

Random seeds enable reproducible experiments but do not remove stochastic variation. One candidate sample may expose a high-scoring wrong class while another misses it. Compare candidate distributions across training periods, and use independent seeds where appropriate to determine whether an apparent gain is robust.

The sampling decision is separate from the final decoding decision. A model may be trained with sampled softmax but later use a different inference-time method to choose tokens. If the inference step calculates a full vocabulary distribution, the training sampler should not be described as the production top-p or temperature sampler. These tools solve different problems.

Sampled softmax is not the same as negative sampling

The phrase negative sampling is often used for methods that construct a binary distinction between an observed pair and deliberately selected alternatives. Such objectives can use logistic losses rather than a normalised multi-class softmax likelihood. Their gradients and interpretations differ from a sampled-softmax approximation, even if both avoid scoring every output class in one training update.

The existing negative sampling guide explains questions such as how to select informative alternatives and avoid false negatives. This article instead asks how to approximate a large-vocabulary multi-class normalisation while preserving a declared relationship to the full softmax model. A new model builder should not treat these as interchangeable API names.

Noise-contrastive estimation is another distinct technique. It reformulates learning in terms of discriminating observed data from noise under a specified statistical relationship. In some applications it supports learning unnormalised models. But sampled softmax and NCE have different losses and corrections, as shown in the TensorFlow candidate-sampling reference. See the separate noise-contrastive estimation guide.

Why does the distinction matter? An engineer can copy a negative-sampling formula into a multiclass training program and receive plausible decreasing losses while changing the meaning of the trained scores. The model may still be useful, but the probability interpretation and comparisons to a full softmax baseline need fresh analysis.

Always name the objective mathematically. Ask whether it is a normalised multi-class likelihood, a sampled approximation to that likelihood or a binary data-versus-noise classifier. The number of negative candidates alone does not identify which objective is being optimised.

Train approximately, evaluate the actual target question

A sampled-softmax training curve is not a substitute for an evaluation on the intended output task. Because the sampled objective can underestimate the full loss, apparent progress might partly reflect how the candidates were selected. For a manageable evaluation batch, calculate full logits and the full softmax loss when the model design permits, and report the exact evaluation procedure and denominator.

For large-vocabulary language prediction, different metrics matter for different reader jobs. Per-token held-out negative log-likelihood measures probability assigned under correct reference prefixes. Free-running generated text requires separate assessment of meaning, factuality, stopping and task completion. A model can achieve one without the other. Do not confuse the training shortcut with a guarantee about generated output quality.

Watch for changes in vocabulary over time. If new terms appear or some classes are much rarer in deployment, a model may perform poorly on the tail even if aggregate loss looks good. Evaluate common and rare groups separately with their actual counts, not merely an average dominated by popular labels.

A reliable candidate selection experiment compares a full-softmax baseline, a sampled-softmax candidate and perhaps a different candidate count under controlled training budgets. But a fair comparison should specify whether budgets are equal in updates, tokens, computing time or energy. Holding one constant may change another, so the reader needs both quality and resource data.

The final criterion is what the application requires. A search-ranking tool may care about retrieving the true next item among many options; a generated-language system may care about coherent and accurate sequences. An inexpensive approximate training step is valuable only when it contributes to the declared outcome.

Ten ways a sampled classifier can look better than it is

Failure one is treating a low subset loss as an exact full softmax loss. The four-class example proves why that is wrong: changing which classes appear in the denominator changes the value. Repair it with an independent full-loss evaluation and clear labels on every plotted metric.

Failure two is using the wrong expected-count correction. A sampler that draws without replacement is not automatically equivalent to one with replacement, and the true-class expected count may need specific handling. Repair it by following one complete documented sampler-and-loss contract rather than composing unrelated formulas.

Failure three is allowing accidental hits to become unlabelled negatives. Repair it with a tested true-class exclusion or masking rule. Failure four is breaking vocabulary identity when reindexing output classes; repair it with a versioned class map linked to weights and labels.

Failure five is selecting candidates from the final test distribution while claiming the training sample was independent. Repair it by fitting any sampler parameters only from authorised training information. Failure six is concluding that a 97-fold reduction in theoretical projection multiplications means a 97-fold reduction in real training time. Repair it by profiling actual total computation.

Failure seven is using an aggressively skewed sampler that neglects rare but important alternatives. Repair it through controlled distribution comparisons and rare-class evaluation. Failure eight is changing the sampler, optimiser and architecture simultaneously, then attributing every difference to sampled softmax. Repair it with properly matched experiments.

Failure nine is forgetting that the output layer may not be the bottleneck at all. Repair it by identifying where time and memory are spent. Failure ten is assuming a good full-vocabulary predictive score means a generated output is safe or factually correct. Repair it with independent task-level verification.

Most of these failures begin with a missing definition rather than a sophisticated mathematical error. Specify which classes were sampled, what ‘negative’ means, how probabilities were corrected and which evaluation actually measures the user’s desired result.

A realistic decision with limited review capacity

Imagine a fictional document-tagging model with 40,000 possible labels and a pipeline that must classify ten thousand new documents each day. The team wants to reduce training cost, not weaken any mandatory review of high-impact categories. It trials an approximate sampled training objective using 1,024 alternatives per batch while retaining a separate full-distribution evaluation subset.

Suppose on 1,000 independently labelled held-out documents the baseline selects the correct top label for 820 records and the sampled candidate for 835. Their top-one accuracies are 820/1,000=82% and 835/1,000=83.5%, a difference of 1.5 percentage points or fifteen records. Those are invented counts, not an actual benchmark result.

The candidate’s training may be faster, but the top-one comparison alone does not reveal whether its wrong labels are more costly, whether rare categories deteriorated, or whether its scores are calibrated. The team should compare per-class results, confusion patterns and the operational action that each label triggers. A small aggregate gain can coexist with a severe decline on one rare safety-related category.

Now suppose the candidate’s output head consumes 20% of the original training time and sampled computation makes that head four times faster. Holding all other work constant, the new total time is approximately 80%+20%/4=85% of original, a 15% overall improvement, not a fourfold end-to-end speed-up. This is a simple version of the bottleneck principle, with explicitly assumed unchanged non-head costs.

The study becomes useful when the team publishes both quality and resource denominators. A model that needs fewer class comparisons but misses important rare categories could be less fit for purpose. A model that preserves accuracy while reducing total training time by a measured amount offers a defensible engineering benefit.

A practical sampler audit should use an explicit small probability table rather than uninspected random calls. Imagine four eligible negative classes with sampling probabilities 0.50, 0.25, 0.15 and 0.10 under a single-draw distribution. These probabilities add to one. If two independent draws with replacement are made, the expected draw count of the first class is 2×0.50=1, while the fourth class has expected count 2×0.10=0.20. The numbers are expected counts, not guarantees about which labels appear in one particular draw.

If the implementation instead requires two distinct sampled classes without replacement, those expected inclusion probabilities differ from the independent-draw calculation. Simply reusing 1 and 0.20 as if they were universal expected counts would make the correction inconsistent with the sampling process. Check whether the library documents expected occurrences, inclusion probabilities, or another convention, and test the actual sampler output before interpreting any corrected score.

For a high-frequency class, drawing the same candidate twice is entirely possible in the with-replacement toy model. The chance of drawing the first class twice in two independent draws is 0.50×0.50=0.25, or 25%. A routine that silently removes the duplicate creates a different candidate set than a routine that retains both draws. This is why the negative candidate count printed in a configuration file does not necessarily equal the number of distinct classes in an effective loss.

The same reasoning applies to rare candidates. Under two draws with replacement and probability 0.10, the probability of drawing the fourth class at least once is 1−(1−0.10)²=1−0.81=0.19, or 19%. That differs from its expected draw count of 0.20. An explanation that interchanges probability of inclusion and expected count will be quantitatively wrong even though the figures are superficially close.

These calculations are deliberately restricted to a transparent toy sampler. Real library candidate samplers may use different distributions, uniqueness rules, rejection methods and expected-count conventions. The useful skill is to identify which random experiment the algorithm implements before making a corrective claim. Model quality still requires the independently evaluated full task rather than a clever calculation about sampled candidates alone.

What to check in the training interface

TensorFlow’s current documented tf.nn.sampled_softmax_loss expects class weights, biases, labels, input activations, the number of sampled candidates and total class count. Its optional sampled_values data specifies the actual candidate IDs and the expected counts for true and sampled classes. Those fields are not generic descriptive metadata; they determine the training computation.

Check tensor shapes against the documented interface. An output weight matrix is indexed by class, while each input activation vector is indexed by feature dimension. The positive labels must have the expected integer type and shape. A model can produce finite loss values with a misaligned class map; the arithmetic running is not proof that the intended label semantics are correct.

Use a tiny test where exact softmax can be computed manually. Compare full and naive subset losses and verify that a sample correction uses the candidate distribution specified by the implementation. Test accidental hits and duplicates explicitly. Then check numerical stability when some logits are large. Standard log-sum-exp techniques reduce overflow or underflow; simple unguarded exponentiation is unsuitable for serious training.

Record which sample count and candidate distribution were used, whether candidates are shared across batch items, and how distributed workers coordinate state. In a multi-worker system, inconsistent class vocabularies can be disastrous: one worker’s index may refer to a different class. Fix vocabulary identity before tuning candidate count.

The strongest debugging evidence is a reproducible unit test tied to a particular library version, plus a separate end-to-end validation run. Do not cite a documentation function signature as proof that a particular project executed it correctly.

Changed-context exercise: scaling from four labels to one hundred thousand

Begin with the four-class toy logits (2,1,0,−1). The full target loss was approximately 0.440190 nats. Omitting class B produced an uncorrected subset loss of about 0.169846. Explain why the difference does not represent learning: the parameters and true label were unchanged, while the normalising comparison lost an alternative with substantial model score.

Now change the output space to 100,000 classes and the feature dimension to 128. A naive full dense output projection requires about 100,000×128=12,800,000 component multiplications for one example. With one target and 999 sampled alternatives, it uses 1,000×128=128,000 corresponding component multiplications. The simplified projection-only ratio is 12,800,000/128,000=100.

Does the new model therefore train one hundred times faster? No. The backbone, sampler, memory traffic and other operations still consume resources. If the head were only one tenth of full-model time and its compute fell one hundredfold with everything else unchanged, total time would be approximately 90%+10%/100=90.1% of original, an improvement of about 9.9%. This answer differs from the projection-only ratio because the denominator is the entire training process.

Change the labels again: some examples have three equally valid target classes rather than one. The training interface must specify whether num_true supports those targets and what meaning they have; it is not enough to pretend each record has exactly one correct class. If the real task is multilabel independent classification, a sampled multiclass-softmax construction might be the wrong objective altogether.

The lesson transfers from a simple four-class probability to large-vocabulary systems: define the objective first, keep sampling semantics explicit, compare resource calculations with actual profiling and verify predictions on a representative complete evaluation task.

Where this approximation is not a good solution

Sampled softmax can be unnecessary when the output vocabulary is small or when modern hardware makes the full projection inexpensive relative to the rest of the network. Introducing stochastic correction and class-sampling complexity may make training harder to reason about without materially reducing resource use.

It can also be inappropriate when the application requires exact normalised probabilities at every training step or when a small candidate set poorly represents high-scoring alternatives. The training approximation may bias updates, and a rare but consequential class may be insufficiently compared. Improving sample design requires evidence rather than assuming every candidate budget is adequate.

The method does not repair erroneous labels, data leakage, unfair representation, obsolete vocabularies or missing source evidence. Those issues can produce confident but incorrect predictions regardless of how many negative candidates are sampled. Fixing the relevant upstream problem may provide greater value than reducing the cost of a flawed classifier.

Because implementations differ across library versions, users should check actual current source documentation before adapting parameter names, shapes or correction formulas. A remembered example from an older compatibility API may not describe the exact runtime in use. This article intentionally explains principles without pretending that the code was executed against an installed TensorFlow environment.

The examples and performance figures are labelled constructions. They explain what a calculation means; they do not claim benchmark improvements or measured speed-ups for an external product.

Frequently asked questions

What is sampled softmax? It is an approximate training objective for classifiers with many possible classes, using the true labels and a selected sample of alternatives rather than a full comparison at every training step.

Does it give an exact normalised probability? Not generally. The finite candidate objective differs from full softmax, so full-vocabulary evaluation may still be needed.

Why do expected counts matter? Candidates can be selected with unequal probabilities. Corrective terms must reflect the actual sampling law to give the intended approximation.

What is an accidental hit? It occurs when a sampled negative candidate equals a true class. An implementation must handle it according to its defined objective, often removing the accidental hit.

Is sampled softmax identical to negative sampling? No. Their losses and probabilistic meanings can differ even though both use selected alternatives.

Is sampled softmax identical to noise-contrastive estimation? No. NCE frames a data-versus-noise discrimination objective with distinct mathematics.

Will using 512 candidates make training 100 times faster? Not necessarily. Projection arithmetic may shrink greatly, but the rest of the model and candidate overhead still contribute to runtime.

Should I use sampled softmax during inference? The documented operation is designed for training; evaluation and inference should follow the intended actual model and task requirements.

Can it work with rare categories? It can, but sample distributions and tail performance require checking. Sparse candidate exposure can weaken learning of important alternatives.

How should students study this? Calculate exact softmax on four logits, then remove one class and see why the loss changes without a parameter update.

Reader routes and original references

For learners, reproduce the four-class calculation and the 100,000-class multiplication example. Then explain why a cheaper calculation is not necessarily an accurate probability. The distinction between an efficient method and a verified result is a valuable general reasoning skill.

For engineers, build a tiny deterministic test with known logits, fixed candidate IDs, valid expected counts and accidental-hit scenarios. Preserve vocabulary identity across training workers and compare the selected model with full-vocabulary held-out measurements. For related but different mechanisms, read negative sampling and noise-contrastive estimation.

For reviewers, demand a clear separation between sampled training loss, full evaluation loss, task accuracy and measured computing cost. The training shortcut by itself does not establish any of those outcomes. The broader How AI Works guide explains how those measurements fit into a complete learning system.

Primary source materials: TensorFlow official sampled_softmax_loss documentation; the TensorFlow team’s Candidate Sampling Algorithms Reference; and the original large-vocabulary neural translation work described in the TensorFlow API reference. These sources define the actual function and approximating objective. All numerical examples here are independently written teaching cases, not reports of executed model training.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading