eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Noise-Contrastive Estimation — Unnormalised Densities, Noise Ratios and Logistic Learning

—

by

eduKate Asia
A smiling student in a blue pinafore holds a pencil over an open book at a classroom desk, with textbooks, a whiteboard and a sunlit window nearby.

Noise-contrastive estimation, or NCE, trains a probabilistic model by asking a classifier to tell real observations apart from deliberately generated noise. Instead of evaluating a prohibitively expensive normalising constant inside every ordinary maximum-likelihood update, the learner uses an explicit noise distribution and a logistic discrimination objective. With suitable assumptions and model capacity, the method can estimate the parameters of an unnormalised density, including its normalisation term.

The essential idea is more subtle than “make real examples score higher than fake examples.” The classifier’s correct odds depend on the actual data density, the probability that noise would produce the same example, and the number of noise samples per real sample. Neglecting any of these terms changes what the model learns. This article walks from first principles to hand-checked counts, stable optimisation, diagnostic failures and transfer to conditional models, while distinguishing classical NCE from word-embedding negative sampling and modern InfoNCE.

Contents

Why normalising a probability model can be hard

A probability distribution must assign nonnegative mass and sum or integrate to one. Suppose a model gives an input x an energy-based score sθ(x). We might write a candidate density as pθ(x) = exp(sθ(x)) / Zθ, where Zθ sums exp(sθ(u)) over every allowed outcome u, or integrates over a continuous space. The score may be simple to calculate for one observed x even while the sum over all u is enormous.

If a language model had to enumerate all possible sequences of arbitrary length to obtain one global partition function, the straightforward sum would be intractable. In other settings, a statistical model has interactions among many variables, and its partition function couples all possible configurations. Some familiar neural networks avoid this difficulty by using locally normalised conditional probabilities; NCE matters particularly when the chosen model is explicitly unnormalised.

Maximum likelihood asks the model to increase probability on observed inputs, but its gradient often includes expectations under the model distribution. That expectation can require sampling, approximation or specialised mathematical structure. NCE changes the problem: compare observed inputs with inputs sampled from a known noise distribution, and learn a boundary with the correct density-ratio form. The hard global normaliser is represented as an estimable model parameter in the classical construction.

Define data, model and noise before training

Call the unknown real-world data distribution p_data(x). We observe independent or otherwise appropriately sampled training examples from it. We choose a tractable noise distribution q(x): one from which examples can be drawn and whose probability or density can be calculated at each sampled point. This last requirement matters. A generator that can emit random-looking samples but cannot supply q(x) may not support the classical objective.

Let k be the number of noise samples drawn per observed data example. The mixed classification dataset contains one positive for every k negatives, so its prior probability of the “data” class is 1/(k+1), and its prior probability of “noise” is k/(k+1). These are sampling-design probabilities, not claims about the frequency of genuine events outside the experiment.

We fit an unnormalised score uθ(x) and a separate log-normaliser parameter c, so log pθ(x) is represented by uθ(x) − c. In an ideal correctly specified model, c tends toward log Zθ. This formulation is helpful because uθ(x) can be evaluated for a particular x without summing over an entire space at every update. The optimisation must still be checked for identifiability and support assumptions.

Derive the classifier rather than memorising a loss

Consider a mixed example labelled D if it came from data and N if it came from noise. Bayes’ rule gives P(D|x) = p_data(x) / [p_data(x) + kq(x)]. The matching modelled classifier is Pθ(D|x) = pθ(x) / [pθ(x) + kq(x)]. The factor k is not decorative. It arises because the training mixture includes k times as many noise observations as data observations.

The corresponding log odds of data versus noise are log pθ(x) − log[kq(x)]. Replacing log pθ(x) with uθ(x) − c gives the trainable logit aθ(x) = uθ(x) − c − log k − log q(x). Pass aθ(x) through the logistic sigmoid to estimate the data-class probability. Negative examples use the complementary probability, sigmoid(−aθ(x)).

The empirical objective maximises the average log probability of correct binary labels: log sigmoid(aθ(x)) on actual observations and log sigmoid(−aθ(z)) on sampled noise points z. The optimiser receives a useful gradient without enumerating all model outcomes each step. The original work studies conditions under which this estimator converges to the desired parameters; in real applications those assumptions require scrutiny.

A categorical case you can check by hand

Imagine exactly 100 real observations in three categories: A occurs 60 times, B 30, and C ten. The empirical data probabilities are therefore 60/100 = 0.60, 30/100 = 0.30 and 10/100 = 0.10. Choose uniform noise with q(A) = q(B) = q(C) = 1/3, and draw two noise samples per real observation, so k = 2.

For category A, the ideal posterior probability that a mixed-sample occurrence came from real data is 0.60 / [0.60 + 2 × (1/3)] = 0.60/1.266666… = 0.473684…. For B the answer is 0.30/[0.30 + 0.666666…] = 0.310345…. For C it is 0.10/[0.10 + 0.666666…] = 0.130435…. These fractions can be checked by multiplying each numerator and denominator by 30 or another common scale.

Notice the surprise: A is the most common real category, yet even an A observation has a posterior below one half of coming from the data class. There are twice as many noise draws as real draws, and the noise generator chooses A one third of the time. A weak explanation might call the classifier “wrong.” The correct explanation is that class priors and sampling rates have changed the denominator.

Reproduce the same result with count expectations

The experiment contains 100 data observations and, by design, 200 noise draws. Uniform noise produces an expected 200/3 ≈ 66.667 occurrences of A. In the mixed sample, the expected A count is 60 + 66.667 = 126.667. The fraction of those A occurrences labelled real is 60/126.667 ≈ 0.473684. That matches the density calculation.

For B, there are 30 data cases and 66.667 expected noise cases. Their ratio is 30/96.667 ≈ 0.310345. For C, the ratio is 10/76.667 ≈ 0.130435. These are expected counts, not a claim that a finite 200-draw random sample must contain a fractional observation. An actual simulation will fluctuate around these expectations, with variance controlled by the noise distribution and sample count.

The main lesson is to preserve denominators. If the same 100 real observations were paired with 400 noise points, k would become four and every ideal classifier probability would change. A researcher should not compare raw discrimination accuracies across different k values as if the class mixtures were identical.

Recover a logit and a normalisation constant

Use an unnormalised score proportional to the simple category weights 6, 3 and 1. Their sum is Z = 6 + 3 + 1 = 10, so the normalised probabilities are 0.6, 0.3 and 0.1. Write u(A) = log 6, u(B) = log 3, u(C) = log 1, with c = log 10. Then the model log-probability for A is log 6 − log 10 = log 0.6.

With k = 2 and uniform q = 1/3, the classifier logit for A becomes log 0.6 − log(2/3) = log 0.9 ≈ −0.10536. The sigmoid of that value is 0.9/(1+0.9) = 0.473684, as before. For B the corresponding density ratio to the weighted noise is 0.3/(2/3) = 0.45; the sigmoid of log 0.45 is 0.45/1.45 ≈ 0.310345.

This demonstration is deliberately small enough that the normaliser can be calculated directly. In the problems NCE was designed to address, c represents something expensive to obtain by direct enumeration. NCE does not make the partition function irrelevant; it changes how the normalisation information is learned. A valid test must ask whether the learned density actually agrees with independent held-out data, not merely whether training labels are separable.

Why the noise distribution matters so much

If q puts almost no probability where real observations occur, separating the two classes becomes trivial. A discriminator can identify the sampling process without learning fine distinctions in the data-rich region. At the other extreme, a q close to p_data creates harder classification examples and potentially informative gradients, although the exact statistical efficiency depends on the problem and the computational cost of sampling and evaluating q.

Support matters mathematically. If q(x) = 0 at an x that real data can produce, log q(x) is undefined and the proposed density-ratio classifier cannot be used as written. A practical noise design should cover the relevant support and avoid numerical probabilities so tiny that logs and gradients become unstable. It may use a smooth broad reference distribution, a mixture with a broad component, or a task-appropriate discrete sampler.

Do not confuse q with the learned pθ. The noise distribution is an explicit part of the estimation procedure; the model distribution is the object being fitted. If q is changed midway through training, the logit correction and experiment interpretation must change too. It is not legitimate to switch to harder negatives while retaining a stale log q value or claiming the original theoretical analysis applies unchanged.

Choose the noise ratio with costs in mind

Let k = 5 and suppose there are 10,000 data examples. Training generates 50,000 noise points and a 60,000-item classification dataset, of which 10,000/60,000 = 1/6 ≈ 16.67% have the data label. A classifier that always predicts “noise” scores 50,000/60,000 = 83.33% accuracy without learning anything about the density ratio. This is why unqualified accuracy can be misleading.

As k increases, the number of negative comparisons and evaluations of q typically grows. More negatives may improve aspects of estimation but consume memory, compute and data throughput. In models with expensive uθ(x), the marginal cost of another noise example can dominate. Choosing k is therefore a statistical and systems trade-off, not a contest to maximise the negative count.

Report k in methods, logs and evaluation tables. Also report the number of real observations, the realised number of noise draws and any importance or class weights. A useful experiment compares validation log-likelihood where tractable, density-ratio diagnostics, held-out classification calibration, downstream utility and compute at equal budgets. It is not enough to say that one configuration achieved “higher accuracy” on a different positive-to-negative mixture.

Logistic loss and stable numerical computation

The expression log sigmoid(a) becomes numerically hazardous when a has large magnitude if computed by first forming sigmoid(a) and then taking its logarithm. Stable implementations use identities based on softplus, for example log sigmoid(a) = −softplus(−a) and log sigmoid(−a) = −softplus(a). These forms avoid some underflow and catastrophic cancellation.

The logit itself should be built from log probabilities or log densities, never from a small q(x) obtained by exponentiating an enormous negative log density unnecessarily. A well-engineered implementation checks that uθ(x), c, log q(x) and log k are finite. It also handles invalid observations deliberately rather than substituting zeros for missing densities and silently changing the objective.

Gradient clipping can help with exploding updates but does not repair a systematically wrong noise model or an incorrect factor k. Likewise, a learning-rate schedule cannot compensate for accidentally using q sampled under a different version from the q evaluated by the classifier. First validate the equation and sampling contract with a tiny enumerated distribution, then scale to large data.

What consistency means in the original research

Gutmann and Hyvärinen’s 2010 conference paper and expanded 2012 journal article analyse NCE for unnormalised statistical models. Under appropriate assumptions, the estimator is statistically consistent: as the amount of suitable data increases, its estimates approach the intended parameters. That does not mean every finite dataset, optimiser, neural parameterisation or approximate sampler must recover the truth.

Correct model specification is a major condition. If p_data is outside the family pθ, optimisation targets a best approximation under the chosen objective and sampling scheme rather than an unattainable true density. Dependence between observations, drifting data, limited support, approximate q values and imperfect optimisation can create further gaps between theory and implementation.

An engineer should separate three questions: does the mathematical objective identify the desired quantity, does the numerical optimiser minimise that objective adequately, and does the trained distribution work on the real downstream task? A training curve may answer part of the second question. It does not, by itself, settle the first or third.

Comparison with ordinary maximum likelihood

Maximum likelihood maximises the sum of log pθ(x) on data. If pθ is normalised with an efficient local operation, this is often a natural objective. NCE is attractive when the normaliser is intractable or expensive to compute, while evaluating a score and a known noise density remains practical. The trade changes the required computations and statistical properties.

In a finite three-category problem like the one above, directly normalising the scores is so cheap that NCE is unnecessary. The toy helps explain the equations, not justify deploying a complicated estimator where a softmax over three outcomes already solves the problem. In a high-dimensional energy-based model, the distinction may be decisive, because summing over configurations grows rapidly.

NCE should also not be marketed as a guaranteed approximation to full maximum likelihood at every k and dataset size. Different objectives can have different finite-sample variance and bias, and optimisation constraints matter. The journal research analyses the relationship, including behaviour with larger noise sample sizes, but practical selection remains empirical and problem-specific.

NCE versus negative sampling in word embeddings

The negative-sampling objective popularised with word2vec also contrasts observed word-context pairs against sampled alternatives using logistic terms. That resemblance is real and useful for intuition. But the word2vec negative-sampling procedure does not automatically estimate a globally normalised probability model or an explicit partition-function parameter in the same way as classical NCE.

A word-embedding model may learn vector geometry helpful for analogy, similarity or downstream classification while not producing calibrated token probabilities. Its negative-sampling distribution is a design choice affecting which distinctions the vectors learn. The original Mikolov and colleagues paper introduces negative sampling as a simplified training objective alongside other improvements to Skip-gram.

When comparing methods, ask what is ultimately needed: a density with meaningful probabilities, a ranking function, an embedding space, or a robust classifier. Those are different reader jobs. A method that succeeds at discriminating positives from noise is not necessarily validated as a probability estimator on an unseen population.

NCE versus InfoNCE and in-batch contrastive learning

InfoNCE is often associated with selecting a positive item from a set containing negatives, using a multiclass softmax-style objective. In contrastive predictive coding, a representation of context is used to score a true future sample against alternatives. This can learn features useful for downstream prediction without ever directly producing a normalised data density over all raw observations.

Classical NCE frames a binary discrimination problem using the known noise probability q(x) and the sampling ratio k in a particular density-ratio logit. These formulations share a family resemblance but should not be treated as synonyms. Their loss functions, estimands, choice of negatives and interpretations differ. An article about InfoNCE-based representation learning is not a substitute for explaining how an unnormalised statistical model estimates its normaliser.

Batch negatives deserve special caution. If other items in a batch are not independent noise draws from the assumed q, or if false negatives are common, the classifier may learn shortcuts. For example, two photos of the same person can be incorrectly labelled as a positive-negative pair. Choosing a larger batch does not automatically solve mislabeled relationships.

Sampling bias and false shortcuts

A contrastive learner can succeed at distinguishing datasets rather than the underlying concept. Suppose positive medical scans were saved with one compression level and generated negatives with another. A classifier might learn compression artefacts, not anatomy. In classical NCE the goal is to identify the model distribution relative to q; in related contrastive systems, unwanted artefacts can still corrupt useful feature learning.

The audit question is whether the discriminator remains informative when superficial dataset fingerprints are removed. Match preprocessing across positives and negatives where the estimation design permits. Examine distributions of resolution, missing values, timestamps, source devices and annotation artefacts. A negative sampler that draws only obviously impossible values is likely to give deceptively easy training signals.

When a model generalises poorly, the repair is not always a larger backbone. Check that q was evaluated exactly at sampled values, that all inputs passed through comparable transforms, and that the validation split blocks information leakage. The mathematical derivation assumes meaningful observed distributions; it cannot protect against a contaminated pipeline by itself.

A second worked example with an asymmetric noise source

Return to the 100-category observations A = 60, B = 30, C = 10, but choose q(A) = 0.20, q(B) = 0.30, q(C) = 0.50. These sum to one. Keep k = 2. The ideal data-class posterior for A becomes 0.60/[0.60+2×0.20] = 0.60/1.00 = 0.60. For B it is 0.30/[0.30+0.60] = 1/3. For C it is 0.10/[0.10+1.00] = 1/11 ≈ 0.0909.

The real data did not change, but the classification probabilities did because noise now generates many more C values and fewer A values. If a learner ignored q and trained against raw positive-negative labels as though all noise categories were equally probable, it would conflate the chosen generator with the data density. This is why the correction term −log q(x) is central, not an optional flourish.

Expected noise counts across 200 draws become 40 A, 60 B and 100 C. The mixed-category data fractions are then 60/(60+40) = 0.60, 30/(30+60) = 1/3 and 10/(10+100) ≈ 0.0909. These calculations offer a compact unit test: if a hand-built implementation cannot reproduce this toy scenario, a large training run is not trustworthy.

Experimental design and honest evaluation

Split the available real observations into training, validation and test groups that reflect the deployment question. If examples cluster by individual, source document or time, split by those groups where leakage would otherwise inflate scores. Training and evaluation draws from q should be tracked separately, especially if a learned or adaptive negative proposal is being used.

Choose evaluation metrics for the intended statistical target. On a small finite dataset, compare recovered category probabilities with the known empirical distribution. On a continuous density, use reliable held-out likelihood approximations or domain-appropriate goodness-of-fit checks, acknowledging when normalisation is still approximate. If the goal is representation quality instead, use a separate downstream probe rather than relabelling discrimination accuracy as density quality.

Report uncertainty where possible. A point estimate of a learned parameter does not communicate how sensitive it is to sampling fluctuations. Repeated seeds, confidence intervals, bootstrap analysis where assumptions justify it, and stress tests for q can reveal whether apparently superior performance is stable. No one metric substitutes for checking support, calibration, optimisation and downstream transfer.

Failure: the negative distribution has zero support

Suppose observations include a rare but legitimate class R, while q(R) = 0 because the negative generator never emits it. Then log q(R) is undefined and the classical classifier is not well specified there. An implementation might silently replace zero with 10^−12 and continue, but that changes the specified procedure and can produce extreme logits dominated by an arbitrary numerical constant.

One repair is a mixture q_new(x) = (1−ε)q_old(x) + εq_broad(x), where ε is a chosen small positive weight and q_broad has support over the relevant region. This may improve numerical and theoretical coverage, though sampling efficiency and evaluation cost need reassessment. Document the chosen ε and rerun the toy tests for the new q.

The changed distribution must be applied consistently in sampling and the log density evaluated by the objective. If training code samples from q_old but supplies q_new in its denominator, it is solving a mismatched estimation problem. Tracing both the sampling method and its probability evaluator in the same versioned configuration is a good defence against this subtle bug.

Failure: classification accuracy looks too good

With k = 9, there are nine noise examples for each real observation. A classifier always predicting “noise” achieves nine correct predictions out of ten mixed samples, or 90% accuracy. That says nothing about the model’s fit to p_data. A dashboard dominated by this accuracy can hide a broken positive-class classifier and create false confidence.

Check per-class log loss, calibration against known posterior probabilities, and density recovery on an enumerated synthetic case. Consider how the evaluation changes if the same model is tested with k = 1 or k = 5. Because class priors shift, unadjusted accuracy is not comparable. Define a common target mixture or compare an invariant estimand with appropriate corrections.

If large k improves accuracy solely by increasing the majority-class share, adding more negatives is not a modelling achievement. Spend some effort on diagnostics that are insensitive to this simple class imbalance. The number of real examples, their effective diversity and the fidelity of q still matter.

Failure: the estimated normaliser drifts

An unnormalised score can trade off with its scale or additive constant in ways that complicate optimisation. If the fitted log-normaliser c changes dramatically while apparent discrimination loss decreases, inspect parameter identifiability, gradient signs, and whether the noise term is present for both classes. Make sure c is trained rather than accidentally detached from the computational graph.

On a toy distribution with known Z = 10, compare the recovered c with log 10 ≈ 2.302585 when the score parameterisation is fixed as the log weights. If the score itself is free to shift by a constant, c may shift with it, so validate the resulting probabilities rather than insisting on one arbitrary raw offset. The relationship among score, normaliser and density is the object of interest.

Numerical stability, insufficient support and optimiser settings can all create drift, but they demand different repairs. Add unit tests for the analytic gradients, verify the meaning of batch averaging, inspect finite values and evaluate independent density ratios. Do not use gradient clipping to conceal a systematic equation error.

Transfer problem: learning rare event patterns

A security analyst observes 200 real events of three kinds: ordinary O = 170, unusual U = 25 and rare R = five. They choose a noise generator with q(O)=0.50, q(U)=0.40 and q(R)=0.10, and draw k=4 noise events for each real event. What are the ideal data-class posteriors, and would a high binary classification accuracy prove good rare-event estimation?

First turn counts into data probabilities: p(O)=170/200=0.85, p(U)=25/200=0.125, p(R)=5/200=0.025. The classifier posterior for O is 0.85/[0.85+4×0.50] = 0.85/2.85 ≈ 0.298246. For U it is 0.125/[0.125+1.60] = 0.125/1.725 ≈ 0.072464. For R it is 0.025/[0.025+0.40] = 0.025/0.425 ≈ 0.058824.

Even O is more likely to have come from the noise-labelled half of the comparison because four negatives are drawn per positive and q favours O substantially. The design has 200 positives and 800 negatives, so a noise-only classifier scores 80% accuracy. That baseline means an 81% score is weak evidence by itself. To assess rare-event modelling, examine the recovered density ratio for R, the true test base rate and the action consequences of false positives and false negatives.

Transfer problem: changing the noise ratio after deployment

Suppose the same trained density model is evaluated through a classifier with k changed from four to one, while q and pθ remain fixed. The data-class posterior must change: for category R it becomes 0.025/[0.025+0.10] = 0.20, not 0.058824. The underlying real-world probability of R remains 0.025. Only the experiment’s sample mixture changed.

This distinction is essential when a classifier score is shown to an operator. A 20% posterior of “real source” in a designed contrastive task is not automatically a 20% chance of fraud, illness or another real event. The sampling scheme does not equal the operational base rate. The same conceptual issue arises in case-control studies and other deliberately rebalanced training sets.

A robust transfer plan documents the new sample ratio, recalculates the required logit correction, and checks the model’s outputs on independent real-world data. If the target environment changes more than k—new population, different noise distribution, altered input processing—fresh validation is mandatory.

Deployment limits and ethical boundaries

NCE is a statistical training technique, not a magic test of truth. It does not prove that model outputs are safe, fair or causally meaningful. A high learned score may reflect training-data frequency or a sampling artefact rather than evidence about an individual. Use additional domain-specific validation, security controls and human review where decisions affect people.

On privacy-sensitive material, drawing noise examples from real records can itself expose information if sampling and logging are careless. Training pipelines should follow access restrictions, data minimisation and retention rules. Negative examples are data with provenance, not disposable anonymous objects by definition.

Cost must also be considered. The method trades an intractable normalisation calculation for repeated noise sampling and score evaluation. If q is expensive or low quality, or if the model already has a tractable normalised objective, NCE may be an inferior choice. A responsible engineering decision compares statistical efficiency, compute, interpretability, operational risk and the downstream task.

Frequently asked questions

Is NCE only for neural networks?

No. The original formulation is about estimation of parameterised statistical models, including unnormalised ones. Neural networks can supply the score function, but NCE’s discriminative principle is not restricted to one architecture. The key requirements concern model form, known noise density, support and valid optimisation.

Does NCE always output calibrated probabilities?

No. A binary discrimination score can be well fitted to its artificial sampling task without establishing externally calibrated event probabilities. The classical method has theoretical properties under specific assumptions, but finite-data neural systems still need independent checks of density recovery and calibration.

How many negative examples should be used?

There is no universally optimal k. More negatives can change statistical efficiency and gradient information but increase cost. Evaluate multiple values at comparable compute budgets, preserve the correct k factor in the logit, and report the training mixture. Always compare against trivial majority-class baselines.

Why do we need to know q(x)?

The optimal classifier’s odds involve p_data(x) divided by kq(x). If q(x) is unknown or evaluated incorrectly, the learner cannot perform the classical density correction as derived. Some related contrastive objectives have different requirements; they should be named and analysed separately.

Is InfoNCE simply another name for NCE?

No. They share a contrastive lineage, but InfoNCE commonly uses a multiclass choice among positives and negatives for representation learning. Classical NCE uses a particular binary density-ratio classifier with a known noise distribution, often to fit an unnormalised statistical model. Their objectives and output interpretations differ.

Reader pathways and primary research

Start with the eduKate AI overview for a broader map of models and training: https://edukatesg.com/2026/08/26/how-ai-works/ . For positive pairs, negative samples and InfoNCE as a different specialist reader job, continue to contrastive representation learning: https://edukatesg.com/2026/09/17/how-contrastive-representation-learning-works/ . To check whether prediction scores are reliable probabilities, consult the separate probability-calibration guide: https://edukatesg.com/2026/10/24/how-ai-works-probability-calibration-reliability-diagrams-brier-scores-temperature/ .

The original NCE proposal is Michael Gutmann and Aapo Hyvärinen, AISTATS 2010, available from the primary conference proceedings: https://proceedings.mlr.press/v9/gutmann10a.html . Their expanded statistical analysis appeared in Journal of Machine Learning Research in 2012: https://www.jmlr.org/beta/papers/v13/gutmann12a.html . The word2vec negative-sampling source is Mikolov and colleagues, 2013: https://arxiv.org/abs/1310.4546 . Contrastive predictive coding and its InfoNCE lineage are described by van den Oord, Li and Vinyals: https://arxiv.org/abs/1807.03748 . All numeric examples above are explicitly constructed educational calculations rather than claims about a deployed model or a performed experiment.