eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Contrastive Divergence — Gibbs Chains, Positive Phases and Biased Updates

—

by

eduKate Asia

Contrastive divergence is a practical way to train certain energy-based probabilistic models when the exact likelihood gradient requires an expensive average over all configurations the model might generate. It begins with real data, takes a small number of stochastic reconstruction steps and adjusts the parameters to favour observed configurations over those reached by the short chain. The method is especially associated with restricted Boltzmann machines.

The essential warning is just as useful as the algorithm: a short Markov chain is not normally an exact sample from the model’s equilibrium distribution. CD-1 can learn useful patterns, but its update is generally biased relative to the full maximum-likelihood gradient. It must not be confused with noise-contrastive estimation, contrastive representation learning or the contrastive loss used to align images and text. This guide works through the mechanism with explicit probabilities, a hand-checkable tiny model and realistic diagnostics.

Contents

Why maximum likelihood becomes expensive

Suppose a model assigns a score to every possible binary image. An image with 100 binary pixels has 2 to the power of 100 possible visible configurations, an astronomically large number. The model does not only need to give high scores to images in the training set. It must also arrange for the relative probability of the enormous space of alternatives to behave sensibly. Normalising those scores requires a quantity called the partition function.

For a small model, one can enumerate every configuration and compute that normalising constant exactly. For a useful high-dimensional model, enumeration is usually impractical. The derivative of log likelihood contains a part that encourages the observed data and a part that depends on what the model itself would produce. The second part is difficult because it involves expectations under the model distribution, not merely evaluations of training examples.

Contrastive divergence replaces the expensive model expectation with statistics obtained after a short Markov chain started from training data. This is computationally appealing because the chain need not wander throughout an immense state space for every update. But that same shortcut creates a central question: what happens when the chain has not mixed and still remembers where it started?

That trade-off defines the method. CD is not a trick that makes the partition function disappear mathematically. It is a computational approximation to a learning signal that would otherwise be expensive to estimate.

Energy, probability and the partition function

An energy-based model assigns an energy E to a configuration. The probability of a configuration is proportional to exp(−E); lower-energy states receive greater probability before normalisation. To turn proportionality into a probability, divide by the partition function Z, the sum of exp(−E) across all configurations. The resulting probabilities must add to one.

In a binary restricted Boltzmann machine, a visible vector v represents observed variables, and a hidden vector h represents latent features. One common energy parameterisation is E(v,h) = −a·v − b·h − vᵀWh. The visible biases a, hidden biases b and weights W are learned. The model distribution over joint visible and hidden states is p(v,h) = exp(−E(v,h)) / Z.

The distinction between joint and visible probability matters. A training example provides v, not usually the hidden states h. To calculate p(v), sum p(v,h) over possible h. In a tiny model, this is easy; with many units, the partition function across all visible and hidden configurations becomes the expensive part.

Energy is a learned scoring convention, not a physical temperature measurement or an automatic measure of truth. A low energy indicates the model favours a configuration under its current parameters. It does not mean the configuration is ethically desirable, causally valid or appropriate to generate.

The restricted architecture makes conditionals tractable

The word restricted refers to the bipartite connection pattern. Visible units connect to hidden units, but there are no within-layer connections in the basic restricted Boltzmann machine. That structure makes hidden units conditionally independent given the visible layer, and visible units conditionally independent given the hidden layer.

For binary sigmoid units, p(h_j=1 | v) is the sigmoid of the hidden bias plus a weighted sum of visible units. Likewise p(v_i=1 | h) is the sigmoid of the visible bias plus a weighted sum of hidden units. The sigmoid function is 1 / (1 + exp(−x)). Its outputs lie between zero and one, making them suitable Bernoulli probabilities.

Conditional independence is a computational gift. Instead of sampling every hidden configuration by enumerating exponentially many joint possibilities, one can calculate each hidden activation probability and sample all hidden units in parallel. Then one can use the hidden sample to draw visible units in parallel. Alternating these conditional steps creates a block Gibbs sampling chain.

This factorisation does not make the full joint normaliser cheap. It only makes each conditional transition cheap. Confusing easy transitions with exact equilibrium sampling is one of the most common conceptual errors when reading descriptions of the algorithm.

The positive phase: what the training observations support

Consider a visible training example v. The positive phase asks how strongly that example activates hidden features under the current model. In a binary RBM, the expected association for weight W_ij under the data-conditioned distribution is v_i times p(h_j=1 | v). If the visible unit is on, the association depends on how likely the hidden unit is to turn on; if the visible unit is off, the product is zero.

Across a batch, positive-phase statistics are averages over training cases. They are not percentages unless you explicitly interpret the binary units as events and state the denominator. If a batch contains 80 examples and the sum of expected visible-hidden products for one pair is 48, its average positive statistic is 48 / 80 = 0.60 association units per example.

This phase is generally straightforward because the visible examples are given and the hidden conditional probabilities factorise. The difficult part is deciding how much to subtract. If learning only increased weights for observed combinations, the model could raise scores indiscriminately and would not learn a coherent normalised distribution.

For intuition, the positive phase says make the data easier for the model to explain. The negative phase says do not make everything equally easy. Contrastive divergence keeps those two directions while approximating the negative part.

The negative phase: what the current model also likes

The exact log-likelihood gradient contains a negative-phase expectation under the model’s own joint distribution. For weight W_ij, this is the model expectation of v_i h_j. Conceptually, the learning rule compares the data-conditioned association against the model’s global association. If a pair occurs too often under the model relative to the data, a correction should make it less favoured.

Sampling correctly from the model may require many alternating Gibbs transitions. The samples need to escape the vicinity of their initial state and represent the distribution’s different modes in appropriate proportions. A short chain that begins at a training example can reconstruct nearby patterns without ever exploring distant high-probability states. Its statistics are then different from the exact model expectation.

CD-k uses k such transitions, where k is the number of complete alternating Gibbs updates under a specified convention. CD-1 is the common one-step example. Even CD-1 has stochastic operations: a Bernoulli probability of 0.75 does not mean a hidden unit deterministically equals one. It means repeated draws would yield one on average 75 percent of the time, under those conditions.

The negative sample can be replaced by a conditional expected hidden activation for a lower-variance estimate in practical implementations. A description should say clearly whether it uses sampled states or conditional probabilities, because mixing those conventions changes both variance and the details of the numerical trace.

A complete CD-1 cycle in plain language

Start with an observed visible example v0. Compute the hidden activation probabilities conditioned on v0. Sample hidden states h0 from those Bernoulli distributions. Next compute the visible reconstruction probabilities conditioned on h0 and sample a new visible vector v1. Finally compute the hidden activation probabilities conditioned on v1.

The positive statistic for a weight can be v0_i times p(h_j=1 | v0). The negative statistic can be v1_i times p(h_j=1 | v1). The update adds the learning rate times positive minus negative, averaged over the chosen mini-batch. Visible-bias and hidden-bias updates use their corresponding positive and negative activation differences.

That is the central recipe, but an implementation still needs more details: whether units are binary or continuous, whether biases are centred, whether the learning rate is scaled by batch size, whether momentum or weight decay is used and whether reconstruction probabilities or sampled states are tracked for monitoring.

The contrast is between the original case and a nearby reconstruction drawn from the model. It is not automatically a contrast between a positive training label and an external negative label. A CD model may be unsupervised and never receive class labels at all.

A correct implementation also separates an optimiser update from a Gibbs sampling transition. The transition changes a chain’s sample, while the optimiser changes learned parameters. Counting those as the same operation makes CD-k comparisons meaningless.

The smallest hand-checkable RBM

Take one binary visible unit v and one binary hidden unit h. Give both biases zero and set their interaction weight w = ln(3), approximately 1.0986. The joint unnormalised weight of a state is exp(wvh). The four possible states (v,h) = (0,0), (0,1), (1,0), (1,1) have weights 1, 1, 1 and 3 respectively.

Their normalising constant is Z = 1 + 1 + 1 + 3 = 6. Thus p(0,0)=1/6, p(0,1)=1/6, p(1,0)=1/6 and p(1,1)=3/6. These four probabilities sum to 6/6 = 1. The visible probability p(v=1) is (1+3)/6 = 4/6 = 2/3.

Now suppose the observed dataset for this toy demonstration contains one case v0=1. The hidden activation probability is p(h=1 | v=1) = sigmoid(ln3) = 3/4. Therefore the positive weight statistic v0 × p(h=1 | v0) is 1 × 3/4 = 0.75.

The exact model negative statistic E[vh] is the probability of state (1,1), because vh is one only in that state. It is 3/6 = 0.50. So the exact log-likelihood weight gradient for this single training observation is 0.75 − 0.50 = 0.25. We can know this exactly only because the model has four joint states.

Every number in this section is calculated for an intentionally tiny illustrative model, not drawn from an experiment.

Sampled reconstruction: two possible outcomes

From v0=1, the first hidden sample h0 equals one with probability 0.75 and zero with probability 0.25. If h0=1, the reconstructed visible unit v1 equals one with probability sigmoid(ln3) = 0.75. If h0=0, v1 equals one with probability sigmoid(0) = 0.50.

One possible sampled path is v0=1, h0=1, v1=0. Conditioned on v1=0, the hidden reconstruction probability is 0.50. The positive weight statistic is still 0.75, while the negative weight statistic is v1 × 0.50 = 0. This one sampled update is learning-rate times 0.75.

Another possible path is v0=1, h0=1, v1=1. For that path the negative statistic is 1 × 0.75 = 0.75, so the sampled weight update is zero. The algorithm does not produce exactly the full maximum-likelihood gradient 0.25 on every single path. It produces different short-chain updates depending on the stochastic draws.

It is helpful to write this out before programming. Otherwise a student may mistake a sample path for an average, or suppose that a Bernoulli probability can be treated as a deterministic hidden state. The update is an estimator; its distribution and expectation are part of the method.

Calculate the expected CD-1 bias exactly

Because the toy model has only one visible and one hidden unit, we can calculate the expected reconstruction distribution. The probability that v1=1 after one complete reconstruction is p(h0=1|v0=1) × p(v1=1|h0=1) plus p(h0=0|v0=1) × p(v1=1|h0=0).

Substituting the known fractions gives (3/4 × 3/4) + (1/4 × 1/2) = 9/16 + 2/16 = 11/16 = 0.6875. Thus p(v1=0) = 5/16 = 0.3125. These are expected frequencies over repeated conditional draws from this fixed starting example, not outcomes of a live simulation.

Using the negative statistic v1 × p(h1=1 | v1), the expected CD-1 negative weight statistic is p(v1=1) × 0.75 + p(v1=0) × 0 = (11/16) × (3/4) = 33/64 = 0.515625.

The expected CD-1 update direction for the weight is 0.75 − 0.515625 = 0.234375, or 15/64. The exact maximum-likelihood direction was 0.25, or 16/64. In this example, the one-step method underestimates the exact direction by 1/64 = 0.015625. The calculation demonstrates finite-step bias without making a universal claim about its sign. In other models and parameter regions the behaviour can differ substantially.

The correct denominator for the comparison is the exact gradient magnitude 0.25 if reporting relative error: 0.015625 / 0.25 = 6.25 percent. Relative error is meaningless when the reference gradient is zero, so do not apply that division blindly to every parameter.

Why more Gibbs steps may help but do not guarantee perfection

Increasing k gives the Markov chain more opportunities to move away from its initial training example and approximate the model’s stationary distribution. Under suitable ergodicity and mixing assumptions, sufficiently long chains can approach the relevant equilibrium distribution. In practice, each extra Gibbs transition costs computation, and the number of transitions required to mix can vary greatly with model structure and parameters.

If the model has several separated modes, a chain may remain trapped near one mode for a long time. Running CD-10 instead of CD-1 can change the estimate, but it does not certify that every important region is represented. A comparison of CD-1, CD-5 and CD-20 therefore answers a practical engineering question about cost and approximate behaviour, not a theorem that larger k is always best.

Batch size also matters. If a batch has 100 training examples, averaging 100 positive and 100 negative contributions can reduce random variability compared with a single sample. It does not eliminate bias caused by starting every chain near its corresponding data case. Randomness and systematic bias are different sources of error.

A good training report names the actual sampling procedure, the number of Gibbs updates and the evaluation criteria. A label such as trained with CD is too vague for another researcher to reproduce the result.

Persistent contrastive divergence changes the starting point

Persistent contrastive divergence, associated with Tieleman’s 2008 work, maintains Markov chains across parameter updates rather than restarting each chain from the current training case. The hope is that persistent chains will explore the model distribution more broadly over time, so the negative statistics approximate the model’s own expectation more faithfully.

Persistence has a cost. Chains can mix poorly, especially if the energy landscape develops deep isolated basins. As parameters change during training, the target stationary distribution also moves. A persistent chain is not magically an exact equilibrium sample. The learning rate, chain count, mixing behaviour and initialization still matter.

The conceptual distinction is straightforward. Ordinary CD-k asks what the model reconstructs after k steps away from these data examples. Persistent training asks what the model’s continuously evolving chains are currently sampling. Neither automatically gives the exact partition function. Each is a different approximation strategy for a difficult expectation.

For teaching, compare a person who always walks two steps away from home with someone who continues a long walk from yesterday’s endpoint. The analogy captures the different starting states, but it is only an analogy: neither walker guarantees representative coverage of a complicated probability landscape.

Use the original persistent-CD research for evidence about the algorithm and experiments, rather than turning one published comparison into a promise of universal superiority.

Contrastive divergence is not noise-contrastive estimation

The names sound alike, and this similarity causes avoidable confusion. Contrastive divergence trains energy-based models by comparing data statistics with statistics after a short Markov chain. Noise-contrastive estimation, or NCE, creates a discrimination problem involving data and samples from a known noise distribution; under suitable conditions it can estimate parameters of unnormalised models without explicitly evaluating the full normaliser.

Both address difficult normalisation or distribution-learning problems, but their mechanisms, objective functions and assumptions differ. CD reconstructs near the data through the model’s conditional transitions. NCE needs an explicit noise sampling process and a classification-style comparison. Changing the name of a CD section to NCE would not make the procedures equivalent.

There is a third terminology trap. Modern contrastive representation learning may compare paired embeddings using similarity scores, positive pairs and negative pairs. Those methods are not simply CD with modern branding. Likewise, a diffusion model’s denoising score-matching loss is not the same as an RBM’s CD-1 update, despite a superficial shared language about reconstruction and noise.

When readers encounter the word contrastive, ask three questions: what are the two distributions or cases being compared, what sampling procedure produced them, and which mathematical objective or approximate gradient is being optimised? These questions reveal the real algorithm far more reliably than the family resemblance of names.

Why reconstruction error cannot prove a good probability model

CD training often monitors reconstruction error because reconstructed visible units are easy to inspect. A low one-step reconstruction error can be encouraging: nearby data patterns may be stable under the model’s transitions. But it is not the same as high held-out likelihood, good global sampling or coverage of all modes.

A model can make a training example easy to reconstruct while assigning too much probability to other unwanted regions. Conversely, a model might preserve meaningful distributional variation and occasionally reconstruct a visibly different but plausible sample. A pixel-level error metric can penalise that difference without assessing whether the probability distribution is sensible.

In small models, compute exact log likelihood where feasible. In larger models, select evaluation techniques appropriate to the model and scientific question, such as justified approximate likelihood estimates, downstream usefulness and held-out diagnostics. Report approximation error where known. Do not substitute training reconstruction curves for validated generalisation merely because they are easy to plot.

Always record the data split. If a training set has 8,000 examples and a held-out set has 2,000, report rates using the appropriate denominator. Improving a training reconstruction average on 8,000 seen examples is not evidence of improved performance on the 2,000 withheld cases.

A practical training audit: units, scaling and randomness

Before trying exotic model changes, audit basic implementation details. Are binary units genuinely Bernoulli samples, or have raw probabilities been passed forward as if they were states? Is the sign of the energy correct? Are positive and negative statistics subtracted in the intended order? Are bias gradients included? Is the mini-batch average divided by its actual number of examples?

Suppose a batch contains 40 samples and the sum of positive-minus-negative weight contributions is 12. The batch-mean statistic is 12 / 40 = 0.30. With a learning rate of 0.05, the corresponding simple update magnitude is 0.015 before any other optimiser effects. If code uses 12 instead of 0.30, the update becomes 0.60, forty times as large. The units are parameter update per mini-batch, derived from association difference per example.

Check random seeds for reproducibility while remembering that fixed seeds do not make a method statistically correct. Compare several seeds and report variability. Check whether each example gets a new chain, whether persistent chains are carried across updates and whether batches are shuffled without leaking held-out cases.

Test a one-visible, one-hidden implementation against the exact four-state model above. If its calculated Z, conditional probabilities, expected CD-1 negative statistic and update cannot match these fractions within numerical tolerance, it is not yet ready for a thousand-dimensional benchmark.

Learning rate, momentum and weight decay are separate choices

CD determines an approximate learning signal. An optimiser determines how parameters move in response. A learning rate that is too large may create unstable dynamics; a rate that is too small may make learning impractically slow. Momentum can smooth or amplify update trends depending on its configuration. Weight decay changes the optimisation problem by penalising parameter magnitude.

These mechanisms should be documented separately. Saying CD-1 failed when the actual cause was a missing batch-size division obscures the real repair. Saying CD-10 succeeded because it used ten steps while also changing the learning rate, momentum and hidden-unit count does not isolate what helped.

A controlled comparison holds most factors constant. For instance, train the same architecture with the same data split and learning-rate schedule under CD-1 and CD-5, then report training cost, held-out evaluation and variability across seeds. If the comparison uses five times as many transitions per update, also discuss computational budget. An iteration is not automatically the correct denominator for efficiency.

Hinton’s practical training guide discusses these configuration choices in the specific RBM context. It is best used as a source of engineering considerations rather than a timeless table of universal hyperparameters.

When an update appears to work but is learning the wrong structure

A model may learn local correlations that are easy to reconstruct yet fail to represent long-range combinations. In an image-like dataset, individual pixels may become predictable from nearby pixels while global shapes remain implausible. The short chain can support such local regularities because it starts on the data manifold and takes only a few steps away.

A diagnostic is to compare short reconstructions with longer free-running samples and, when possible, with exact or controlled likelihood estimates. If one-step reconstructions look convincing but long-run samples are poor, the model may not have learned a useful global distribution. The exact interpretation depends on architecture and evaluation, so treat this as a warning pattern, not an infallible theorem.

Another failure is mode neglect. Suppose a hypothetical dataset has 600 examples from pattern family A and 400 from family B. If generated samples repeatedly show almost only A, the model may be misrepresenting the mixture. Do not report model coverage from three cherry-picked generated images. Count a suitably designed sample set, describe the classification rule and report the denominator.

The crucial question is not whether individual reconstructions resemble training inputs. It is whether the probability model supports the diversity, frequency and structure required by the intended task.

Bias is not merely a small arithmetic nuisance

The toy example showed a small difference between the expected CD-1 direction and the exact gradient. It would be wrong to generalise that 6.25 percent discrepancy to a realistic model. With complex energy landscapes, short-run sampling can produce substantially different updates, and analytical work has identified important limitations on simple interpretations of CD learning.

Sutskever and Tieleman’s 2010 analysis is particularly useful because it examines CD-1 properties rather than merely plotting a successful experiment. Their work shows that the CD-1 update rule for binary RBMs is not generally the gradient of any function and provides examples of problematic learning dynamics under certain regularisation conditions. Thus it is careless to promise that every CD training run follows the gradient of a single well-behaved scalar objective.

This does not mean CD is useless. It means the learner must distinguish an effective computational heuristic from an exact optimisation guarantee. A method can be valuable because it offers practical updates at manageable cost even when its global theoretical properties are limited.

When writing about AI, keep that distinction visible. Fast approximate learning, exact probability inference, calibrated uncertainty and faithful representation are different outcomes and require different evidence.

Transfer exercise: a new dataset with a rare but important pattern

A tutor builds a hypothetical RBM for binary feature vectors describing safe classroom equipment states. Of 1,000 approved examples, 950 describe ordinary configurations and 50 describe a rare but valid emergency configuration. The model gives low one-step reconstruction error on ordinary cases, but a longer set of generated configurations almost never contains the rare valid pattern. The tutor concludes that CD-1 has learned the data distribution accurately because 95 percent of cases are common.

What is wrong? A high aggregate reconstruction score can conceal poor representation of the rare mode. The share 50 / 1,000 = 5 percent is small but not zero, and the task gives the rare configuration independent practical importance. A model used to reason about equipment should not be validated solely by accuracy on the dominant mode.

A useful repair plan is to evaluate each subgroup separately, inspect generated frequency and transition behaviour, compare longer-chain or persistent-sampling approaches under controlled conditions, and consider whether the architecture or data representation is appropriate. If there are 200 held-out examples with 10 rare ones, report the performance on the rare subgroup using denominator 10, not 200.

The changed context shifts attention from how quickly CD updates weights to what the learned distribution must faithfully represent. The transfer principle is that an approximate training method is judged against the downstream claim, not only against its internal reconstruction statistic.

Where contrastive divergence remains useful

Contrastive divergence remains a valuable teaching vehicle for energy-based probability learning. It connects conditional independence, Gibbs sampling, normalisation, approximate maximum likelihood, bias, variance and evaluation. Those ideas appear across probabilistic modelling, even when a particular modern application uses a very different architecture or training objective.

Its appeal is clearest for models whose conditional distributions can be sampled cheaply and whose exact global expectations are expensive. It can also provide a baseline when developing or testing alternative sampling methods. A tiny RBM is an excellent environment for building intuition because the exact answer can be enumerated and compared with approximate updates.

There is no need to claim that all contemporary generative AI is trained with CD. Transformer language models commonly use other objectives; diffusion models often rely on denoising-related training; modern energy-based approaches may use many other methods. Being historically important and conceptually powerful does not make an algorithm universally dominant today.

A student who understands why CD-1 is biased will read claims about other approximate AI training methods more carefully. That is a more durable learning outcome than memorising a recipe for sampling h0 and v1.

FAQ: Is CD-1 the same as one gradient descent step?

No. The numeral refers to the number of Gibbs reconstruction transitions used for the negative-phase approximation, under the algorithm’s chosen counting convention. The parameter update happens after collecting the statistics. A single optimiser step can use CD-1 or CD-k, and multiple optimiser steps can each use new short chains. Mixing these counts makes experiment reports hard to reproduce.

FAQ: Must hidden units be sampled at the final step?

Not necessarily. Some practical implementations use sampled hidden states to evolve the chain and conditional hidden probabilities to calculate expected association statistics at the negative endpoint. That can reduce noise for the statistic being estimated. The implementation should state exactly which state is sampled and which probability is used; the toy arithmetic in this article uses hidden probabilities for the association terms.

FAQ: Why not calculate the partition function exactly?

For a tiny model, do exactly that. It is a valuable test. But the number of joint binary configurations grows exponentially with the number of units. A direct sum over all states rapidly becomes impractical. Special model structures and alternative estimation methods can help, but they do not make exact global normalisation routinely cheap for every high-dimensional energy model.

FAQ: Does increasing k always improve generalisation?

No universal guarantee follows. More transitions can change the negative-phase approximation and cost more computation. Whether the result improves depends on mixing, model architecture, optimiser configuration, sample size and evaluation objective. Compare methods under a comparable computational budget and independent validation rather than assuming that a larger numeral is proof of quality.

FAQ: Can a perfect one-step reconstruction prove a good generator?

No. A model that reconstructs nearby training cases can still assign an unsuitable distribution to distant or rare configurations. One-step reconstruction assesses one local behaviour. Evaluating a generative model requires evidence about held-out cases, diversity, likelihood where estimable and the intended practical use. The metric and denominator should match the claim.

FAQ: Is CD training the same as modern contrastive image-text learning?

No. Contrastive image-text systems commonly learn embeddings by comparing matching and nonmatching pairs under a representation objective. CD is a short-chain approximation for training certain energy-based probability models. The shared adjective refers to comparing alternatives, not to an identical loss, model family or sampling process.

FAQ: What is the safest first debugging test?

Implement the one-visible, one-hidden model in this article. Check that its joint unnormalised weights are 1, 1, 1 and 3, giving Z=6; its exact weight gradient on v=1 is 0.25; and its expected CD-1 negative statistic is 33/64. Once these are correct, test batches, randomness and model scaling. Do not hide a sign error inside a large neural network.

For a different way to train unnormalised probabilistic models, read How AI Works | Noise-Contrastive Estimation; compare the discrimination-based noise distribution with CD’s short Gibbs chain. For another perspective on matching distributions, read How AI Works | Optimal Transport. These are distinct mechanisms, not renamed versions of the same page. The How AI Works hub organises the wider learning journey.

A beginner can start with the four-state probability table and verify that the probabilities sum to one. An intermediate learner can derive each conditional sigmoid and the expected CD-1 update. An advanced learner can experiment with mixing time, compare persistent chains and design a held-out evaluation where a low reconstruction error would be misleading. Each level should produce a checkable explanation, not merely a numerical output.

Primary research and precise sources

The historical foundation is Geoffrey Hinton, Training Products of Experts by Minimizing Contrastive Divergence (Neural Computation, 2002); the author’s original research page is Training Products of Experts. Practical implementation context is in Hinton, A Practical Guide to Training Restricted Boltzmann Machines (University of Toronto technical report, 2010). For persistent chains read Tieleman, Training Restricted Boltzmann Machines Using Approximations to the Likelihood Gradient (ICML, 2008). For important theoretical limits read Sutskever and Tieleman, On the Convergence Properties of Contrastive Divergence (AISTATS, 2010). The worked numbers are hand-derived examples and are not presented as measured outcomes.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading