eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Maximum Mean Discrepancy — Kernel Two-Sample Tests, Bandwidths and Distribution Shift

—

by

eduKate Asia

Did you know? Two datasets can have similar averages but still come from meaningfully different distributions. Maximum mean discrepancy, usually shortened to MMD, compares samples through a kernel so that it can detect differences in how data are distributed, not just differences in one reported mean. It is useful for two-sample testing and as a diagnostic for some kinds of distribution shift, but the result depends on the kernel, sample design and statistical test. A small number does not prove that two populations are identical.

The practical answer is to define which populations you are comparing, select a kernel that reflects relevant differences, compute a finite-sample MMD statistic, and use a justified null-distribution procedure such as a properly designed permutation test if you need a statistical decision. Keep grouped observations together, avoid tuning the kernel on the same final test without accounting for selection, and distinguish a detected data shift from evidence that a predictive model has become wrong.

This guide develops the kernel formula from first principles, solves an exact toy example including a permutation p-value, diagnoses estimator and bandwidth errors, and transfers the method to a new population. The numerical samples are invented teaching cases, not claims of a completed production experiment.

Contents

What question a two-sample test actually answers

A two-sample test asks whether independent observations from two populations support the hypothesis that their underlying distributions are the same. That is a different question from whether one classifier’s accuracy is high, whether a regression line fits, or whether the two datasets have the same number of rows. A distribution includes how probability is allocated across possible outcomes, not merely one average.

Imagine comparing readings from a temperature sensor before and after recalibration. The two samples might share a mean while one has a wider spread, heavier tails or a second mode. A mean-difference test can miss some of these changes. A kernel method represents comparisons between observations in a flexible similarity function, potentially capturing more aspects of distribution shape.

Maximum mean discrepancy is the difference between mean representations of two distributions in a reproducing kernel Hilbert space, measured in a suitable norm. That phrase can sound abstract, but the finite-sample calculation is direct: compare within-sample similarities for the first group, within-sample similarities for the second, and cross-group similarities, with exact normalisation.

The original paper A Kernel Two-Sample Test by Gretton and colleagues (Journal of Machine Learning Research, 2012) formalises MMD and several testing procedures. It also discusses the computational cost of quadratic estimators and more efficient approximations. Its claims are about statistical methods under assumptions, not a promise that every shift relevant to AI will be detected.

A useful diagnostic report says which features, population and sampling periods were compared. An MMD score without this context can be as misleading as a school test mark without naming the subject, maximum marks or assessment conditions.

Why a similarity function can reveal more than a raw average

A kernel k(x,x’) computes a similarity-related quantity between two observations under mathematical conditions that make it a valid positive-semidefinite kernel. The linear kernel k(x,x’)=x·x’ focuses on mean differences in the original feature space. A Gaussian radial basis function kernel, often written exp(−||x−x’||²/(2σ²)), can capture richer differences depending on its bandwidth.

Think of the kernel as a rule for asking which observations resemble each other. Within a population, observations often show characteristic similarity patterns. If two populations differ, their within-group similarities may not balance their cross-group similarities. MMD summarises that imbalance in a principled way.

Kernel choice changes what differences the statistic can detect. The linear kernel can return zero for two distributions with identical means even if their variances differ. A suitable characteristic kernel, such as a Gaussian RBF under standard conditions, can distinguish different distributions at the population level through their kernel mean embeddings. Finite samples still introduce uncertainty and limited power.

Feature scaling can completely alter the distances used by an RBF kernel. If one feature is measured in thousands of units and another in fractions, raw Euclidean distances may be dominated by the first unless scaling or kernel design is intentional. Fit transformations under a properly separated data protocol and preserve the same transformation for both groups.

The kernel also defines what counts as a relevant shift. A test using only word-length features cannot detect a change in factual meaning that leaves lengths unchanged. The measurement representation, not just the mathematical test, determines which population differences can become visible.

The population formula and its finite-sample version

At the population level, squared MMD can be expressed as E[k(X,X’)] + E[k(Y,Y’)] − 2E[k(X,Y)], where X and X’ are independent draws from the first distribution, Y and Y’ are independent draws from the second, and the cross term draws one observation from each. The expected within-group similarities are compared with the expected cross-group similarity.

For finite samples of sizes m and n, a common biased estimator includes diagonal self-comparisons and uses denominators m² and n² for within-sample sums and mn for the cross sum. The resulting estimator is the squared distance between the two empirical mean embeddings and is nonnegative for a valid positive-semidefinite kernel, apart from numerical precision error.

Another estimator removes same-observation diagonal comparisons. Its within-group denominators are m(m−1) and n(n−1), and the cross-group term remains proportional to 1/(mn). This U-statistic-style estimate can be unbiased for the population squared MMD under suitable independent-sampling assumptions. Unlike the biased empirical squared norm, it can be negative for a small finite sample.

Those details matter because a negative unbiased estimate is not necessarily a coding error or a claim that a mathematical squared distance is negative. The true population squared MMD cannot be negative; the unbiased finite-sample estimator can fluctuate below zero. Clipping it to zero changes the estimator and may alter downstream tests.

When reporting numbers, state whether the value is MMD or squared MMD, which estimator was used, sample counts and the kernel parameters. A threshold meaningful for a squared linear-kernel statistic is not automatically meaningful for a Gaussian-kernel statistic. Exact labels prevent innocent-looking comparisons that have no mathematical basis.

Worked example: four observations and a linear kernel

Take two fictional samples: P has values 0 and 2, while Q has values 1 and 3. Both contain two observations, so m=n=2. Choose the scalar linear kernel k(a,b)=ab. The sample mean of P is (0+2)/2=1. The sample mean of Q is (1+3)/2=2. With a linear kernel, biased empirical squared MMD equals the squared mean difference: (1−2)²=1.

Check through the full kernel formula rather than relying on the shortcut. The P-with-P similarity matrix consists of 0×0=0, 0×2=0, 2×0=0 and 2×2=4, summing to four. Divide by m²=4 to get one. The Q-with-Q products are 1,3,3,9, summing to sixteen; divide by n²=4 to get four.

The four cross-group products are 0×1=0, 0×3=0, 2×1=2 and 2×3=6, summing to eight. Divide by mn=4 to get two, then multiply the cross mean by two in the formula. Squared biased MMD is therefore 1+4−2×2=1. The denominators are not optional; forgetting the factor of two would produce the wrong answer.

Now compute the unbiased estimator using only distinct observations within each group. For P, both ordered off-diagonal products are zero, yielding 0/[2×1]=0. For Q, the two ordered off-diagonal products are 1×3=3 and 3×1=3, yielding 6/2=3. The cross term remains four, so unbiased squared MMD estimate is 0+3−4=−1.

That negative result is possible because the unbiased estimator is not an empirical squared norm. It does not say that the population discrepancy is negative. The example deliberately uses tiny samples where estimator variability is large; it is not adequate evidence for claiming a meaningful shift in an application.

A fully checked permutation test on the same four numbers

Suppose we want to test whether the labels P and Q could be exchangeable under a null hypothesis of identical distributions. With four pooled observations 0,1,2,3, there are C(4,2)=6 distinct ways to choose which two belong to P while leaving the other two in Q. Each allocation yields a squared mean difference under the linear kernel.

List the six possible P pairs. If P receives (0,1), its mean is 0.5 and Q’s mean is 2.5, so squared difference is 4. If P receives (0,2), the means are 1 and 2, yielding 1. If P receives (0,3), both means are 1.5, yielding 0.

For P=(1,2), the means are again 1.5 and 1.5, yielding 0. For P=(1,3), means are 2 and 1, yielding 1. For P=(2,3), means are 2.5 and 0.5, yielding 4. Thus the complete set of permutation statistics is 4,1,0,0,1,4.

Our originally observed grouping was P=(0,2), Q=(1,3), giving statistic one. Four of the six possible assignments have statistic at least one. The exact permutation tail proportion under this finite exchangeable assignment model is 4/6≈0.6667. This is not evidence for rejecting equality at a conventional small significance level.

The arithmetic offers an important lesson. A visibly nonzero MMD statistic does not automatically establish a statistically meaningful difference. The null distribution and sample size matter. In real datasets, unrestricted permutation of correlated rows can invalidate the test. If multiple readings come from the same person, device or document, treat the dependence structure as part of the design rather than shuffling every row freely.

This numerical case is intentionally tiny so every allocation can be audited. Larger tests use suitable random permutations, asymptotic thresholds or other methods with explicit assumptions and error-control properties. Their calibration should be based on the chosen estimator and kernel, not copied from the toy calculation.

Gaussian RBF bandwidth is a decision about scale

The Gaussian RBF kernel compares distance through a positive bandwidth parameter σ. When σ is extremely small relative to most pairwise distances, the kernel becomes close to zero for most distinct observations. When σ is extremely large, many observations look nearly identical with kernel values close to one. Either extreme can reduce a test’s ability to see the differences that matter.

Visualise two clusters separated by a moderate distance. If the kernel bandwidth is much smaller than distances between nearly all points, cross and within similarities may both be sparse. If bandwidth is much larger than the full dataset spread, subtle structure can disappear into almost constant similarities. A suitable scale depends on data geometry and the alternatives the test aims to detect.

A common heuristic is to use a median pairwise distance, but that is not a theorem guaranteeing high statistical power. It can perform poorly on mixtures, high-dimensional datasets or when the shift occurs on a different scale from the bulk of the observations. The original and follow-up kernel testing literature studies how selection affects power.

One primary paper on the issue is Optimal Kernel Choice for Large-Scale Two-Sample Tests by Gretton and colleagues. It examines kernel selection with the aim of obtaining more powerful tests. This is a researched statistical decision, not permission to repeatedly tune σ on final test outcomes without adjusting the significance procedure.

Practical experiments can examine several predeclared bandwidths and use a valid multiple-testing or aggregation strategy. If kernel choice is informed by the same labels and statistic subsequently tested as if the choice were fixed, the reported p-value can be too optimistic. Design the selection and testing protocol as one whole procedure.

High dimensions, dependent features and distance concentration

High-dimensional data create a particular challenge for distance-based kernels. Pairwise distances can become less informative when many coordinates contain irrelevant variation. An RBF kernel on raw embeddings may respond primarily to scale, norm or an incidental measurement difference rather than a meaningful semantic shift. MMD can still be useful, but representation design and test power require explicit attention.

Standardising features can help when coordinates have comparable measurement meaning. Yet indiscriminate standardisation may also erase a real change in scale that the test was meant to detect. Decide whether the monitoring question concerns absolute feature values or a relation after normalisation. Keep the transformation definition stable across compared populations.

Correlation among coordinates matters. Two dozen measurements from the same underlying sensor do not provide twenty-four independent sources of evidence. A kernel that treats them as independent geometric directions can effectively overweight that sensor. Domain knowledge or a suitable representation may be needed to match the distance geometry to the problem.

An embedding learned on a separate task can make MMD more sensitive to certain semantic differences and blind to others. A language embedding may group paraphrases well but fail to preserve a rare regulatory qualifier. If the monitored hazard is missing negation, a representation that loses that feature cannot be saved by choosing a more expensive kernel.

The point is not to abandon kernel tests in high dimensions. It is to evaluate power against meaningful simulated or held-out shifts, use honest sample design and avoid interpreting a mathematically exact statistic as a universal measure of semantic distance.

A distribution difference is not automatically a model failure

Suppose an AI model was trained to classify school-library books. The incoming books begin including more science titles than before. Feature distribution P(X) has changed. Whether the classifier’s accuracy also changes depends on how well it handles the new books and whether the relationship between book features and labels remains stable. MMD on X alone cannot determine that relationship.

Other kinds of shift are different. The target prevalence can change, or the conditional relationship P(Y|X) can change while the marginal feature distribution appears similar. An MMD test applied only to available input features may miss a damaging change in labels or decision criteria. Use delayed outcome data and task-specific model monitoring when available.

Conversely, MMD may flag a shift in an irrelevant feature. Imagine a document model that performs equally well after scanned documents become slightly brighter because of a hardware change. A pixel-feature kernel may show a strong distribution difference even though recognition accuracy is unchanged. The alert is a prompt for investigation, not automatically a directive to retrain.

Define in advance which shifts would matter for a decision. Monitoring data quality, covariate distribution and real-world outcomes as separate panels avoids conflating warning types. A shift detector can be a useful early signal while the definitive performance metric remains unavailable, but uncertainty should be communicated.

Readers interested in moving probability mass between distributions can compare the optimal transport guide. Optimal transport measures a different notion of discrepancy, based on transport costs and couplings, while MMD compares kernel mean embeddings. They are not numerically interchangeable.

A careful workflow for automated distribution monitoring

First define the reference and current populations. A baseline of last year’s records may be inappropriate if the product or collection method changed in a documented way. Record time windows, units, data provenance, sampling filters and expected composition. Without this context, a shift alarm can describe an intentional rollout rather than an unexpected problem.

Second choose the measured representation and kernel before examining final alert outcomes. Decide whether the comparison concerns raw features, standardised variables or a verified embedding. Consider grouped observations, serial correlation and known seasonal patterns that affect independence. A sound test begins with a sampling design, not a kernel function pasted into production.

Third establish a validation regime. Simulate relevant known shifts and null conditions to measure false alarm rate and detection power under representative sample sizes. A test that rejects almost everything in ordinary daily fluctuation is not useful. A test that never rejects any realistic harmful shift is not reassuring.

Fourth compute and store the statistic with exact estimator, counts and kernel settings. If using permutation inference, make the shuffle scheme match the exchangeability structure. Use a held-out calibration design or valid adjustment if hyperparameters were selected using labelled comparison data.

Fifth link the alert to operational response. An elevated statistic may trigger a feature audit, label review, human inspection or targeted performance test. It should not independently authorise destructive model retraining or deployment. The user needs an investigation route, not an opaque p-value without context.

Sixth log later confirmed outcomes. Did the shift indicate real performance degradation or harmless input variation? Those records can improve future monitoring decisions, but they must not be used to retroactively relabel an already reported significance result as something it was not.

Ten MMD mistakes that create confident but invalid conclusions

The first mistake is calling MMD a classifier accuracy score. It compares distributions, not predictions with ground-truth labels. The second is forgetting which estimator is used, then treating a negative unbiased estimate as proof of impossible mathematics. The third is mixing MMD with squared MMD without changing thresholds.

The fourth is using an arbitrary kernel bandwidth and assuming every distribution difference is equally detectable. The fifth is fitting feature transformations on both labelled groups in a way that makes the test question change, then reporting the old interpretation. The sixth is selecting a kernel after inspecting many test outcomes while using an unadjusted p-value as though no selection occurred.

The seventh is shuffling individual rows when observations form clusters, repeated measures or temporal dependencies. That can break exchangeability and miscalibrate a permutation test. The eighth is treating failure to reject equality as proof of identical distributions. A test can have low power, especially with small samples or an unsuitable kernel.

The ninth is assuming a detected shift necessarily harms the deployed AI model. Important changes might be harmless, while damaging conditional changes might escape a feature-only comparison. The tenth is comparing raw MMD scores from different kernels, feature sets or data scales as though they share a universal unit.

Repair begins with specificity: name the population, feature representation, kernel, estimator, sample counts, null hypothesis, test calibration and decision use. A high-quality statistical report is often longer than the single p-value that appears on a dashboard.

Why the unbiased statistic may be negative without contradiction

The word ‘squared’ can mislead people into assuming every number generated by an MMD formula must be nonnegative. The population squared discrepancy is indeed nonnegative, as is the biased empirical squared-norm estimator when the kernel matrix is mathematically valid. The unbiased U-statistic estimate, however, subtracts diagonal self-comparisons and is no longer constrained to be nonnegative on every finite sample.

Our toy linear-kernel example produced biased estimate one and unbiased estimate minus one. The difference does not mean that either computation was arithmetic nonsense. They estimate the same population quantity under different finite-sample constructions, with different bias and variance behaviour. In sufficiently small samples, the distinction is visible and important.

Clipping every negative unbiased value to zero may simplify a display, but it also changes the statistic’s sampling distribution and can introduce bias. A testing procedure that expects the original unbiased estimator should not silently receive a clipped version. It is better to report the sign honestly, explain what it means and compare with the correct null distribution.

Another subtlety is that not every function casually called a similarity metric is a valid positive-semidefinite kernel. If a user invents an arbitrary matrix and applies the biased formula, the empirical squared expression may lack the expected nonnegative property. Validate the mathematical assumptions before using an observed negative number to diagnose software precision.

The original JMLR paper provides the rigorous treatment behind the estimators. For practical education, the key lesson is to distinguish the truth being estimated from the random number produced by one finite dataset.

Computational cost and sample-size trade-offs

Computing every pairwise kernel similarity for m observations from one sample and n from another uses matrices with roughly m², n² and mn entries. When the groups are both large, memory and time demands can become substantial. A naive program that builds every matrix at once may exhaust memory even though the statistical formula is correct.

Blockwise computation can reduce peak memory by processing subsets of pairs and accumulating sums with careful numerical precision. Approximate feature maps, linear-time estimators and incomplete U-statistics can reduce cost at the price of approximation or statistical trade-offs. The right method depends on available sample size, hardware and power requirements.

More data does not automatically cure a poor kernel. A monitoring system can spend large amounts of compute on a test whose representation ignores the harmful shift. Conversely, a well-designed small sample may answer a focused question efficiently, while still needing adequate size for a defensible significance claim.

Use memory and runtime budgets based on the planned pair counts. If both groups have 10,000 observations, a single dense within-group matrix has 100 million entries; at eight bytes per double-precision value this is about 800 million bytes before overhead and additional matrices. That arithmetic illustrates why a naive full-matrix implementation can become expensive.

These are computational constraints, not reasons to conceal uncertainty. If a fast approximation replaces an exact estimator, report that fact and evaluate its error or operating characteristics for the actual use case.

Changed-context exercise: comparing classrooms rather than independent rows

Transfer the idea from sensor readings to educational data. Two schools submit vocabulary assessment responses. Each school has 40 students, and each student answers 10 items, yielding 400 rows per school. A careless analyst might treat all 800 responses as independent examples and shuffle them freely in a permutation test.

But responses from one student share ability, experience and classroom conditions. Ten responses from one learner are not ten independent students. The effective sampling unit for a comparison may need to be the student or even the classroom, depending on the question. A row-wise permutation can produce an unjustifiably narrow null distribution and exaggerated evidence.

Suppose the legitimate objective is to compare the distribution of student-level summary features between two groups of 40 students each. Then the study should represent each student’s relevant features in a consistent way and consider appropriate cluster or hierarchical structure. It should also check whether differences in test form, instruction or sampling explain an apparent shift.

Imagine one school used a harder vocabulary form. MMD may detect a difference in the observed response distributions, but it cannot tell whether the underlying vocabulary capability differs without ensuring comparable assessment conditions. The statistical result describes the recorded distributions; interpreting causes demands additional evidence.

The changed-context solution is to redefine the unit and features, ensure a valid comparison and resist attributing a shift to students before checking measurement design. This is a stronger transfer of statistical intelligence than merely substituting ‘student’ for ‘sensor’ in the code.

What a significant result should—and should not—authorise

A small p-value can indicate that the observed statistic would be unusual under the specified null model and testing procedure. It is not the probability that the null is true, not a measure of the practical size of the effect, and not a guarantee that an AI model needs to be replaced.

Similarly, a nonsignificant result does not certify equality. A small test set, high-dimensional noise or poorly chosen kernel may leave the procedure unable to detect meaningful differences. If equivalence within a practical tolerance is the real business question, a suitable equivalence-testing or estimation framework may be more appropriate than a conventional failure-to-reject statement.

Actions have costs. Retraining a model after every distribution alarm may introduce instability or degrade performance. Ignoring every alarm can allow real harm to continue. A defensible policy combines statistical evidence with task-specific effect size, independent outcome checks and authorised review.

Data privacy also constrains monitoring. Kernel comparisons can be conducted on features without revealing every raw individual record to every reviewer, but the transformed features and similarity matrices may still contain sensitive information. Apply appropriate access controls, retention rules and governance rather than assuming a derived statistic is automatically harmless.

The purpose of MMD in responsible systems is to improve questions and verification. It should help humans and software understand when observed data differ and where investigation is justified, without pretending to know the cause or final operational decision.

Frequently asked questions about MMD

What does maximum mean discrepancy measure? It compares distributions through the difference between their kernel mean representations. A finite-sample estimate is computed from within- and cross-group kernel similarities.

Is MMD a distance between individual points? No. It measures a discrepancy between distributions or empirical samples under a chosen kernel, not one pair of observations.

Does zero MMD prove the datasets are the same? Not in an ordinary finite-sample test. At the population level a characteristic kernel can distinguish distributions, but finite samples and test power still matter.

Why can an unbiased squared MMD estimate be negative? The unbiased finite-sample estimator removes self-similarity terms and can fluctuate below zero even though the true squared discrepancy is nonnegative.

What does an RBF bandwidth do? It sets the scale over which observations are considered similar. Very small or very large choices can reduce power for the differences of interest.

Must I use a permutation test? No. MMD has several inferential methods with distinct assumptions. A permutation test is intuitive when observations are exchangeable under the null, but grouped and time-dependent data require careful design.

Is MMD the same as Wasserstein distance? No. Wasserstein and related optimal-transport methods use a transport cost and coupling; MMD uses a kernel mean difference.

Can MMD detect all bad model drift? No. It may detect irrelevant feature differences and miss conditional changes that matter for labels. Monitor performance and outcomes separately.

How should an agent report an MMD alert? State sample counts, period, representation, kernel, estimator, calibrated significance evidence and what remains unknown; route consequential decisions to the authorised owner.

Practical reader pathways and original research

If you are learning machine learning, work through P=(0,2) and Q=(1,3) with a linear kernel. Recompute the biased estimator of one, the unbiased estimator of minus one and the six exact label assignments. Explain why the resulting permutation proportion of four sixths is not evidence of a strong statistical difference.

If you operate models, define meaningful population comparisons and group dependencies before selecting kernels. Use held-out test calibration or a valid adaptive procedure, and connect alerts to human review of actual performance. For related mechanisms, read Optimal Transport and the broader How AI Works pathway.

If you teach statistics, ask learners to change the feature representation and explain whether the same kernel would still detect the shift. Then replace independent observations with repeated measures and explain why the old permutation procedure breaks. The goal is statistical judgement that transfers across problems.

Primary sources: Gretton et al. (2012), A Kernel Two-Sample Test; Optimal Kernel Choice for Large-Scale Two-Sample Tests; Sejdinovic et al., Equivalence of Distance-Based and RKHS-Based Statistics; and Schrab et al., Efficient Aggregated Kernel Tests Using Incomplete U-statistics.

All small numerical experiments and case studies in this article are illustrative and do not imply an external program was executed. For wider educational context, see learning that transfers into life.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading