A large model knows how to classify an image.
A smaller model is asked to learn from it.
The smaller model does not receive the larger model’s entire internal state.
It receives selected teaching signals.
That is knowledge distillation.
Quick Read
Knowledge distillation trains a student model to reproduce selected behaviours of a teacher model. Instead of learning only from one-hot ground-truth labels, the student may learn from the teacher’s probability distribution, intermediate features, attention maps or other structured signals.
Recent reviews continue to treat knowledge distillation as a major model-compression family because it can transfer useful predictive behaviour into a smaller architecture suitable for resource-constrained deployment.
One-sentence answer: distillation is lossy because a smaller student cannot inherit every degree of freedom or internal representation of the teacher, so the training process chooses which teacher behaviours are important enough to survive.
Hard Labels Throw Away Teacher Structure
Suppose an image is labelled “cat”.
A hard label says only cat = 1, everything else = 0.
A teacher model might instead output something like:
- cat 0.78
- fox 0.12
- dog 0.07
- car 0.0001
That distribution contains relational information.
The teacher believes the image is much more fox-like than car-like.
Distillation can use that extra structure.
Soft Targets Carry Dark Knowledge
The phrase “dark knowledge” is often used for information contained in the teacher’s non-maximal probabilities.
The teacher’s uncertainty encodes similarities among classes.
A hard label collapses all wrong classes into the same zero.
A soft distribution preserves a richer geometry of alternatives.
Temperature Changes the Teaching Signal
Distillation often applies a temperature parameter to soften the teacher’s logits before converting them to probabilities.
A higher temperature spreads probability mass more broadly across alternatives.
This can expose relationships that would otherwise be almost invisible because one class dominates the ordinary softmax.
The temperature is therefore not cosmetic. It controls which distinctions are amplified for the student.
Response-Based Distillation Transfers Outputs
The simplest form asks the student to imitate the teacher’s final predictions.
The teacher remains a black box whose internal representation is not copied directly.
The student learns the teacher’s decision surface through output behaviour.
This can be highly effective because the receiver ultimately cares about outputs.
Feature Distillation Transfers Internal Representations
Other approaches align intermediate features between teacher and student.
This can encourage the student to develop similar internal abstractions, not merely similar final predictions.
But architectures can differ substantially.
A small student may not have enough representational capacity to reproduce the teacher’s internal feature space faithfully.
Attention Distillation Transfers Where the Teacher Looks
Some methods transfer attention patterns or saliency-like structures.
The student is taught not only what answer to produce but which parts of the input or representation deserve emphasis.
This narrows the teaching signal toward selected mechanisms.
A Smaller Student Has a Capacity Ceiling
If the student is only slightly smaller than the teacher, it may imitate many behaviours well.
If the capacity gap is enormous, some teacher distinctions cannot fit.
The loss is not necessarily training failure.
It can be a representational limit: the student does not have enough parameters, depth, width or architectural expressiveness to carry the teacher’s full behaviour.
The Teacher Can Transfer Its Mistakes Too
A teacher is not truth.
If it is biased, overconfident or systematically weak on a subgroup, imitation can propagate those defects.
Distillation can therefore preserve errors extremely efficiently.
The teacher must be audited as a source, not worshipped as an oracle.
Teacher Ensembles Can Distil Collective Behaviour
Several large models can act as teachers.
The student then learns from an averaged or otherwise combined signal.
This can compress the behaviour of an expensive ensemble into one deployable model.
The ensemble’s diversity is compressed into one parameter set.
Self-Distillation Removes the Large-Teacher Requirement
Knowledge can also be distilled from previous generations or branches of the same architecture.
Self-distillation shows that the technique is not only about model size.
It is about using a richer predictive distribution or representation as a teaching target.
Distillation and Pruning Remove Different Things
Pruning starts from one model and deletes parts.
Distillation trains a separate student to approximate selected behaviour of a teacher.
The pruned model retains architectural ancestry.
The distilled student can have a very different architecture.
Distillation and Quantisation Remove Different Freedom
Quantisation reduces numerical precision inside one model.
Distillation reduces behavioural complexity by training a new model under selected teacher signals.
These techniques can be combined, but their failure modes should still be measured separately.
Rare Capabilities Are Easy to Distil Away
Suppose 99.9% of training examples exercise ordinary behaviour.
A rare multilingual pattern, specialist domain or safety edge case appears only occasionally.
The student may reproduce the teacher’s average performance while failing precisely on those low-frequency abilities.
Distillation must therefore be evaluated beyond aggregate accuracy.
Calibration Can Change
Two models can make the same class predictions and assign very different confidence.
If a downstream system uses confidence thresholds, that difference matters.
A successful student should therefore be tested not only for accuracy but for calibration, uncertainty and decision behaviour near operational thresholds.
Education: Distillation Is Not Memorising the Teacher’s Answer
A strong tutor does not simply hand a child final answers.
The tutor exposes patterns, alternatives, misconceptions and the shape of reasoning.
A student then compresses years of expert experience into a smaller personal working model.
This is an analogy, not machine-learning mechanics. The shared principle is that a learner can inherit useful structure without reproducing the teacher’s entire internal history.
A CivDJ View: Distillation Must Test What Did Not Transfer
Forward: does the student reproduce the teacher where the receiver needs it?
Backward: which teacher capabilities vanished during transfer?
Rotate: do the gaps grow under new domains, rare classes, language shifts or adversarial conditions?
A successful transfer should be mapped by preserved and lost capability, not only by one headline score.
When Knowledge Distillation Is Exactly the Right Loss
Use it when a strong teacher is too expensive for deployment, a smaller student has enough capacity for the target behaviour, teacher signals add useful structure beyond hard labels, and evaluation covers the capabilities the receiver genuinely requires.
When Distillation Becomes Dangerous
- The teacher’s errors or biases are transferred.
- The student capacity gap is too large.
- Rare behaviours disappear despite stable average accuracy.
- Confidence calibration changes near decision thresholds.
- The student is treated as equivalent to the teacher without behavioural comparison.
- Distillation is combined with pruning or quantisation and cumulative losses are not isolated.
A Practical Distillation Audit
- Teacher: which model or ensemble supplies the signal?
- Signal: logits, probabilities, features, attention or another representation?
- Temperature: how is the target softened?
- Capacity: can the student express the behaviours being transferred?
- Edges: which rare capabilities were explicitly tested?
- Calibration: did confidence behaviour change?
- Master: is the teacher retained as a reference for regression testing?
Research Anchors
Recent surveys in 2024 and 2025 continue to describe knowledge distillation as a major efficiency and model-compression technique, including for large models and resource-constrained deployment. The literature increasingly studies architecture gaps, multimodal transfer, iterative combinations with pruning, and the problem of preserving subtle reasoning behaviour.
Continue Through eduKateSG
Continue with How Lossy Works | Model Pruning, How Lossy Works | Quantisation, and the companion How Lossy Works | Low-Rank Approximation.
Final Thought: A Student Can Carry the Teacher’s Shape Without Carrying the Teacher’s Size
Distillation is powerful because imitation can preserve behaviour without preserving mechanism.
Its danger is exactly the same.
The student may look like the teacher where we measure—and be different where we forgot to look.