VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

How Super Intelligence Works | Distillation and Specialisation — How Large Models Transfer Capability Into Smaller Focused Models

Three students work together at a classroom table, reviewing open textbooks and notes.

Knowledge distillation transfers useful behaviour from a larger or stronger teacher model into a smaller student model. The goal is not to copy every parameter. It is to train the student so its outputs, probability distributions or internal representations approximate important behaviour of the teacher under a defined task distribution.

Specialisation is the related process of adapting a general model toward a narrower domain, task, latency target or operating environment. A specialised model may be smaller, faster and cheaper than the general model while performing strongly on the work it was designed for.

This article explains distillation and specialisation: teacher–student training, hard labels, soft targets, temperature, response distillation, sequence distillation, domain adaptation, parameter-efficient fine-tuning, LoRA, QLoRA, quantisation, pruning, routing and evaluation. It also explains the central danger: a smaller model can imitate the teacher’s strengths and its errors.

The classic paper Distilling the Knowledge in a Neural Network describes transferring knowledge from an ensemble or large model into a smaller deployable model. DistilBERT later demonstrated distillation during pretraining for a smaller, faster general-purpose language representation model.

Previous: 029 — Synthetic Data. Distillation closes Part 3 by asking how broad learned capability can be compressed or focused without retraining every downstream system from scratch.


The Hidden Transition: Copy Behaviour, Not Weights

Suppose a teacher model has 100 billion parameters and a student has 10 billion. There is no one-to-one parameter mapping. Distillation instead gives the student examples of how the teacher behaves and trains the student to reproduce useful aspects of those outputs.

The student can therefore learn a smoother decision boundary than it would learn from hard labels alone. The teacher’s probability distribution contains information about which alternatives it considers similar or plausible.

The transfer is behavioural. A smaller student can approximate the teacher on the selected task distribution without becoming an identical internal machine.

Hard Labels Versus Soft Targets

A hard classification target says the correct class is CAT = 1 and every other class = 0. A teacher model might instead output CAT 0.80, TIGER 0.12, DOG 0.06, CAR 0.02.

The soft distribution tells the student more. Tiger is considered more similar to cat than car. Training on these relative probabilities can communicate structure that the one-hot hard label discards.

This “dark knowledge” intuition is central to classic distillation.

Temperature in Distillation

The teacher’s softmax distribution can be softened using a temperature greater than 1. Higher temperature makes lower-probability alternatives more visible, exposing relative similarities among classes.

The student can be trained to match this softened distribution, often alongside the ordinary hard-label objective.

Distillation temperature is a training hyperparameter. It should not be confused with generation temperature used when sampling language-model tokens at inference, even though both rescale logits before softmax.

A Tiny Worked Distillation Example

Teacher logits over four classes produce softened probabilities: CAT 0.55, TIGER 0.25, DOG 0.15, CAR 0.05. Student initially predicts CAT 0.40, TIGER 0.10, DOG 0.10, CAR 0.40.

A distillation loss penalises the student for placing too much mass on CAR and too little on TIGER and CAT. Backpropagation updates student parameters toward the teacher distribution.

A hard-label loss can simultaneously reinforce CAT as the correct answer. The combined objective balances ground truth with teacher structure.

Why Use a Teacher at All?

The teacher has already absorbed broad patterns through expensive pretraining and post-training. Distillation reuses that learned competence instead of asking the smaller student to rediscover everything from raw data under the same compute budget.

This can reduce deployment cost, memory and latency. The trade-off is that the student has less capacity and may fail on tasks outside the distillation distribution.

Teacher Quality Sets an Upper Boundary—But Not a Perfect One

A weak teacher produces weak targets. A biased teacher can transmit bias. An overconfident teacher can teach overconfidence.

Ground-truth labels, external verification and multiple teachers can prevent the teacher from becoming the only authority.

A student can occasionally outperform the teacher on particular metrics through regularisation or better task adaptation, but the training pipeline should not assume this automatically.

Response Distillation for Language Models

For open-ended language tasks, the teacher can generate full responses rather than class probabilities. The student then trains on prompt-response pairs created by the teacher.

Example prompt: “Explain why 3x = 21 gives x = 7 to a Secondary 1 learner.” A strong teacher generates a concise explanation. The student learns to reproduce similar instructional behaviour.

This overlaps with synthetic data. The distinction is intention: the teacher responses are generated specifically to transfer capability into the student.

Sequence-Level Distillation

Language output is a sequence. Instead of matching every vocabulary probability at every step, the student can train on teacher-generated sequences as target outputs.

This is simpler operationally and can produce strong behaviour, but it loses some of the teacher’s uncertainty information. One generated answer exposes only one path through a much larger distribution.

Logit Distillation

If teacher logits are accessible, the student can match token distributions directly. This transfers richer information but requires storing or computing large probability vectors.

For very large vocabularies and datasets, logit storage can be expensive. Practical systems may keep only top-k logits or use on-the-fly teacher inference.

On-Policy Distillation

A student can generate its own attempts and ask the teacher to correct or score them. This targets states the student actually visits rather than only teacher-generated outputs.

The method can reveal student-specific failure modes, but teacher inference remains expensive and the feedback loop needs quality controls.

Distillation During Pretraining

DistilBERT applied distillation during pretraining rather than only after task-specific fine-tuning. The published model reduced size while retaining much of BERT’s language understanding performance under the reported evaluations.

This shows that distillation can produce a smaller general-purpose base, not only a narrow classifier.

Task-Specific Distillation

A large general model can teach a small student for one bounded task: sentiment classification, document routing, entity extraction or school question tagging.

Because the task is narrow, the student can be much smaller than the teacher while preserving the needed performance.

This is specialisation through teacher supervision.

Domain Specialisation

A general model can be adapted for a domain such as law, medicine, finance, mathematics or education. Specialisation can use domain pretraining, supervised fine-tuning, preference data, retrieval, adapters or distillation.

The objective is not simply “know more domain words”. It is better performance on representative domain tasks under the deployment constraints.

Specialisation Is Not the Same as Memorising a Knowledge Base

If the domain changes frequently, retrieval may be better than baking current facts into parameters. A specialist legal model can understand legal language while still retrieving the current statute.

Specialisation should target durable patterns: terminology, reasoning forms, document structure and task behaviour. Changing state belongs in authoritative sources.

Full Fine-Tuning

Full fine-tuning updates most or all model parameters on downstream data. It offers maximum flexibility but requires substantial memory and compute for large models.

It also creates a full specialised checkpoint for each task, which can be expensive to store and maintain.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning updates only a small subset of parameters or adds small trainable modules while keeping the base model mostly frozen.

This reduces training memory and makes it easier to maintain many specialised variants.

LoRA

The LoRA paper proposes freezing pretrained weights and learning low-rank matrices that represent task-specific updates to selected weight matrices.

Because the added matrices are much smaller than the full weights, the number of trainable parameters drops dramatically. The adaptation can often be merged with the base weights for deployment.

LoRA is specialisation, not classical teacher-student distillation, but both pursue efficient transfer from a broad foundation into a more focused deployment.

A Low-Rank Intuition

Suppose a large weight matrix has dimensions 4,096 × 4,096. Updating every entry requires more than 16 million trainable values.

A rank-8 LoRA update represents the change as the product of matrices 4,096 × 8 and 8 × 4,096—about 65,536 values before other details. The hypothesis is that useful task adaptation can lie in a much lower-dimensional update space.

QLoRA

The QLoRA paper combines a frozen quantised base model with LoRA adapters, allowing fine-tuning of very large models with much lower memory usage under the reported setup.

Quantisation reduces the memory footprint of the base representation; LoRA limits the trainable parameters. The combination makes specialisation accessible on smaller hardware.

Quantisation Is Not Distillation

Quantisation represents weights or activations using fewer bits. The model can keep broadly the same architecture and parameters, only represented more compactly.

Distillation trains a student to imitate a teacher. Quantisation compresses numerical representation. They can be combined, but they are different mechanisms.

Pruning Is Not Distillation

Pruning removes weights, neurons, heads or structures judged unnecessary. It directly reduces parts of the existing model.

Distillation trains another model or representation to reproduce behaviour. A pruned model can itself be distilled or fine-tuned afterward.

Specialisation Through Routing

Another strategy keeps multiple specialised models and routes each request to the appropriate one: coding, mathematics, translation or lightweight chat.

Routing avoids forcing one small model to cover every domain. The cost is orchestration complexity and the risk of sending a task to the wrong specialist.

Mixture of Experts

Mixture-of-experts architectures contain specialised internal expert modules and a routing mechanism that activates only a subset for each token or input.

This is not the same as distilling one teacher into one student, but it reflects the same broad idea that specialised capacity can be allocated selectively.

Why Smaller Models Matter

Smaller models can reduce latency, inference cost, memory use and energy consumption. They can run closer to the user, sometimes on edge devices, improving privacy and availability.

A school or business may not need the largest model for a fixed document-classification task. A specialist student can be more practical.

Latency Versus Capability

A larger teacher may answer a complex question better but take longer and cost more. A distilled student can serve high-volume routine requests quickly.

The system can route difficult cases back to the teacher, creating a cascade: cheap student first, strong teacher when confidence or validation indicates difficulty.

Teacher–Student Cascades

A cascade uses the small model for ordinary cases and escalates uncertain examples. This preserves efficiency without pretending the student covers every long-tail problem.

Escalation criteria can use confidence, input type, failed verification or explicit user request.

Distillation for Tool Use

A strong agent can generate tool-use trajectories. A smaller student trains on those trajectories to learn which tool to call and how to format arguments.

The simulator or tool schema can verify calls. Incorrect teacher trajectories should be filtered rather than copied.

Distillation for Reasoning

Teacher models can generate worked solutions for mathematics, code or planning. Students train on verified solution traces and outcomes.

The key word is verified. A polished teacher explanation can contain a hidden logical error. Deterministic tools and answer checks remain valuable.

Distillation for Retrieval

A large cross-encoder or reranker can label relevance pairs for a smaller bi-encoder retriever. The student learns a cheaper similarity function that approximates the teacher’s ranking behaviour.

This makes high-quality retrieval possible at much larger search volumes.

Distillation for Classification

A high-capacity teacher can produce soft class probabilities on a huge unlabeled dataset. A small classifier trains on those soft labels plus real ground truth.

The student gains more supervision without manual labels for every example.

Specialisation Data Quality

A specialist is only as representative as its adaptation data. Fine-tuning on one customer’s wording can make it brittle to another customer.

Use held-out domain data and broad regression sets to measure both gains and lost generality.

Catastrophic Forgetting

A model specialised aggressively on one narrow domain can lose performance on capabilities it previously had.

Parameter-efficient methods, mixed training data and regression evaluation can reduce or detect forgetting.

Over-Specialisation

A student trained only on concise customer-service answers may refuse to produce longer explanations even when needed.

Specialisation should match a clearly bounded operating role. If requirements broaden, routing to a general model may be better than forcing one specialist to stretch.

Teacher Bias Transfer

If the teacher systematically makes one mistake, the student can reproduce it at scale. This is especially likely when teacher output replaces ground truth.

Mix verified labels, multiple teachers or external checks into the training process.

Student Capacity Limits

A tiny student cannot necessarily represent every teacher behaviour. Distillation must choose which capabilities matter most.

Performance loss can concentrate in rare, compositional or long-context tasks even when average benchmarks remain strong.

A Distillation Failure Map

Level 1: weak teacher. Level 2: unrepresentative distillation prompts. Level 3: poor target temperature or loss balance. Level 4: student too small. Level 5: teacher errors copied. Level 6: average quality preserved but long-tail capability lost. Level 7: domain specialisation causes forgetting. Level 8: deployment exceeds specialist scope.

This eduKateSG map helps identify why a compact model missed the target rather than declaring distillation itself a failure.

Worked Diagnosis: Student Matches Benchmark but Fails Long Inputs

The distillation set may contain short prompts only, while the teacher was evaluated on longer contexts. Add long-context examples and measure whether the student architecture can support them.

Worked Diagnosis: Student Too Confident

If distillation uses only hard teacher outputs, the student loses teacher uncertainty information. Soft targets or calibration data may improve behaviour.

Worked Diagnosis: Specialist Forgets General Language

The fine-tuning mixture may be too narrow or aggressive. Mix a small amount of general data, reduce adaptation strength or route general tasks to the base model.

A Practical Distillation Experiment

Define one deployment goal: reduce inference cost for a support classifier while preserving 98% of the teacher’s task accuracy. Select a student size and representative prompt set.

Generate teacher logits or labels, combine them with ground truth, train the student, then evaluate real held-out examples, calibration, latency and memory.

Only ship if the student meets both quality and efficiency thresholds.

Independent Exercise 1: Hard Versus Soft Target

Teacher says CAT 0.60 and TIGER 0.35. What information disappears if the student sees only hard label CAT?

Answer

The student loses the teacher’s indication that TIGER is a plausible alternative and much closer to CAT than unrelated classes.

Independent Exercise 2: Quantisation

A model’s weights are stored in 4-bit representation but no student is trained. Is that distillation?

Answer

No. That is quantisation. Distillation requires teacher-to-student training or behaviour transfer.

Independent Exercise 3: Specialist Scope

A small model distilled for invoice extraction is asked to write creative fiction. Should failure surprise us?

Answer

No. Its deployment scope is narrow. Route out-of-scope tasks to a general model.

Independent Exercise 4: Teacher Error

The teacher gives a wrong answer that passes no external verification. Should the student train on it?

Answer

No. Filter or correct the example. Teacher authority is not ground truth.

Independent Exercise 5: LoRA

Does LoRA create a smaller base model?

Answer

Not necessarily. LoRA makes adaptation parameter-efficient by learning small low-rank updates while the base model remains. It is specialisation, not automatically model-size compression.

The Distillation Loss in More Detail

A common distillation objective combines two losses. The first compares the student with ground-truth labels. The second compares the student’s softened output distribution with the teacher’s softened distribution.

If α controls the mixture, a simplified objective can be written as α × hard-label loss + (1−α) × distillation loss. Different implementations scale the temperature-related term carefully because temperature changes gradient magnitude.

The purpose of the mixed loss is to preserve the real target while using the teacher’s richer relative probabilities.

KL Divergence and Distribution Matching

Distillation often uses cross-entropy or KL divergence to measure the difference between teacher and student probability distributions.

If the teacher assigns [0.70, 0.20, 0.10] and the student [0.40, 0.30, 0.30], distribution-matching loss pushes the student toward the teacher’s shape rather than merely telling it which class ranks first.

This makes the student sensitive to the teacher’s notion of similarity between alternatives.

Temperature Squared and Gradient Scale

Classic distillation formulations multiply the soft-target loss by a factor related to temperature squared. Softening logits changes derivative scale, so the correction keeps the relative contribution of distillation stable.

The exact formula depends on implementation. The practical point is that temperature affects both what information is exposed and how strongly gradients act.

Choosing Temperature

Low temperature makes the teacher distribution sharp, approaching hard labels. Higher temperature reveals lower-ranked alternatives but can make the distribution nearly uniform if pushed too far.

Tune temperature on held-out data rather than copying a value blindly. The right setting depends on class count, teacher calibration and task.

Intermediate Feature Distillation

The student does not have to imitate only final outputs. It can match teacher hidden representations, attention maps or intermediate features.

This may transfer richer structure, especially when teacher and student architectures align. It also complicates training because layers may have different dimensions.

Projection Layers for Feature Matching

If teacher hidden size is 1,024 and student hidden size is 512, a learned projection can map one representation space into the other for comparison.

The projection becomes part of the distillation objective. It should not be mistaken for evidence that the two internal representations are identical.

Attention Distillation

A student can be trained to approximate selected teacher attention patterns. This may help preserve relationships among input positions.

Attention matching is only one possible signal. Similar attention maps do not guarantee identical reasoning or output quality, so downstream evaluation remains necessary.

Layer Mapping

A 24-layer teacher and 6-layer student cannot match layers one-to-one. A distillation scheme may map every fourth teacher layer to one student layer or choose semantically useful checkpoints.

Layer mapping is a design decision. Poor mapping can constrain the student toward representations it cannot reproduce efficiently.

Teacher Ensembles

The teacher can be an ensemble of models rather than one model. Averaged probabilities can be better calibrated and less noisy than a single teacher.

The student compresses ensemble behaviour into one deployable network, which was one of the motivations in the classic Hinton distillation paper.

Specialist Teachers

A general teacher can be combined with specialists for classes it finds difficult. The student learns from both broad and fine-grained expertise.

The training pipeline needs a clear rule for combining teacher signals. Conflicting teachers should not silently create contradictory targets.

Teacher Confidence Should Not Override Ground Truth

If the teacher is confidently wrong on a labelled example, ground truth should usually dominate under the task definition.

Teacher outputs are additional supervision, not sovereign truth. Distillation works best when teacher quality is high and verified.

Unlabelled Data for Distillation

Teacher models can label large unlabelled corpora, creating soft targets without manual annotation. This expands student training data cheaply.

The student can then learn from real input distribution even when labels are scarce. Teacher errors remain the main risk.

Domain-Matched Distillation Data

A student intended for Singapore secondary maths should train on representative Singapore secondary maths prompts, not generic conversation alone.

The teacher may be general, but the distillation distribution defines which slice of teacher capability the student inherits most strongly.

Broad Distillation Before Narrow Specialisation

One strategy first distils a general student across broad data, then fine-tunes the student for a target domain. Another distils directly on the domain.

Broad-first preserves more generality; direct domain distillation can be cheaper. The deployment role determines the better trade-off.

Specialisation Through Continued Pretraining

A base model can continue self-supervised pretraining on domain text such as legal cases, scientific papers or code before task-specific tuning.

This changes parameters using domain distribution rather than explicit teacher responses. It is another specialisation route distinct from distillation.

Specialisation Through SFT

Supervised fine-tuning teaches a model domain tasks with labelled prompt-response examples. This can be combined with distillation by having a teacher generate or improve the targets.

The line between synthetic instruction data and teacher distillation often depends on whether teacher behaviour itself is the transfer objective.

Specialisation Through Preference Optimisation

After domain SFT, preference learning can shape the specialist toward domain-specific response standards: concise legal citations, educational scaffolding or code-review conventions.

Again, each training stage serves a different purpose. Specialisation is a stack, not one algorithm.

Specialisation Through Retrieval

Not every domain adaptation requires parameter updates. A general model connected to a high-quality domain retrieval system can behave like a specialist because it receives the right current evidence.

Retrieval is especially attractive when domain facts change frequently. Parameter specialisation is stronger for durable language, reasoning and format patterns.

When Retrieval Beats Fine-Tuning

If the main failure is missing current policy, fine-tuning on last month’s policy simply bakes staleness into weights. Connect the current document source instead.

If the failure is inability to understand the policy’s legal language even when supplied, domain adaptation may help.

When Fine-Tuning Beats Prompting Alone

If the same specialised behaviour must occur across millions of requests and prompts become long, brittle or expensive, parameter adaptation can make the behaviour more native to the model.

Measure whether the gain justifies maintaining a separate checkpoint or adapter.

Adapters as Deployable Specialisations

A base model can remain shared while each domain has a small adapter: education, finance, coding or support.

At runtime the system loads or selects the relevant adapter. This reduces storage compared with one full fine-tuned model per domain.

Adapter Routing

The application can classify the task and activate the corresponding adapter. Wrong routing becomes a new failure mode.

A fallback to the general base or multi-adapter composition can handle ambiguous tasks, but composition should be evaluated because adapters can interfere.

Merging Adapters

Some LoRA adaptations can be merged into base weights for deployment. This removes separate adapter operations at inference but creates a new specialised checkpoint.

Keep version records so the merged model can be traced back to its base and adapter training data.

Quantisation After Distillation

A distilled student can be quantised further, producing additional memory and speed savings.

Compression methods stack, but errors can stack too. Evaluate after each transformation rather than assuming the benefits are additive without quality loss.

Pruning After Distillation

A student can also be pruned to remove low-value parameters or structures. Fine-tuning may recover some lost performance.

The final system should be evaluated under the exact compressed configuration that will ship.

On-Device Specialisation

Small models can run on phones, laptops or embedded hardware. This reduces network latency and can keep sensitive data local.

Hardware constraints become part of model design: memory, power, accelerator support, thermal limits and model format.

Edge Privacy

Running locally can reduce transmission of raw data to a server, but the device still needs access control, secure model files and careful logging.

Privacy improvement is architectural, not guaranteed merely by using a smaller model.

Offline Availability

A specialised student can continue functioning without network access. This matters in classrooms, field devices or locations with unreliable connectivity.

The trade-off is freshness: local weights do not automatically receive updated policy or knowledge. Local retrieval stores or periodic sync may be needed.

Distillation and Latency Budgets

A 50-millisecond classification target and a 10-second research task have different model budgets. Distillation is most valuable when runtime constraints are explicit.

Measure time-to-first-token, tokens per second, memory footprint and total task latency rather than parameter count alone.

Throughput

A smaller model can serve more concurrent requests on the same hardware. This can matter more economically than a small difference in benchmark accuracy.

The deployment decision should compare quality per unit cost or throughput under realistic load.

Distillation and Energy

Smaller runtime models generally require less compute per inference. Actual energy depends on hardware, batch size, utilisation and data-centre conditions.

Do not infer precise environmental savings from parameter count alone. Measure the deployed workload if the claim matters.

Knowledge Loss Is Often Uneven

A student may preserve common facts and lose rare knowledge. It may retain short reasoning and lose long compositional tasks. Average accuracy can hide these losses.

Break evaluation down by difficulty, rarity, context length and domain.

Tail Evaluation

Construct a long-tail set of rare classes, unusual wording and hard combinations. Compare teacher and student directly.

If the student fails tails that matter operationally, route those cases to the teacher rather than pretending average parity is enough.

Calibration After Distillation

A student can match teacher accuracy while becoming more or less calibrated. Evaluate probability quality separately if confidence drives decisions.

Temperature scaling or post-hoc calibration can help, but it should be measured on held-out data.

Robustness After Distillation

Test typos, paraphrases, missing fields, adversarial formatting and domain shift. Smaller students can rely more heavily on shortcuts.

The student must survive the input variation expected in real use.

Safety After Distillation

Safety behaviour can regress even when ordinary benchmarks remain strong. Include refusal calibration, permission boundaries and harmful-content evaluations appropriate to the deployment.

If teacher safety is transferred through generated responses, verify that the student does not over-refuse harmless tasks or under-refuse unsafe ones.

A Teacher–Student Evaluation Matrix

Columns: teacher, student, compressed student. Rows: core task, long tail, calibration, latency, memory, throughput, robustness, safety and cost.

The matrix makes trade-offs explicit. A smaller model may win deployment even if it loses a few percentage points on one benchmark, provided the lost capability is not critical.

Cascades as a Control Strategy

Run the student first. If verification passes and confidence is sufficient, accept the result. Otherwise escalate to the teacher.

This architecture converts capability differences into routing rather than a binary choice between cheap and strong models.

Cascade Failure Modes

If the student is confidently wrong, confidence-based escalation fails. Add external verification or input-type rules for known difficult cases.

If too many cases escalate, the cascade saves little cost. Measure escalation rate and teacher load.

Specialisation Registry

For each specialist, record base model, adaptation method, data version, intended tasks, prohibited tasks, evaluation set, latency, memory, owner and fallback route.

This prevents a collection of adapters and student models from becoming an undocumented zoo.

Version Compatibility

An adapter trained for one base checkpoint may not work correctly with another. A distilled student’s tokenizer can also differ from the teacher’s.

Treat base–adapter–tokenizer combinations as versioned artefacts.

Distillation Data Provenance

Store prompt source, teacher model version, generation settings and verification status with each teacher-generated example.

If the teacher is later found to have a systematic error, provenance helps identify affected student data.

Independent Evaluation Must Not Be Teacher-Only

If student quality is measured only by agreement with the teacher, teacher mistakes are invisible. Include ground truth, tool verification or human domain evaluation.

Teacher agreement is useful for transfer fidelity; task correctness is the ultimate deployment requirement.

A Clementi-Style Distillation Decision Sheet

Question 1: Which runtime constraint requires a smaller or focused model? Question 2: What capability must be preserved? Question 3: What teacher evidence is available—logits, responses, representations or rankings? Question 4: What student size can plausibly hold the target behaviour?

Question 5: What real ground truth protects against teacher errors? Question 6: Which long-tail tasks must survive? Question 7: What latency, memory and cost target defines success? Question 8: What tasks will escalate to the teacher?

Distillation succeeds when the student meets a bounded operating specification, not when it resembles the teacher in every possible way.

Independent Exercise 6: Soft Target

Teacher gives [0.51, 0.48, 0.01] for three classes. What does hard-label training lose?

Answer

It loses the information that the first two classes are almost equally plausible and the third is very unlikely.

Independent Exercise 7: Routing

A specialist handles 95% of invoices well but fails handwritten scans. What is a sensible architecture?

Answer

Route ordinary invoices to the specialist and escalate handwritten or failed-verification cases to a stronger vision-capable model or human process.

Independent Exercise 8: Retrieval Versus Specialisation

The model misunderstands current tax rates only because they change frequently. Should you distil a new model every week?

Answer

Probably not. Retrieve the current authoritative rates. Specialise the model for interpreting tax documents, not for memorising volatile values.

Independent Exercise 9: Regression

Student is 3× faster but loses most rare-language capability. Can average English benchmark parity justify deployment everywhere?

Answer

No. Deployment scope should exclude unsupported languages or route them to another model. Evaluate the populations the system will actually serve.

A Full Deployment Case: Distilling a School Support Assistant

Suppose a large teacher model handles school support questions well but is too expensive for high-volume routine use. The deployment goal is a smaller student that can classify requests, answer common policy questions from retrieved sources and escalate unusual cases.

Build the distillation set from real anonymised task shapes: scheduling, fees, lesson materials, location, cancellation and curriculum questions. The teacher produces candidate responses, but policy facts are checked against authoritative documents before acceptance.

Train the student on teacher responses plus hard labels for classification and tool schemas. Evaluate on private real-user prompts the teacher never generated. Measure classification accuracy, grounded-answer correctness, refusal calibration, latency and escalation rate.

If the student handles 90% of routine cases while preserving required quality, route those cases locally or on cheaper infrastructure. Escalate ambiguous policy questions, unusual language and failed verification to the teacher. The result is a system architecture, not merely a compressed checkpoint.

Distillation Can Be Iterative

A first student may expose systematic gaps. The teacher generates targeted examples for those gaps, the student retrains and the regression suite expands.

This iterative loop should remain anchored to independent evaluation. Otherwise the student and teacher can converge on shared mistakes.

Do Not Distil What the System Can Retrieve Exactly

A student does not need to memorise every current room number, timetable or policy date. Distil the skill to read and apply the data; retrieve the current value at runtime.

This separation keeps small models useful longer because changing facts do not require continual re-distillation.

Specialisation and Organisational Ownership

A specialist model needs an owner who defines its scope and updates its evaluation set. “Finance specialist” is too broad unless the permitted tasks, source systems and escalation rules are clear.

Ownership also prevents orphaned adapters from remaining in production after policy or base-model changes.

Retirement Criteria

A student should be retired or retrained when deployment distribution changes materially, the teacher architecture changes, the base tokenizer becomes incompatible, evaluation drops below threshold or the cost advantage disappears.

Model lifecycle is part of specialisation. A compact model is not a permanent asset simply because it once passed its benchmarks.

What Progress Should Look Like

Progress is not just smaller parameter count. A successful distillation project preserves the required quality, lowers latency or cost, fits the target hardware, maintains calibration and safety, and has a clear fallback path for cases outside the student’s competence.

These are observable system outcomes. Compression without operational benefit is only a smaller file.

Independent Exercise 10: Volatile Knowledge

A specialist is accurate on last month’s school timetable but stale this month. Which layer should change first?

Answer

The live data or retrieval layer. The model should specialise in interpreting schedules, while the current schedule remains external state.

Independent Exercise 11: Teacher Agreement

Student matches teacher 99% of the time, but both fail the same 10% of real ground-truth cases. Is distillation successful?

Answer

Not for the deployment objective. High teacher agreement only proves transfer fidelity. Ground-truth performance still fails.

The Distillation Acceptance Test

Before publishing a student model, define an acceptance table rather than one average benchmark. Core task accuracy must exceed its threshold. Long-tail cases must stay within an allowed gap from the teacher. Calibration must remain usable if confidence controls routing. Safety and permission tests must not regress.

Then add operating constraints: memory footprint, time to first response, throughput under expected concurrency and cost per completed task. The purpose of distillation is deployment efficiency, so runtime measurements belong beside model-quality measurements.

Finally test fallback. Deliberately send inputs outside the student’s scope and confirm that the system escalates rather than fabricates competence. A specialist without a working escape route is brittle even if its in-scope score is excellent.

A Three-Band Student Model

One practical design divides cases into three bands. Green: the student handles the task and verification passes. Amber: the student produces a candidate but uncertainty, novelty or failed validation triggers teacher review. Red: the input is clearly outside the specialist’s authorised domain and routes immediately to another system.

This architecture turns model capacity into an explicit operating envelope. The student does not need to become a smaller copy of the teacher across every possible task; it needs to own its green band reliably and recognise the rest.

Distillation Economics Need End-to-End Measurement

A model that is 50% cheaper per token may not reduce total task cost if it produces more retries, escalations or human corrections. Measure cost to a verified outcome.

Conversely, a student that is slightly less accurate in free-form prose can still be the better system if deterministic checks catch its mistakes cheaply and routine volume is enormous. Economics belongs to the full workflow, not the checkpoint alone.

Specialisation Can Improve Privacy by Reducing Reach

A specialist designed only for local document classification may not need open-web access, broad tool permissions or a large persistent memory. Narrowing model role can therefore reduce the amount of data and capability exposed in the system.

This is an architectural benefit rather than an inherent property of small models. A poorly configured small model can still receive excessive access.

A Final Specialist Compatibility Box

Base: which foundation model or teacher. Method: distillation, SFT, LoRA, QLoRA, continued pretraining or combination. Scope: exact tasks owned. Evidence: ground truth and teacher source. Runtime: latency, memory and hardware.

Fallback: teacher, general model or human. Freshness: what facts remain external. Regression: capabilities that must not disappear. Owner: team responsible for updates. Retirement: conditions that invalidate the specialist.

If those fields are missing, “specialised model” is a description of intent rather than a controlled deployment asset.

Independent Exercise 12: Cost Per Outcome

Student inference is half the price of teacher inference but needs twice as many retries. Is it automatically cheaper?

Answer

No. Compare total cost and latency to a verified result, including retries, escalation and human correction.

Independent Exercise 13: Specialist Envelope

A maths specialist receives a legal contract. What is the strong system behaviour?

Answer

Recognise the input as out of scope and route to a model or human process designed for legal work rather than improvising from a narrow specialist.

The final rule is scope before compression: write the student’s operating envelope first, then choose the smallest model that can meet it with evidence. If the scope remains undefined, every lost capability looks like a surprise. If the scope is explicit, distillation becomes an engineering trade-off that can be measured, routed and repaired.

This specification-first approach also makes later replacement easier: a newer student can be tested against the same envelope without redefining success.

Frequently Asked Questions About Distillation and Specialisation

What is knowledge distillation?

Training a student model to approximate useful behaviour or representations of a stronger teacher model.

Why use soft targets?

They reveal relative probabilities among alternatives, transferring more information than a one-hot hard label.

Does distillation always make the model smaller?

Often that is the goal, but distillation can also transfer between similarly sized architectures or specialise a model.

What is LoRA?

A parameter-efficient fine-tuning method that freezes base weights and learns low-rank updates to selected matrices.

Is LoRA distillation?

No. It is an adaptation technique. It can be combined with teacher-generated data or distillation objectives.

What is QLoRA?

An efficient fine-tuning method using a quantised frozen base model plus LoRA adapters to reduce training memory.

What is specialisation?

Adapting a broad model toward a narrower domain, task, latency profile or hardware environment.

Can a specialist outperform its teacher?

On some narrow metrics it can, especially when task-specific adaptation and regularisation help, but this must be demonstrated rather than assumed.

Distillation Turns Broad Capability Into Deployable Focus

A giant model is not always the right runtime. Distillation and specialisation let teams transfer useful capability into models that are smaller, faster, cheaper or better matched to one domain.

The trade-off is scope. Compression can erase long-tail ability; specialisation can cause forgetting; teacher errors can propagate. The student must be evaluated on the exact work it will own.

Continue through the How Super Intelligence Works hub. Previous: 029 — Synthetic Data. Next: 031 — Inference.

How to Decide Whether Distillation Was Worth It

A successful distillation project should be judged on the complete deployment trade-off: preserved task quality, measured regressions, latency, throughput, memory footprint, serving cost, calibration and failure recovery. A smaller model is not automatically a better model; it is better only when the efficiency gain exceeds the capability lost for the real workload.