A neural network can contain millions or billions of learned parameters.
Then somebody removes a large fraction of them.
The model still works.
That sounds like proof that the deleted parts were useless.
It is not always that simple.
Quick Read
Model pruning removes learned components from a neural network to reduce storage, memory traffic or computation. Pruning can target individual weights, neurons, channels, attention heads, filters or larger blocks. Modern 2025 reviews continue to treat pruning as a distinct model-compression family alongside quantisation, low-rank methods and knowledge distillation.
One-sentence answer: pruning is lossy because the network’s original parameter space is deliberately reduced, preserving enough task performance for the intended receiver while deleting degrees of freedom, redundancy and sometimes rare capabilities.
Why Large Networks Can Contain Removable Structure
Training does not usually ask for the smallest possible network.
It asks for a network that learns successfully.
Overparameterisation can make optimisation easier because many routes through parameter space can fit the training objective. After training, some parameters may contribute little to ordinary predictions.
Pruning asks which of those routes can disappear without breaking the behaviours the receiver still needs.
Unstructured Pruning Removes Individual Weights
The simplest version can set selected weights to zero.
Magnitude pruning often begins from an intuitive rule: very small weights may contribute less to the network’s output than large ones.
Remove enough small weights and the network becomes sparse.
The difficulty is practical. A sparse matrix can contain many zeros while ordinary hardware still spends time moving and multiplying them unless software and hardware exploit sparsity efficiently.
Structured Pruning Removes Whole Units
Instead of deleting isolated weights, structured pruning removes larger components: channels, filters, heads, layers or blocks.
This can produce hardware-friendly reductions because the resulting matrices are physically smaller and dense.
But larger deletions are more disruptive.
Removing a whole feature channel can erase a capability that individual weight pruning might have preserved through alternate routes.
The Pruning Criterion Is a Theory of Importance
Magnitude is only one criterion.
A component can also be scored by gradients, estimated sensitivity, activation statistics, second-order approximations, redundancy with other features or changes in validation loss after removal.
Every criterion answers a different question:
Important for what behaviour, under what data distribution, measured how?
The pruning rule does not merely find importance. It defines operational importance.
One-Shot Pruning Is Fast and Brutal
Delete a chosen fraction of parameters in one pass and evaluate what remains.
This can work surprisingly well at modest sparsity.
At higher pruning ratios, sudden removal can push the network far from the region of parameter space where its learned behaviours were stable.
Iterative Pruning Lets the Network Recover
Remove a smaller fraction.
Fine-tune.
Remove more.
Fine-tune again.
This staged process gives remaining parameters a chance to absorb work previously carried by deleted components.
Loss happens. Recovery reorganises what remains.
Fine-Tuning Does Not Restore the Deleted Model
This point matters.
After pruning and recovery, the smaller network can regain headline accuracy.
That does not mean it has recreated every behaviour of the original network.
Two models can score identically on a benchmark while differing on rare classes, calibration, robustness, multilingual behaviour or unusual prompts.
Redundancy Is Sometimes Insurance
Engineers often describe removable parameters as redundant.
But redundancy can serve several roles.
- Multiple units may represent similar features.
- Different routes may support the same ordinary output under different contexts.
- Extra capacity may help adaptation to new data.
- Rare failure modes may depend on components that contribute little to average accuracy.
Delete redundancy and the model can become efficient.
Delete too much and you may also delete resilience.
The Lottery Ticket Idea Changed How People Think About Pruning
The Lottery Ticket Hypothesis popularised the possibility that dense randomly initialised networks may contain smaller subnetworks capable of learning strongly when trained with suitable initialisation and procedure.
The deeper lesson for this series is not that every network contains one magical tiny answer.
It is that capability can be distributed unevenly across a network, and the full trained parameter count may not be necessary for every task.
Pruning Before, During or After Training Changes the Question
Post-training pruning asks what can be removed from an already learned network.
Pruning during training asks which capacity should be allowed to survive as learning unfolds.
Pruning at initialisation asks whether useful sparse structure can be identified before full expensive training.
Recent pruning reviews distinguish these timing regimes because they create different computational and optimisation trade-offs.
Attention-Head Pruning Makes the Loss Visible Conceptually
Transformer models contain many attention heads.
Some pruning methods remove heads judged less important.
If performance remains stable, the system has shown that the full set of attention routes was not necessary for that evaluation.
But a removed head may have specialised in a narrow pattern rarely represented in the benchmark.
Average usefulness and universal usefulness are not the same.
Pruning and Feature Selection Are Cousins, Not Twins
Feature selection removes input variables before or during modelling.
Pruning removes internal learned components of the model itself.
The first narrows what the model can see.
The second narrows how richly the model can process what it sees.
Pruning and Quantisation Remove Different Freedom
Quantisation keeps many parameters but reduces the number of representable values each parameter can take.
Pruning removes parameters or structures entirely.
One coarsens numerical precision.
The other removes dimensions of the computation.
The Wrong Benchmark Can Bless the Wrong Pruning
Suppose a model is pruned until its benchmark score drops by less than one percent.
That sounds safe.
But if the benchmark underrepresents minority classes, unusual languages, low-light images or rare safety cases, the pruning process may remove exactly the capability the test fails to measure.
A pruning audit must therefore test the receiver’s real edge conditions, not merely the average case.
Education: Pruning Is What Revision Does When Done Well
A student cannot carry every worked example into an examination.
Good revision reduces hundreds of pages into a smaller working system: principles, patterns, checks and representative examples.
Poor revision prunes the wrong thing—keeping surface steps while deleting the reasoning that lets the method transfer.
This is an analogy, not neural-network mechanics. The shared idea is selective removal under a performance test.
A CivDJ View: Pruning Must Be Tested Forward and Backward
Forward: can the smaller model still perform the intended task?
Backward: which removed components would have mattered if the receiver, domain or edge case changed?
Rotate: does the pruning remain safe across classes, environments, languages, hardware constraints and future adaptation?
The smaller model should not be judged only on the path that made it look successful.
When Model Pruning Is Exactly the Right Loss
Use it when deployment cost matters, removable redundancy has been demonstrated, the target hardware can exploit the resulting structure, and evaluation covers the behaviours the deployed receiver genuinely needs.
Pruning is strongest when the removed capacity is truly unnecessary for the intended operating envelope.
When Pruning Becomes Dangerous
- Average benchmark accuracy hides rare capability loss.
- Unstructured sparsity does not translate into real hardware speedup.
- Recovery fine-tuning overfits the same benchmark used to justify pruning.
- Redundancy needed for adaptation or robustness is deleted.
- The original model is discarded before the smaller model is fully validated.
- Pruning is combined with quantisation or distillation and losses accumulate without attribution.
A Practical Pruning Audit
- Unit: weights, neurons, channels, heads, layers or blocks?
- Criterion: how is importance defined?
- Timing: before, during or after training?
- Recovery: how is performance restored?
- Hardware: does theoretical sparsity create real efficiency?
- Edges: which rare or minority behaviours were retested?
- Master: is the original model retained as a reference?
Research Anchors
Recent 2025 surveys continue to treat pruning as a major model-compression family and distinguish pruning by granularity and timing. The literature also emphasises the central trade-off between reduced resource use and accuracy or capability degradation.
Continue Through eduKateSG
Continue with How Lossy Works | Quantisation, How Lossy Works | Feature Selection, and the companion article How Lossy Works | Knowledge Distillation.
Final Thought: A Smaller Network Is Not the Same Network Minus Waste
Pruning can reveal remarkable redundancy.
But every removal is still a hypothesis about what the future will not need.
The best pruning process proves that hypothesis across the real receiver space before it celebrates the smaller model.