A tiny message can hide a giant theory.
Suppose I give you a million data points and say, “They all lie exactly on this one equation.” That sounds wonderfully compressed.
But what if the equation itself takes 999,900 characters to write?
We have not performed a miracle. We have moved most of the description into the model.
This is one of the most important corrections to naive thinking about compression: when a model helps us describe data, the model is part of the bill.
Quick Read
- A model can compress data by capturing regularities.
- The model itself has a description cost.
- A very simple model may leave a large residual.
- A very complicated model may fit the data perfectly but become expensive to describe.
- Minimum Description Length formalises the idea of balancing model cost and data-given-model cost.
- Overfitting can be seen as a model becoming too elaborate for the genuine structure it captures.
- The smallest useful explanation is not necessarily the model with the lowest training error.
The One-Sentence Answer
If a model makes the data cheaper to describe, honest compression must count both the cost of describing the model and the cost of describing what the model still fails to explain.
Two Bills, Not One
Imagine we want to describe a sequence of points.
Option A: send every point independently.
Option B: send a line equation, then send only the small deviations of each point from that line.
If the data really is close to a line, Option B may be much shorter.
But the total description is:
total description = model description + residual description
Make the model simpler and the first term shrinks, but the residual may grow. Make the model more elaborate and the residual may shrink, but the model becomes expensive.
A Straight Line Versus a Monster Polynomial
Suppose ten observed points roughly follow a straight trend, with a little noise.
A straight line gives a compact story: two parameters plus residuals.
Now fit a high-degree polynomial through every point exactly. The residual becomes zero.
Have we found a better explanation?
Not automatically. The polynomial may require many coefficients. We removed residual cost by inflating model cost.
This is the compression view of overfitting. A model can memorise the data rather than capture a reusable regularity.
What Minimum Description Length Is Trying to Do
Minimum Description Length, usually abbreviated MDL, is a principled way to think about model selection using this trade-off.
The broad intuition is simple enough for a Secondary student:
Prefer the explanation that lets you describe the model and the data together most economically.
That is more subtle than “choose the simplest model.” A model can be too simple and fail to capture obvious structure. Nor is it “choose the best fit.” A perfect fit can be purchased by an absurdly complicated model.
MDL asks us to pay for both.
The Theory Must Earn Its Complexity
A more complicated model is justified when it reduces the unexplained part by more than it costs.
Suppose Model A requires 100 bits to specify and leaves 10,000 bits of residual. Total: 10,100 bits.
Model B requires 2,000 bits to specify and leaves only 2,500 bits of residual. Total: 4,500 bits.
Model B is more complicated, but it earns that complexity.
Now suppose Model C requires 20,000 bits and leaves zero residual. Total: 20,000 bits.
Perfect fit, terrible description length.
Scientific Theories as Compression
There is a beautiful analogy here.
Before a theory, we may possess thousands of observations. After a good theory, many of those observations become consequences of a smaller set of principles plus initial conditions.
Kepler’s laws compress patterns in planetary motion. Newtonian mechanics unifies a wider family of motions under fewer principles. The periodic table organises many chemical regularities. Evolution by natural selection provides a generative framework linking vast biological observations.
But the analogy must remain disciplined. Scientific theories are judged by evidence, prediction, causal adequacy, explanatory power and empirical survival—not by file size alone. Compression is a useful lens, not the entire philosophy of science.
A Model Can Smuggle In Prior Knowledge
Suppose sender and receiver already share a sophisticated language model. Then a short prompt or correction sequence may allow a large text to be reconstructed or predicted with relatively little additional information.
Is the message astonishingly small? Yes.
Is the entire system astonishingly small? No. The shared model may contain billions of parameters and embody enormous prior computation and training data.
This is another form of borrowed space. The model is part of the receiver’s installed context.
Why “Just Use a Bigger Model” Eventually Hurts
A richer model can discover more structure, but it creates several costs.
- More parameters may need storage.
- Training may consume more computation.
- Encoding may become slower.
- Decoding may require more memory.
- The model may fit accidental noise.
- Maintenance and versioning become harder.
- Distribution shift can make specialised structure brittle.
Compression is not impressed by complexity for its own sake. Complexity must reduce the remaining description enough to justify itself.
The Lookup Table That “Compresses Everything”
Imagine someone builds an enormous table containing every file they expect to encounter. Each file receives a short ID:
1 → giant movie A 2 → giant movie B 3 → giant database C
Now the transmitted representation may be one byte.
Have we compressed those movies to one byte?
Only if we conveniently refuse to count the giant lookup table already installed at the receiver.
This example exposes a general rule: external model complexity does not disappear merely because it is amortised or pre-shared.
Compression, Generalisation and Memorisation
A useful model captures a pattern that applies across many observations. That is why it can reduce description length repeatedly.
Memorisation is different. If I store every answer individually, I may reproduce the training set perfectly, but I have not necessarily found a shorter generative rule.
Generalisation is valuable because one piece of structure explains many cases.
Compression rewards reusable structure.
A Grammar Is a Model Too
Human language gives us another intuition.
Instead of memorising every grammatical English sentence ever spoken, a learner acquires reusable patterns: word classes, phrase structures, tense relationships, agreement, clause combinations and conventions.
The grammar acts like a model. Individual sentences are instances generated within or interpreted through that model.
But an overcomplicated grammar that merely lists every sentence would miss the point. The value lies in capturing regularities that apply beyond one example.
Education: Notes Versus Mental Models
A student can copy 50 pages of notes or build a smaller conceptual structure that organises those pages.
For example, a learner who understands conservation can connect apparently separate problems in energy, momentum or mass balance. A learner who understands cause and consequence can organise historical events more effectively than one who stores isolated dates.
The internal model compresses the curriculum by finding relations.
But there is a warning. A student’s model must actually reconstruct the relevant detail when needed. A beautifully simple slogan that cannot recover mechanisms, evidence or exceptions is underfitting.
Primary School: Rule Plus Exceptions
Give a child the list:
2, 4, 6, 8, 10, 13, 14, 16
Ask for a compact description.
“Even numbers from 2 to 16, except 12 is replaced by 13” is shorter than listing everything only if the learner already understands even numbers and exceptions.
Now compare it with an excessively elaborate rule invented just to reproduce every number. Students can feel the difference between useful pattern and forced complexity.
Secondary School: Fit and Simplicity
Plot several points with a roughly linear trend. Ask students to draw a simple line of best fit. Then show them a wildly curving line that passes through every point.
Which line explains the underlying pattern better?
This is a visual doorway into model complexity, noise and overfitting.
JC and Beyond: MDL as an Inference Principle
At a more advanced level, MDL treats learning as finding regularity that permits a short joint description of model and data. Different formal versions handle coding details carefully, but the central intellectual move is consistent: do not evaluate a model only by how well it fits; evaluate the complexity required to obtain that fit.
This creates a bridge to Bayesian thinking, information criteria and regularisation, though the frameworks are not identical. All wrestle, in different mathematical languages, with a familiar problem: how much complexity should evidence justify?
The Model Can Be Wrong in Two Directions
- Underfitting: the model is too simple and leaves predictable structure unexplained.
- Overfitting: the model becomes so tailored to the observed sample that it spends complexity on noise or accidental detail.
Compression gives us a clean mental picture. Underfitting pays too much in residuals. Overfitting pays too much in model description.
What Counts as the Model?
This question depends on the accounting boundary.
If every device already implements the same standard decoder, we may amortise that decoder over billions of files. If a specialised dictionary is sent once per archive, perhaps it belongs to archive overhead. If a model is unique to one object, its cost should be attributed more directly.
There is no single bookkeeping convention for every engineering purpose. But there is a discipline:
Do not claim the model is free merely because you placed it outside the payload.
A Theory That Compresses Yesterday May Fail Tomorrow
Models encounter changing worlds.
A model that fits one era of language, markets, traffic or user behaviour may lose efficiency when the distribution shifts. Residuals grow because reality stops matching expectations.
This is another reason generalisation matters. A compact model must capture stable enough structure to remain useful beyond the sample that produced it.
The Deep Connection to Explanation
Good explanations often feel compressive because they replace many disconnected facts with a small number of relations.
“Objects fall” is a fact. A theory of gravity connects falling objects, planetary orbits and tides within a wider structure. The theory earns complexity by organising more phenomena.
Likewise, a good educational concept acts as a reusable model. It should help the learner predict, reconstruct and transfer—not merely repeat a definition.
A Useful Test for Any “Smart Compression” Claim
- What model or dictionary is being used?
- Where is that model stored?
- How large is it?
- Was it trained or transmitted earlier?
- How much residual information remains?
- Does the model generalise beyond the demonstration data?
- What computation is required at encode and decode time?
- Would the claimed compression still look impressive if the full system cost were counted?
The Deeper Point
A model is a machine for turning regularity into economy.
But every machine has a size.
The profound compression question is therefore not:
How small can I make the data?
It is:
What combination of reusable model and remaining surprise gives the smallest, most useful total description?
The model captures what repeats. The residual records what it misses. Their sum tells us whether the theory truly earned its place.