What if compression is really a game of prediction?
Suppose I ask you to complete this sentence:
The sun rises in the ____.
You probably do not need many guesses. “East” is strongly expected. Now suppose I ask for the next result from a fair coin toss. Suddenly prediction becomes much weaker.
That difference—between what is expected and what is surprising—is one of the deepest ideas in compression.
A good compressor does not merely look backward for exact repetition. It can also build a model of what is likely to happen next, then spend fewer bits on expected events and more bits on unexpected ones.
Quick Read
- Prediction reduces description length when the next symbol, value or pattern is not equally likely to be everything.
- A model assigns probabilities to possible next events.
- Common, predictable events can receive shorter codes; surprising events require more information.
- Predictive compression often stores residuals—the difference between prediction and reality.
- Context matters: the same symbol may be predictable in one context and surprising in another.
- Better prediction can create better compression, but more complex models also cost computation, memory and sometimes model-description space.
- Compression therefore connects probability, modelling, language, images, audio, science and learning.
The One-Sentence Answer
Prediction helps compression because when a model can assign high probability to what actually happens, the system needs fewer bits on average to identify that outcome among the alternatives.
Compression Does Not Need Perfect Prediction
People sometimes hear “prediction” and imagine prophecy. That is not what a compressor needs.
A compressor can benefit from prediction even when it is frequently wrong. What matters is whether its probability assignments are useful on average.
Suppose a source produces only A and B. If A occurs 99% of the time and B only 1%, treating them as equally likely wastes an opportunity. A good code can give the common event a cheaper representation and reserve more description length for the rare event.
The compressor is effectively saying: “I expected A. Tell me only when the world deviates from that expectation, and pay proportionally for how surprising the deviation was.”
Surprise Has a Price
In information theory, an event with probability p is associated with an information quantity proportional to minus the logarithm of p. We do not need to turn this article into a mathematics lecture to appreciate the intuition.
- If something is very likely, learning that it happened tells you relatively little.
- If something is unlikely, learning that it happened tells you more.
- If a coding scheme mirrors those probabilities well, expected events can be represented economically.
This is why an efficient compressor wants a good probability model.
From Guessing to Residuals
Consider a temperature sequence:
30.0, 30.1, 30.1, 30.2, 30.2, 30.3
If you store every reading independently, you repeatedly describe values close to 30. A predictor might instead guess that the next value will resemble the previous one.
Then the compressor can describe the error:
actual = predicted + residual
If the residuals are usually tiny—0, +0.1, -0.1—those residuals can often be represented more compactly than the original absolute values.
This pattern appears everywhere: predict first, encode what the prediction missed.
Images: The Pixel Beside Me Knows Something
Natural images are not random grids. Neighbouring pixels often resemble one another because physical surfaces, lighting and edges change with some continuity.
Suppose one pixel has brightness 142. A neighbouring pixel is more likely to be near 142 than to be an arbitrary value with no relationship at all. A lossless image compressor can exploit such local correlation by predicting a pixel from nearby pixels and encoding the prediction error.
Flat regions become easy. Smooth gradients become manageable. Edges create larger residuals because the world changed more sharply than the local model expected.
The compressed representation becomes, in effect, a record of where the image surprised the predictor.
Audio: The Next Sample Is Not a Stranger
Audio also has temporal structure. Adjacent samples in many signals are related. The waveform does not normally teleport from one arbitrary amplitude to another at every instant.
Predictive coding can exploit this continuity. A decoder that knows the same prediction rule can reconstruct the original from the prediction plus the encoded residual.
Again, the principle is not mystical:
model the regular part encode the irregular part
Text: Context Changes Probability
Text makes prediction especially vivid.
After the letters “th”, the next letter in English is not equally likely to be every character. After “q”, the letter “u” is common in ordinary English words. After “Singapore”, a space is more likely than many punctuation marks or letters.
Move from characters to words and the contextual structure grows richer. After “peanut butter and”, “jelly” may be likely in some corpora. After “Once upon a”, “time” becomes unusually probable.
A context model asks, in one form or another: given what I have already observed, how should I distribute probability over what may come next?
The better that distribution matches reality, the more effectively a subsequent entropy coder can convert probabilities into compact bit sequences.
Prediction and Coding Are Different Jobs
This distinction is easy to miss.
A predictive model estimates probabilities or expected values. A coding method turns those probabilities or residual statistics into bits.
For example, arithmetic coding and related entropy-coding ideas can approach the information content implied by a probability model over long sequences. But the coder is only as good as the probabilities it receives.
A brilliant entropy coder paired with a terrible model still wastes information. A strong model paired with a crude coder may also leave efficiency on the table.
Compression quality emerges from the whole pipeline.
The Model Learns What Is Normal
Prediction forces us to define normality.
If a model has seen many similar sequences, it can form expectations. Then common patterns cost little and deviations cost more.
This creates an unexpected conceptual bridge between compression and anomaly detection. A pattern that is expensive to encode under a model may also be a pattern the model finds surprising.
That does not mean every anomaly detector is a compressor, or every compressor is an anomaly detector. But both can be understood through the relationship between expected structure and residual surprise.
Better Prediction Is Not Always Free
Suppose Model A looks only at the previous symbol. Model B looks at the previous thousand symbols, consults a giant table, runs a neural network and performs a large search.
Model B may predict better. But now ask:
- How much memory does it need?
- How much time does it consume?
- Does the decoder need the same model?
- Must model parameters be transmitted?
- How robust is it when the data distribution changes?
- Does the extra compression justify the added system cost?
The compression problem has expanded. We are no longer merely minimising file size. We are designing a useful system.
When Prediction Fails
A predictor built for one distribution can perform badly on another.
A language model trained on English conventions may be poor for executable binaries. An image predictor tuned for natural photographs may not be ideal for sparse diagrams. A predictor based on slow change can be surprised by a sudden discontinuity.
This matters because a model is not reality. It is a bet about regularities in reality.
Compression exposes bad bets quickly: when predictions are systematically wrong, residuals become expensive.
Prediction Can Be Adaptive
One solution is adaptation.
Instead of fixing probabilities forever, a compressor can update its model as data arrives. If a symbol becomes common, its estimated probability rises. If a pattern disappears, the model can gradually reduce its weight.
Adaptive compression is therefore a miniature learning system: observe, update expectations, code according to the new state.
But adaptation introduces synchronisation requirements. Encoder and decoder must update compatibly, or their internal states diverge and reconstruction fails.
The Decoder Must Predict the Same Thing
This is crucial.
If the encoder predicts 142 and stores a residual of +3, the decoder must independently reproduce the prediction 142 before adding +3. If it predicts 141, the reconstructed value becomes wrong.
Predictive lossless compression therefore depends on deterministic agreement about model state, context and update rules.
A compact representation is only useful when both sides inhabit the same predictive world.
Primary School: Guess the Next Pattern
Give a child:
2, 4, 6, 8, __
They will probably answer 10. Then give:
2, 9, 4, 17, 3, __
Prediction becomes less confident.
Ask which sequence can be described more economically. The learner discovers that predictability and compressibility are related.
Secondary School: Encode the Error
Give students a sequence that mostly increases by 5:
20, 25, 30, 35, 41, 46, 51
Use “previous value + 5” as a predictor. Most residuals become zero; one becomes +1.
Now the original numbers can be reimagined as a simple rule plus a small record of deviations.
JC and Beyond: Cross-Entropy as a Compression Lens
At a more advanced level, the connection becomes quantitative. If the true data source follows one distribution but our model assigns another, the average code length associated with coding under the model reflects how well those probabilities match the source.
This is one reason cross-entropy appears so naturally in machine learning. It is not merely an arbitrary training loss. It is tied to the informational penalty paid when our predictive distribution differs from the observed outcomes.
Better probability assignments reduce expected surprise under the model. Under suitable coding schemes, that means shorter expected descriptions.
Compression Is a Test of Understanding—But Only in a Narrow Sense
There is a seductive claim that “to understand is to compress.” It contains something valuable, but we should be careful.
A model that captures real regularities can often compress observations. Newtonian mechanics compresses many motions into general laws plus initial conditions. Grammar compresses many acceptable sentences into reusable rules. Scientific theories replace lists of isolated observations with structures that generate expectations.
But compression alone is not identical to human understanding. A system can exploit statistical regularity without possessing explanation, causality, purpose or lived meaning. Compression is evidence that some regularity has been captured—not proof that every important dimension has been understood.
Students Predict Too
Reading itself is predictive.
Experienced readers anticipate grammar, genre, argument structure and likely vocabulary. Skilled mathematicians anticipate which transformation may simplify an expression. Scientists predict what a graph should roughly look like before calculating every value.
Those predictions reduce cognitive search. When reality differs, attention spikes.
Learning therefore creates better internal predictive models. Expertise often feels fast because fewer possibilities remain equally plausible.
A Useful Question for Any Compressor
Ask:
What does this compressor believe is likely?
That question reveals the model hiding inside the mechanism.
If it expects repetition, it will reward repetition. If it expects local smoothness, it will reward smoothness. If it expects familiar contexts, it will reward familiar contexts. If the world behaves differently, the bill arrives as larger residuals or longer codes.
The Deeper Point
Compression is often described as removing redundancy. Prediction gives that phrase a sharper meaning.
Redundancy is what the model did not need to be told because it could already expect it.
The compressed message concentrates attention on what the model could not safely infer.
So a useful way to think about predictive compression is:
past/context → prediction prediction + actual → surprise surprise → coded residual prediction + residual → reconstruction
The better the model, the smaller the surprise tends to become.
And that leads naturally to the next question: if the model is doing so much work, should we count the model itself as part of the message?