A compressor that remembers everything may eventually become worse at understanding what is happening now.
That sounds wrong. Compression thrives on memory. The past supplies dictionaries, probabilities, templates and predictions. Surely more history must always help.
Only if the future continues to resemble the past.
When the source changes, yesterday’s perfect model can become today’s burden. Old patterns occupy memory. Old counts distort probabilities. Old dictionary entries crowd out new ones. The compressor needs a skill that sounds almost anti-intellectual:
forget.
Quick Read
- Compression memory is useful only while past structure remains predictive.
- Non-stationary sources change their statistics over time.
- Stale models can increase code length by assigning probability to patterns that no longer occur.
- Sliding windows, decay, resets and dictionary eviction are forms of deliberate forgetting.
- Forget too slowly and adaptation lags; forget too quickly and stable structure is lost.
- Forgetting can reduce memory use, search cost and error propagation as well as improve prediction.
- The correct forgetting rate depends on how quickly the source changes.
The One-Sentence Answer
Forgetting improves compression when old information has stopped predicting the current source, because removing stale state lets the model adapt faster and spend its limited memory on patterns that still earn their cost.
A Perfect Model of Yesterday
Suppose a stream contains English prose for one million characters. The compressor learns excellent word and letter statistics.
Then the stream switches permanently to executable machine code.
The old model keeps expecting spaces, vowels and common English sequences. Those expectations are now wrong.
Every misplaced probability costs bits.
The model must unlearn.
Concept Drift: The Source Moves
Machine learning often calls this kind of change concept drift or distribution shift. The statistical relationship governing observations changes over time.
Compression experiences the same phenomenon directly because code length responds to predictive error.
A model trained on stale behaviour produces longer descriptions. Compression ratio becomes a sensor for drift.
Sliding Windows Forget by Distance
A sliding-window model keeps only the most recent N observations.
As new data enters, old data exits automatically.
This gives recent evidence more influence because ancient observations are no longer counted.
The window size becomes a forgetting rate. Small windows adapt quickly but estimate probabilities noisily. Large windows are stable but slow to react.
Exponential Decay Forgets Gradually
Instead of dropping old observations suddenly, a model can reduce their weight over time.
Recent evidence counts strongly. Older evidence fades.
This creates a soft memory horizon rather than a hard window boundary.
The decay rate again expresses a belief about the world: how quickly do yesterday’s statistics become irrelevant?
Dictionaries Need Eviction
A finite dictionary cannot keep every phrase ever encountered.
When full, it needs an eviction policy. Remove entries not used recently? Remove those producing little saving? Keep rare but long phrases? Reserve space for new candidates?
This is memory economics. Every dictionary entry occupies capacity that another pattern could use.
Forgetting creates room for learning.
Caches Learned This Long Ago
Computer caches face the same problem. Capacity is finite, so old items are evicted according to policies intended to keep likely future hits.
A cache that never forgets fills permanently. A cache that forgets randomly wastes useful state.
Good forgetting is prediction about future reuse.
Resetting Can Be Better Than Slowly Recovering
Sometimes a regime change is obvious: a new file begins, a new language is detected, a new sensor mode activates.
Instead of slowly diluting the old model, the compressor can reset state immediately.
A reset sacrifices useful history if the detected change was false. But when the boundary is real, it prevents a long period of regret.
Boundaries and forgetting are therefore connected.
Forgetting Helps Error Recovery
Suppose decoder state depends on the entire history since the beginning of a stream. One corrupted symbol may affect predictions forever.
Periodic resets limit that dependency. After a reset, old corruption no longer influences the state.
Forgetting has become resilience.
Forgetting Helps Random Access
If a block depends on unlimited prior context, decoding it independently is impossible.
Bounded context means a decoder needs only a limited history—or a restart snapshot—to begin.
Again, sacrificing some long-range compression buys locality.
The Hard Part Is Knowing What to Forget
Old information is not automatically useless.
A motif may disappear for an hour and return. A seasonal pattern may recur every year. A character in a novel may vanish for 200 pages and matter again at the ending.
Simple recency is not always enough.
An intelligent memory policy tries to estimate future utility, not merely age.
Hierarchical Memory Can Forget at Different Speeds
Some structure changes quickly; other structure is stable.
In language, the immediate topic may change every paragraph while grammar changes slowly. In traffic, second-to-second flow fluctuates while weekday commuting patterns remain stable.
A multiscale model can maintain fast-changing local state and slower long-term state simultaneously.
Forgetting need not have one clock.
Human Memory Is Selectively Forgetful
People forget details while retaining gist, categories and skills.
This can be frustrating, but a mind that retained every sensory detail with equal priority would face an enormous retrieval problem.
Forgetting can be understood partly as prioritisation: preserve structures likely to matter, allow low-value detail to decay.
Human memory is not a compression algorithm, but the analogy is useful because both systems operate under finite resources.
Education: Old Wrong Models Must Sometimes Be Removed
A student learns that “a heavier object falls faster.” Later physics instruction corrects the misconception.
Simply adding the correct rule beside the old one may not be enough. Under pressure, the older intuitive model can still fire first.
Learning sometimes requires active replacement, discrimination and inhibition—not endless accumulation.
Good knowledge systems need deletion policies too.
Primary School: Which Clues Still Matter?
Play a guessing game where the hidden rule changes halfway through. Pupils must notice that old evidence no longer predicts new answers.
Ask when they should stop trusting the first rule.
Secondary School: Sliding Average
Give students a time series whose mean changes halfway through. Compare an average over all past observations with an average over only the most recent ten.
The full-history estimate is stable but slow. The short-window estimate adapts quickly but fluctuates more.
They can see the stability–adaptation trade-off.
JC and Beyond: Non-Stationary Coding
At higher levels, universal and adaptive coding for non-stationary sources considers models that switch, mixtures that track changing experts, context-tree weighting variants and other strategies designed to keep regret controlled when source statistics evolve.
The mathematics becomes sophisticated, but the core problem remains recognisable: old evidence is useful until it isn’t.
A Forgetting Checklist
- How quickly can the source change?
- How much old data still predicts the present?
- Should forgetting use a hard window or gradual decay?
- Which dictionary entries have stopped earning their space?
- Can a regime change be detected explicitly?
- Would a reset improve recovery?
- Which long-term patterns must survive short-term change?
- How much regret is caused by adapting too slowly versus too quickly?
The Deeper Point
Compression loves memory because memory turns repetition into references and experience into prediction.
But memory has value only while it remains connected to the future.
When the world changes, yesterday’s knowledge can become today’s noise.
Sometimes making room for the next pattern begins by letting the last pattern go.