If the past can compress the future, how much of the past should we keep?
Imagine a book repeats one sentence every ten pages. A compressor remembering only the previous paragraph will miss the pattern. Give it a larger memory and the repetition becomes visible.
Now imagine the repeated sentence returns after ten million pages.
Should the compressor remember everything forever, just in case?
Memory creates compression opportunities, but memory itself has a cost. That tension sits underneath dictionaries, sliding windows, context models, caches and adaptive predictors.
Quick Read
- Past data can act as a dictionary or predictive context for future data.
- Larger memory can expose longer-range repetition and dependence.
- Memory consumes RAM, search time, energy and decoder state.
- Sliding-window compressors deliberately forget old material.
- Finite-state models summarise history rather than storing it literally.
- The best context length depends on how far useful dependencies extend.
- Remembering irrelevant history can hurt as well as help.
The One-Sentence Answer
A compressor should remember enough history to capture useful recurring structure, but not so much that storing, searching and synchronising that history costs more than the compression it earns.
The Past Is a Dictionary
Dictionary compressors exploit an elegant fact: if a sequence has already appeared, we may refer back to it instead of transmitting it again.
The already-decoded past therefore becomes shared side information.
past sequence + pointer → repeated sequence
But pointers only help if the referenced material remains inside the accessible dictionary or window.
Sliding Windows Are Controlled Forgetting
Many practical schemes keep only a bounded recent history. As new bytes arrive, old bytes fall out of the window.
This looks wasteful. What if something old repeats?
It may indeed be missed. But bounded memory makes implementation practical. Pointers stay limited. Search structures remain manageable. Decoders know exactly how much history must be retained.
Forgetting is not necessarily failure. It is resource allocation.
A Bigger Window Is Not Free
Increase the window and several things happen.
- More distant matches become possible.
- More memory is required.
- Search can become more expensive.
- Pointers may need more bits.
- Cache behaviour can worsen.
- Decoder state becomes larger.
A match from far away must save enough bytes to pay for the longer reference and the machinery required to find it.
Context Models Remember Differently
A predictive compressor does not always store the past as literal bytes. It may summarise history into counts, probabilities or model state.
For example, instead of remembering every previous occurrence of “th”, it may remember how often each letter followed “th”.
This is compressed memory: the system stores what the past taught it rather than the entire past itself.
Finite State: Many Histories, One Relevant State
Suppose future probabilities depend only on whether a machine is ON or OFF. The exact sequence of button presses that led there may not matter.
Many different histories collapse into the same current state.
A finite-state model exploits this by remembering a sufficient summary rather than a complete diary.
This is another form of compression inside the compressor: compress the past into the state needed to predict the future.
Markov Thinking: How Far Back Matters?
A first-order Markov model assumes the next state depends on the current state rather than the entire history. Higher-order models keep more context.
If the source truly has short memory, a compact model works well. If dependencies stretch far back, short contexts miss information.
The question becomes empirical: how much additional predictive power do we gain by remembering another step?
Language Has Long Memory
The next letter depends strongly on nearby letters. But the next word can also depend on a topic introduced paragraphs earlier. A pronoun may refer to a character named pages ago. A conclusion may echo an opening claim.
Short context captures spelling and syntax. Longer context captures discourse and topic.
Yet longer context also contains enormous irrelevant detail. The challenge is not merely remembering more. It is remembering what remains predictive.
Images Have Spatial Memory
For images, “past” can mean neighbouring pixels, previous rows, earlier tiles or repeated regions elsewhere in the picture.
A tiny neighbourhood captures local smoothness. A larger search can discover repeated textures or objects.
Again, the memory horizon determines which regularities are visible.
Memory and Locality Pull Against Each Other
Long dependency chains can improve compression but make random access harder. To decode one region, the receiver may need a large preceding history.
Blocks and restart points deliberately cut memory. They sacrifice some cross-boundary compression to regain locality and resilience.
This is why memory cannot be optimised independently of access.
Memory and Errors
If future decoding depends on a long remembered history, one corrupted symbol can contaminate later state.
Resetting state periodically limits the blast radius.
So reliability gives us another reason to forget.
What Should Be Forgotten?
The ideal compressor would preserve information proportional to its future usefulness.
Recent context may be highly predictive. Old context may become irrelevant. Some rare structures remain useful for a long time. Others should decay quickly after a regime change.
Adaptive systems therefore face a memory-policy question: what deserves retention?
Distribution Shift Turns Memory Into Baggage
A compressor trained on English prose switches to source code. Old word statistics can mislead prediction.
Forgetting now becomes essential. A model that remembers everything equally may adapt too slowly.
Useful memory is not simply large memory. It is relevant memory.
Human Expertise Is Selective Compression of the Past
An experienced teacher does not remember every sentence spoken in every lesson. They remember patterns: which misconception recurs, which explanation works, which question reveals understanding.
Experience becomes useful because individual events are compressed into reusable state.
Memory that retained every detail equally would be overwhelming.
Education: Working Memory Is a Hard Budget
A learner cannot keep an unlimited number of unfamiliar elements active at once. Good instruction therefore builds chunks and external supports so important relationships survive without consuming excessive working memory.
Expertise changes the effective memory budget because familiar structures become compressed units.
The learner is not gaining infinite memory. They are using representation more efficiently.
Primary School: How Far Back Does the Pattern Reach?
Give pupils a sequence where the next symbol depends on the previous one, then another where it depends on the previous three. Ask how much history they need to predict correctly.
They discover context length directly.
Secondary School: Window Size Experiment
Take a long repeated text and search for matching phrases using only the previous 20 characters, then 100, then 1,000.
Record how many matches become available and how search effort changes.
JC and Beyond: Sufficient Statistics
At higher levels, statistics gives us a beautiful formal idea: a sufficient statistic can preserve all the information in the sample relevant to estimating a parameter under a model.
That is not identical to compression memory, but the intellectual connection is strong: replace a large history with a smaller state that preserves what matters for the future inference task.
A Memory Checklist
- How far back do useful dependencies extend?
- Can history be summarised rather than stored literally?
- How much RAM does the context require?
- How expensive is searching the larger memory?
- How many bits do longer references require?
- Does old history remain relevant after distribution shift?
- How does memory affect random access and error propagation?
- Where should the system deliberately reset or forget?
The Deeper Point
Compression uses the past as a prediction machine.
But the past is infinite in principle and memory is finite in practice.
So every compressor eventually answers a philosophical-sounding engineering question:
Which parts of history still deserve to influence the future?