Yesterday, the API responses compressed beautifully.
Today, nothing is broken.
The decoder still works.
The messages are simply getting larger.
The world changed before the compressor admitted it.
Corpus drift is what happens when the family that justified a shared compression model changes enough that the old model no longer predicts the new members efficiently.
This is the fourth pillar beneath How Compression Works | The Corpus. The master owns the idea that families expose shared structure. Corpus Drift owns the moment that family resemblance weakens, moves or splits.
Quick Read
Corpus drift occurs when the statistical or structural distribution of incoming data changes relative to the corpus that trained a dictionary, predictor, chunking policy or other shared model. The system may remain perfectly decodable while compression ratio, latency or hit rate degrades. Drift can come from new templates, languages, software versions, customer populations, sensors, formats or behaviour. Good systems monitor residual size and subgroup performance, distinguish temporary anomalies from persistent change, and introduce versioned replacement models without destroying the ability to decode historical data.
old corpus → shared model → new data → rising residual / falling reuse → diagnose drift → segment or retrain → versioned successor → monitor again
Compression Is a Prediction About the Future
Every specialised compressor carries an assumption:
future messages will resemble the past enough that yesterday’s learned structure remains useful.
When that assumption weakens, code lengths grow.
The failure can be gradual and operationally quiet.
Residual Growth Is the First Symptom
A good model predicts common structure cheaply.
What it fails to predict appears in the residual.
If average compressed bytes per message rise while raw message size stays similar, the model may be paying more for surprise.
Residual size becomes an operational sensor for model fit.
A Template Change Can Cause Instant Drift
Version 1 of an application emits twelve familiar JSON fields.
Version 2 renames half of them and adds nested metadata.
The old dictionary still decodes old content perfectly.
It simply stops helping as much with the new schema.
Language Drift Can Be Slow
Vocabulary changes.
Product names change.
New abbreviations appear.
Old phrases disappear.
A dictionary trained years ago may remain useful while gradually becoming less optimal.
User-Population Drift Changes the Corpus Without Changing the Software
The same application enters a new country.
Users write in different languages.
Usage shifts from desktop to mobile.
A new customer segment generates different payloads.
The code can remain identical while the source distribution changes underneath it.
Sensor Drift Can Look Like Corpus Drift
A sensor is replaced.
Resolution changes.
Noise characteristics change.
The compressor sees a new statistical family.
Before retraining, ask whether the world changed or the observation instrument changed.
Format Drift Can Be Artificial
Pretty-print JSON instead of minifying it.
Change whitespace.
Reorder fields.
The semantic content may be similar while byte representation shifts.
Canonical Form owns the deeper question of whether irrelevant representational differences can be normalised before compression.
Not Every Bad Day Is Drift
A one-off bulk export appears.
Compression ratio collapses for an hour.
Then normal traffic returns.
That is an anomaly, not necessarily a new family state.
Drift requires persistence, recurrence or a structural explanation strong enough to justify model change.
Monitor Distribution, Not Only Global Average
Average compression ratio looks stable.
But one customer segment has degraded badly while another improved.
A global average can hide subgroup drift.
Monitor by relevant family partitions such as:
- endpoint;
- language;
- software version;
- region;
- message type;
- customer class;
- sensor generation.
Deduplication Has Drift Too
Yesterday, backups shared many chunks.
Today, an application starts rewriting entire database files in a new layout.
Logical information may be similar while chunk identity hit rate falls.
Deduplication owns exact reuse. Corpus Drift asks why exact reuse stopped happening at the old rate.
Chunk-Boundary Drift Can Be Self-Inflicted
Change chunking parameters.
Now identical corpus structure may be divided differently.
Reuse falls even if source content did not change much.
The model changed, not necessarily the world.
Delta Encoding Has Drift Too
Successive versions used to be close.
A new build process begins regenerating files in a completely different order.
Delta sizes rise.
Delta Encoding owns relational difference. Drift explains why old base-selection assumptions can become poor.
Dictionary Drift Is the Clearest Example
Shared Dictionaries learn recurring structures from representative samples.
When future messages stop resembling those samples, dictionary references become less useful and more bytes must be encoded from scratch.
The First Repair Is Not Always Retraining
Before building a new model, ask:
- Did the data format accidentally change?
- Did instrumentation change?
- Did a small subgroup suddenly dominate traffic?
- Did preprocessing break?
- Did the old dictionary stop being loaded correctly?
An operational regression can imitate statistical drift.
Retraining Has a Switching Cost
Dictionary v2 compresses new messages better.
But:
- encoders need it;
- decoders need it;
- caches need it;
- historic content may still require v1;
- monitoring must distinguish versions.
The retraining decision should compare future savings against deployment and compatibility cost.
Retraining Too Often Can Be Worse Than Mild Drift
A new dictionary every hour might track traffic perfectly.
It might also create:
- version explosion;
- cache churn;
- distribution overhead;
- harder debugging;
- historic compatibility burden.
Compression is a system objective, not a leaderboard for the smallest latest sample.
Segmentation Can Beat One Constantly Moving Model
Suppose English and Japanese payloads increasingly diverge.
Instead of retraining one global dictionary repeatedly, maintain stable family-specific dictionaries.
Corpus drift can reveal that the original corpus boundary was too broad.
A Family Can Split
One product becomes two.
One message schema branches.
One customer population develops distinct behaviours.
The right response may be two models rather than one updated model.
A Family Can Merge Too
Two formerly separate APIs converge on one schema.
Maintaining two dictionaries may now waste operational effort.
Drift monitoring can reveal increasing cross-family similarity as well as divergence.
Forgetting Can Be Necessary
A model trained forever on every historical regime may dilute current structure.
Forgetting owns the broader problem of deciding when old patterns should stop influencing the present.
Corpus Drift supplies one reason: historical data can become a bad prior for current messages.
Memory Can Be Too Long
A streaming compressor adapts from recent history.
If memory is too long, it reacts slowly to regime change.
If memory is too short, it forgets useful stable patterns.
Memory owns that horizon trade-off.
Streaming Makes Drift Visible Faster
Batch systems may discover performance loss at the end of a reporting period.
Streaming systems can observe compression cost continuously.
That creates an opportunity for faster detection—but also a temptation to overreact to noise.
Regret Measures the Price of Adapting Too Slowly
If an ideal current model would use 100 MB but the stale model uses 130 MB, the extra 30 MB is part of the cost of model mismatch.
Regret owns the general cost of choosing the wrong model too early or keeping it too long.
Drift Detection Needs a Baseline
Compression ratio varies naturally with content.
So define expected ranges by family and workload.
Track:
- raw bytes per object;
- compressed bytes per object;
- dictionary hit behaviour;
- dedupe hit rate;
- delta size;
- latency;
- fallback frequency.
The model should be judged against the same kind of traffic it is supposed to serve.
Currentness Should Be Explicit
Which corpus trained the model?
From which dates?
Which products and languages?
What changed since then?
Compression artefacts need provenance just as datasets do.
Rollback Must Remain Possible
Dictionary v3 is deployed.
Latency spikes.
Compression improves but CPU cost is unacceptable.
A safe transition can return to v2 without making old and new content ambiguous.
The Decoder Contract Limits How Fast You Can Adapt
Encoder-side optimisation is easy if only the encoder cares.
Shared dictionaries require receiver readiness.
The best new model cannot be used until the decoding ecosystem can identify and obtain it reliably.
Compression Drift Is a General Model-Limit Lesson
A model succeeds inside an operating envelope.
When the source distribution leaves that envelope, old performance guarantees weaken.
The mature response is neither panic nor denial.
It is detection, diagnosis, controlled adaptation and verification.
A Better Corpus-Drift Model
versioned corpus → model → live payloads → per-family compression telemetry → change detection → source-vs-pipeline diagnosis → retrain / segment / retain → versioned rollout → compatibility + rollback → continued monitoring
A 30-Lens Corpus Drift Audit
- Model: what shared compression state is drifting?
- Training corpus: what family created it?
- Date: when was that corpus sampled?
- Current traffic: what family is arriving now?
- Raw size: did source object size change?
- Compressed size: did residual cost change?
- Ratio: how much efficiency was lost?
- Subgroup: where is degradation concentrated?
- Template: did schema or format change?
- Language: did vocabulary distribution change?
- User population: did the audience change?
- Region: did geographic mix change?
- Sensor: did observation hardware change?
- Preprocessing: did canonicalisation or ordering change?
- Chunker: did boundaries or parameters change?
- Dictionary loading: is the correct shared state actually used?
- Anomaly: temporary spike or persistent regime?
- History: have similar shifts happened before?
- Hit rate: did dedupe reuse fall?
- Delta: did base similarity fall?
- Latency: did computational cost change?
- Segmentation: should the corpus split?
- Merge: have formerly separate families converged?
- Retraining: what benefit would a new model deliver?
- Switching cost: what deployment burden follows?
- Version: will historic decode state remain stable?
- Fallback: what if the new model is unavailable?
- Rollback: can transition be reversed?
- Threshold: what performance change justifies action?
- Return: did the new model improve held-out live data after deployment?
Laboratory 1: Template Shock
Create twenty messages under template A, then ten under a substantially different template B. Track which shared phrases stop being useful and when you would introduce a new dictionary.
Laboratory 2: One Average Hides Two Families
Construct subgroup data where one family improves and another deteriorates while the global average stays unchanged. Explain why aggregate monitoring misses drift.
Laboratory 3: World or Pipeline?
Compression suddenly worsens. Create two diagnostic branches: the source distribution genuinely changed, or preprocessing accidentally began reordering fields. List tests that distinguish them.
For Primary Readers
You make abbreviations for words your class uses every day. Next term the class studies a new subject. Some old abbreviations stop saving much time. The shared vocabulary needs updating.
For Secondary Readers
Explain why a model can remain correct enough to decode while becoming less efficient because the data family changed.
For Advanced Readers
Model corpus drift as non-stationarity in the empirical source distribution. Adaptation policy balances excess codelength against retraining, model distribution, decoder compatibility, segmentation and temporal stability.
Common Misconceptions
- “If decompression still works, the model is still good.” Correct reconstruction and efficient compression are different properties.
- “Any bad compression period means retrain immediately.” Temporary anomalies and pipeline failures can imitate drift.
- “One average compression ratio is enough.” Subgroup degradation can hide inside a stable global mean.
- “Retraining always fixes drift.” The real problem may be changed preprocessing, chunking or instrumentation.
- “Replace the old dictionary with the new bytes.” Historic compressed data may require immutable old versions.
Research Corridor
- RFC 8878 — Zstandard Compression — dictionary identity and the need for correct decoder-side shared state.
- Zstandard Manual — dictionary and streaming APIs.
- eduKateSG — Forgetting.
- eduKateSG — Regret.
Frequently Asked Questions
What is corpus drift in compression?
It is a change in the incoming data family that makes a previously useful shared compression model less well matched to current data.
How can you detect corpus drift?
Track compression ratio, residual size, dedupe hit rate, delta size, latency and subgroup performance against stable baselines, then investigate persistent changes.
Does drift mean old data become undecodable?
Not necessarily. Drift usually means the model is less efficient for new data. Historic decoding remains possible if the old shared state and version identities are preserved.
Final Thought: A Family Is a Moving Target
The corpus made the model smart.
The next corpus can make that same model old.
Corpus compression stays intelligent only when it remembers that the family resemblance it learned was an observation about a particular world state, not a permanent law of the source.
THE CORPUS · FOUR PILLAR LEGS
Return to The Corpus, or continue through Deduplication, Delta Encoding and Shared Dictionaries. Return to the Information & Representation Hub.