A thousand short messages each contain phrases that appear only once inside each message.
Compress them independently and those phrases barely look repetitive.
Train on the family and the repetition becomes obvious.
A shared dictionary moves recurring structure out of each small message and into a reusable model that encoder and decoder both know.
This is the third pillar beneath How Compression Works | The Corpus. The master owns cross-document learning. This article owns the shared vocabulary itself: how it is trained, identified, distributed, versioned and eventually replaced.
Quick Read
A shared compression dictionary is reusable side information derived from representative samples of a data family. It contains recurring content and/or statistical tables that let small related payloads reference patterns that may not repeat enough inside any one payload to learn efficiently on the fly. The decoder must possess the correct dictionary, so dictionary identity and compatibility become protocol concerns. Specialised dictionaries can improve compression inside their domain but can become stale when the corpus changes.
representative corpus → train dictionary → assign identity/version → distribute to encoder + decoder → compress family members → monitor residuals → retrain or retire
Why Small Files Benefit So Much
A long file gives a compressor time to learn patterns from earlier bytes.
A tiny file may end before enough repetition appears.
But if thousands of tiny files belong to one family, a corpus-trained dictionary can provide the learning upfront.
The setup cost is paid once and amortised across many messages.
Amortisation owns the broader economic logic.
The Dictionary Is Side Information
The compressed payload can be smaller because some descriptive burden has moved elsewhere.
That means total system cost includes:
- dictionary bytes;
- distribution;
- storage;
- version management;
- payload bytes;
- decoder compatibility.
A 100 KB dictionary is not “free” because it is not inside each 2 KB response.
It becomes cheap only when enough payloads reuse it.
Training Data Defines What the Dictionary Knows
Train on English support tickets.
The dictionary learns different recurring structure from one trained on JSON telemetry, legal contracts or medical codes.
The corpus therefore acts like a prior over future messages.
Representative Beats Merely Large
A million samples from one narrow subcategory can produce a worse general dictionary than a smaller but representative training set.
Volume cannot compensate for a badly drawn corpus boundary.
Repeated Templates Are Excellent Dictionary Material
API responses often repeat:
- field names;
- status phrases;
- schema fragments;
- HTML boilerplate;
- protocol headers;
- domain terminology.
Each payload may contain a particular field name only once.
Across millions of responses, it is overwhelmingly common.
Zstandard Makes Dictionary Identity Explicit
RFC 8878 defines an optional Dictionary_ID field and a dictionary format for Zstandard. A decoder using dictionary-compressed content must use the correct dictionary; the specification makes dictionary identity part of the decoding contract.
This turns a compression optimisation into a compatibility problem:
small payload + wrong dictionary = not a slightly worse decode; it is the wrong decoding state.
Dictionary IDs Are Pointers, Not Dictionaries
The identifier tells the decoder which shared object is required.
The dictionary itself must still be provisioned somehow.
RFC 8878 explicitly notes that dictionaries are provided out of band rather than embedded automatically in the compressed content.
The smaller payload therefore depends on infrastructure outside itself.
Provisioning Is Part of the Protocol
Before using dictionary 42, both endpoints need to agree that dictionary 42 means the same bytes.
Systems can achieve this through:
- bundling dictionaries with software;
- negotiating supported versions;
- fetching from trusted storage;
- content-addressing dictionary objects;
- shipping them during setup.
The right mechanism depends on application and trust boundary.
A Dictionary Should Be Immutable Once Its Identity Is Published
If “dictionary 42” silently changes bytes, old payloads become ambiguous or undecodable.
A safer model is:
new dictionary bytes → new version/identity.
Versioned immutability makes historic compressed content reconstructable.
The Decoder’s Cache Becomes Valuable Shared State
Load a dictionary once.
Decode ten thousand messages with it.
Now setup and parsing cost can be amortised too, not only dictionary bytes.
Bulk-processing APIs in compression libraries exist partly because reusing prepared dictionary state can improve throughput.
Specialisation Improves Prediction
A legal dictionary knows recurring legal byte sequences.
A telemetry dictionary knows telemetry.
As the family becomes narrower, predictable structure can increase.
But extreme specialisation creates fragility outside the target family.
One Global Dictionary Can Become Mediocre at Everything
Combine invoices, source code, sensor logs, novels and encrypted blobs into one training set.
The dictionary has finite capacity.
Patterns from unrelated domains compete for space.
Corpus partitioning can therefore outperform one universal shared dictionary.
Too Many Dictionaries Create Routing Overhead
One per language.
One per endpoint.
One per customer.
One per month.
Compression improves, but now the system must select, distribute, cache and retire many dictionary versions.
The optimal partition balances modelling gain against operational complexity.
Dictionary Selection Can Be a Classification Problem
Before compression, identify which family the payload belongs to.
Then choose the corresponding dictionary.
Misclassification can reduce compression or create compatibility failure if the decoder receives the wrong identity.
Fallback Paths Matter
The receiver does not have the preferred dictionary.
What now?
- send uncompressed;
- use ordinary compression without the dictionary;
- negotiate an older common dictionary;
- deliver the dictionary first.
A robust protocol defines the failure path before it needs it.
The Dictionary Is a High-Leverage Dependency
One small shared object can support millions of payloads.
Corrupt or remove it and the blast radius can be enormous.
The Corpus master makes the principle explicit: the most reused bit may be the most valuable bit.
Trust Matters When Dictionaries Are Distributed
A decoder should not blindly accept arbitrary shared state from an untrusted source simply because it improves compression.
Dictionary provenance, integrity and authorised distribution belong to the security design.
A Dictionary Can Leak Corpus Structure
A trained dictionary may contain fragments characteristic of its training family.
If the training corpus is sensitive, dictionary distribution can itself require privacy review.
Never assume “model” means “contains no source information.”
Security and Compression Have an Old Tension
RFC 8878 repeats a general compression-security warning: secrets should not be compressed together with attacker-influenced content in ways that allow compressed length to reveal information.
Shared dictionaries do not remove this concern.
Compression state can become part of a side channel when system boundaries are poorly designed.
Dictionary Performance Should Be Measured on Held-Out Data
Evaluate only on the samples used to train the dictionary and performance can look artificially excellent.
Hold out representative future-like data.
Measure:
- compressed size;
- latency;
- memory;
- coverage across subgroups;
- fallback frequency.
A dictionary is useful when it generalises to new members of the family.
Residual Size Is a Drift Sensor
Yesterday, messages compressed very well.
Today, residual size rises steadily.
Maybe the corpus changed.
The fourth pillar, Corpus Drift, owns that failure mode.
Retraining Is a Version Transition, Not an In-Place Edit
New training data produce dictionary v2.
Old payloads may still require v1.
The system now has two legitimate historical decoding states.
Retirement policy must consider how long old compressed objects need to remain readable.
Shared Dictionaries and Deduplication Exploit Different Repetition
Deduplication asks whether a chunk is exactly identical to a stored chunk.
A shared dictionary asks whether new data contain recurring shorter structures and statistical regularities learned from the family.
One reuses objects.
The other reuses a predictive vocabulary.
Shared Dictionaries and Delta Encoding Exploit Different Relationships Too
Delta Encoding chooses a particular related base object.
A dictionary is shared across many objects and need not correspond to one complete ancestor.
Education Works the Same Way
A teacher says “photosynthesis.”
For an expert learner, one word activates a large shared conceptual dictionary.
For a beginner, the word is only a label.
The message becomes compact only because the receiver possesses the shared model.
A Better Shared-Dictionary Model
define family → sample representative corpus → train immutable dictionary → assign version/identity → trusted provisioning → negotiate/select → compress/decompress → measure held-out performance → detect drift → publish successor while preserving old decode state
A 30-Lens Shared Dictionary Audit
- Family: which payloads share structure?
- Boundary: what is excluded?
- Training set: how was it sampled?
- Representativeness: do major subgroups appear?
- Size: how large is the dictionary?
- Capacity: which patterns win space?
- Overfit: does training performance exceed held-out performance sharply?
- Identity: how is the dictionary named?
- Version: are bytes immutable per version?
- Provisioning: how does the decoder obtain it?
- Trust: who is authorised to publish it?
- Integrity: how are bytes verified?
- Negotiation: how do endpoints agree?
- Fallback: what happens when it is unavailable?
- Cache: can prepared state be reused?
- Memory: what decoder cost is added?
- Latency: does setup dominate small payloads?
- Amortisation: how many messages reuse it?
- Compression gain: how much does it actually save?
- Subgroup fairness: which family members compress worse?
- Routing: how is dictionary choice made?
- Misclassification: what if the wrong family is chosen?
- Privacy: could dictionary content reveal training data?
- Security: can shared compression state create a side channel?
- Currentness: is the family changing?
- Residual: is compressed size trending upward?
- Retraining: what triggers a new version?
- Retention: how long must old versions survive?
- Compatibility: can historic payloads still decode?
- Return: does the dictionary improve new real payloads, not only training samples?
Laboratory 1: Build a Phrase Dictionary
Collect fifty short messages from one fictional system. Find recurring phrases and build a tiny shared dictionary. Compare total size with compressing each message as though no shared vocabulary existed.
Laboratory 2: Wrong Family
Apply a dictionary trained on weather reports to recipes. Discuss why specialised shared state can fail outside its corpus.
Laboratory 3: Version Transition
Create dictionary v1, then change the message template substantially. Design a migration where old payloads remain readable while new payloads use v2.
For Primary Readers
Agree that one star symbol means “Please remember to bring your book tomorrow.” Once everyone knows the shared dictionary, one mark can carry a long message.
For Secondary Readers
Explain why a dictionary trained across many short related messages can see repetition that no one message contains twice.
For Advanced Readers
Model shared-dictionary compression as amortised side information learned from an empirical source distribution. System design must account for dictionary entropy model, distribution shift, decoder provisioning and versioned compatibility.
Common Misconceptions
- “The dictionary is free because it is not inside every payload.” Its cost is amortised, not nonexistent.
- “More training data always makes a better dictionary.” Representativeness and family coherence matter.
- “A dictionary ID contains the dictionary.” It identifies required shared state; the bytes must be provisioned separately.
- “Updating a dictionary in place is harmless.” Historic payloads need stable decoding state.
- “One global dictionary is simplest and therefore best.” It may dilute specialised structure and compress poorly across heterogeneous domains.
Research Corridor
- RFC 8878 — Zstandard Compression — dictionary IDs, dictionary format and out-of-band dictionary provisioning.
- Zstandard Manual — dictionary APIs and reusable dictionary processing.
- eduKateSG — Amortisation.
- eduKateSG — Side Information.
Frequently Asked Questions
What is a compression dictionary?
It is shared data used by a compressor and decompressor to represent recurring patterns or statistical structure more efficiently than learning everything from each payload independently.
Why do both sides need the same dictionary?
Because compressed references are interpreted relative to that shared state. Using the wrong dictionary does not preserve the intended reconstruction.
When should a dictionary be retrained?
When representative held-out performance degrades enough to justify the operational cost of introducing and supporting a new version.
Final Thought: The Short Message Has a Long Memory Behind It
The payload looks tiny.
The shared vocabulary that makes it tiny may have been learned from millions of earlier messages.
A shared dictionary compresses a family by turning yesterday’s repeated structure into today’s common language.
THE CORPUS · FOUR PILLAR LEGS
Return to The Corpus, or continue through Deduplication, Delta Encoding and Corpus Drift. Return to the Information & Representation Hub.