How Compression Works | The Corpus — Why One File Can Become Smaller When It Belongs to a Family

A file can look unique when you examine it alone and repetitive when you place it beside its family.

Imagine one school report. It contains a logo, headings, table structure, standard comments, dates and student-specific details. Compress it alone and you may find some repetition.

Now collect 100,000 reports produced from the same system.

Suddenly most of each document is not unique at all. The template repeats. Phrases repeat. Fonts repeat. Layout rules repeat. Only a relatively small set of fields changes from student to student.

The right compression object was never one report. It was the corpus.

Quick Read

• Redundancy can exist across files, not only inside them.
• A corpus exposes shared templates, vocabulary, headers, structures and repeated assets.
• Cross-document dictionaries can exploit patterns no isolated-file compressor can see.
• Deduplication stores repeated chunks once and references them from many objects.
• Version histories are especially compressible when most content stays unchanged.
• Corpus-level compression trades independence for shared infrastructure and dependency.
• The best unit of compression may therefore be a collection, repository, dataset or stream rather than an individual file.

The One-Sentence Answer

A file can become cheaper to describe when it belongs to a family because the compressor can represent shared structure once at corpus level and spend per-file bits mainly on what makes each member different.

The Family Resemblance Problem

Take 1,000 invoices from one company.

Each invoice may contain:

• the same logo;
• the same legal footer;
• the same table headings;
• the same payment instructions;
• the same typography;
• similar product descriptions;
• different customer names, quantities and totals.

If every invoice is compressed independently, each compressor must rediscover much of the same structure.

A corpus-aware system can learn the template once.

document = family template + document-specific residual

Cross-Document Dictionaries

A dictionary built from one file knows only that file’s repeated sequences. A dictionary trained across thousands of related files can contain recurring phrases and byte patterns that appear once per document but thousands of times across the collection.

This is important. A phrase occurring once in each file is invisible as within-file repetition. Across the corpus, it is extremely repetitive.

The boundary of observation determines the redundancy we can see.

Deduplication: Store the Chunk Once

Suppose 10,000 virtual machines contain the same operating-system libraries. Storing a complete independent copy for every machine wastes enormous space.

Deduplication divides data into chunks, identifies identical chunks and stores one physical copy with references from all users.

This is not ordinary entropy coding. It is compression through shared identity.

The system says: “We already own these bytes. Record another reference, not another copy.”

Version Histories Are Families Across Time

A document edited every day may produce 365 versions. If each version is stored independently, repeated material dominates.

Version-control systems exploit relationships among versions by storing objects, deltas or shared content so unchanged material need not be paid for repeatedly.

The corpus is temporal: yesterday, today and tomorrow belong to one evolving family.

Software Repositories Make This Obvious

A software repository may contain thousands of revisions where only a small fraction changes each time.

Source files share syntax, identifiers and libraries. Commits share history. Branches share ancestors.

Treating every snapshot as an unrelated archive would ignore the strongest structure in the dataset: common ancestry.

Compression becomes graph-aware. Relationships among objects matter.

Websites Are Repetition Machines

Visit 1,000 pages on one website. Navigation, headers, footers, CSS classes, scripts, icons and recurring phrases may appear everywhere.

Compress each response in isolation and much of that repetition is rediscovered repeatedly. Shared dictionaries, caching and common resources move reusable structure outside each individual payload.

The web page becomes a thin layer of difference sitting on top of shared infrastructure.

The Corpus Can Teach the Model

Large collections reveal statistical regularities unavailable in one sample.

A single legal document may contain a phrase once. A million legal documents reveal that the phrase is conventional. A single medical note may contain an abbreviation once. A large corpus reveals its domain frequency and context.

Corpus-trained models can therefore make stronger predictions on future members of the same family.

This is learned side information amortised across a population.

But Families Change

A corpus model assumes future objects resemble the corpus that built it.

Change the template, language, product, sensor or user population and the old dictionary may become less useful. A highly specialised corpus model can perform brilliantly inside its family and poorly outside it.

This is the price of specialisation.

Corpus Compression Creates Dependencies

If one file depends on a shared dictionary, that file is no longer fully self-contained. If deduplicated chunks live in a central store, losing the store can affect many logical files. If a base version is corrupted, dependent deltas may become unusable.

Shared structure creates shared failure modes.

Good systems therefore protect dictionaries, chunk stores, indexes and base objects carefully. The more reuse a component provides, the larger its blast radius when it fails.

The Most Reused Bit May Be the Most Valuable Bit

A byte appearing once has one local job. A shared dictionary phrase referenced by ten million documents has enormous systemic leverage.

This suggests a new way to value information: not only by its size, but by how many other descriptions depend on it.

Compression creates hubs.

And hubs deserve stronger protection.

Corpus Boundaries Matter

Which files belong in the same family?

Mix English novels, encrypted backups, satellite images and executable binaries into one training corpus and a shared model may become mediocre at everything.

Partition too aggressively and you lose cross-group repetition.

Corpus construction is therefore another boundary problem. We need collections coherent enough to share structure but broad enough to justify the shared infrastructure.

Human Culture Is Corpus Compression

A proverb is short because generations share the stories, assumptions and social patterns that let it unfold. A scientific term is compact because textbooks, laboratories and professional training form a vast supporting corpus.

One person’s sentence becomes shorter because civilisation has already paid to build the surrounding library.

Language is not merely compressed inside an individual mind. It is compressed across communities.

Education: A Curriculum Is a Corpus

One lesson makes more sense because earlier lessons exist. “Use conservation of energy” is short only after the learner has encountered energy, work, kinetic energy, potential energy and system boundaries.

A curriculum therefore accumulates shared state.

Good sequencing lets later lessons reuse earlier concepts instead of rebuilding everything from zero. Poor sequencing assumes a corpus the learner never received.

Primary School: Find What Every Page Shares

Give pupils five worksheets with the same header and different questions. Ask them which parts need to be stored five times and which could be represented once as a template.

The family structure becomes visible immediately.

Secondary School: Version a Paragraph

Write one paragraph, then create five revisions changing only a sentence each time. Compare storing five complete copies with storing one base plus changes.

Students see temporal corpus compression directly.

JC and Beyond: Joint Modelling Changes the Rate

At higher levels, the mathematics of joint and conditional distributions explains why related observations should not always be coded independently. Shared statistical structure lowers the joint description relative to naive separate treatment.

Practical corpus compression turns that principle into dictionaries, models, chunk stores, references and deltas.

A Corpus Checklist

1. What structure repeats across files?
2. Which assets or chunks are exact duplicates?
3. Which files are versions of one another?
4. Can a shared dictionary or model capture family structure?
5. How much setup cost will be amortised?
6. What dependencies does shared storage create?
7. How will shared components be protected?
8. When should the corpus be split because its members no longer share enough structure?

The Deeper Point

Compression depends on what we decide to call the object.

If the object is one file, we see one set of patterns.

If the object is a million-file corpus, we see another.

Sometimes the most important redundancy does not live inside the file at all.

The file becomes smaller when we finally notice that it was never alone.