Ten files contain the same 8 MB library.
A naive system stores it ten times.
A deduplicating system asks a different question:
Have these bytes already been stored somewhere else?
This is the first pillar beneath How Compression Works | The Corpus. The master owns cross-file redundancy generally. This article isolates exact shared identity: how many files can reuse one stored chunk without becoming one file.
Quick Read
Deduplication reduces storage by identifying repeated content and storing one physical copy with references from many logical objects. The key design choices are where chunk boundaries are placed, how chunk identity is verified, whether chunks are fixed-size or content-defined, how references are indexed, and how shared components are protected. Deduplication differs from ordinary entropy coding because the main saving comes from recognising identity across objects rather than assigning shorter codes to frequent symbols.
files → chunking → chunk identity → duplicate lookup → store new chunk once / reuse old chunk → logical file references → reconstruction
Deduplication Is Compression Through Identity
Ordinary compression asks:
Can this sequence be represented more efficiently?
Deduplication asks:
Is this sequence literally the same content we already possess?
That distinction matters because exact reuse can operate across file boundaries, directory names and versions.
Whole-File Deduplication Is the Simplest Case
Two files are byte-for-byte identical.
Store one physical object.
Record two logical references.
This works brilliantly for exact copies and poorly when one byte changes because the whole-file identity changes.
Chunking Makes Partial Reuse Possible
Split files into smaller pieces.
Now two files can differ in one region while sharing the rest.
A backup containing yesterday’s 4 GB database and today’s slightly modified 4 GB database may be able to reuse most unchanged chunks instead of storing another full 4 GB.
Chunk Size Is a Trade-Off
Large chunks reduce metadata and lookup overhead.
Small chunks increase the chance that local changes leave more surrounding chunks reusable.
The useful chunk size depends on workload, storage cost, indexing cost, memory and expected change patterns.
Fixed-Size Chunking Has a Boundary Problem
Imagine cutting a file every 1 MB.
Insert one byte near the beginning.
Every later fixed boundary shifts.
The content remains mostly the same, yet many chunks become different because the cutting grid moved relative to the bytes.
Content-Defined Chunking Tries to Follow the Data
Instead of cutting only by absolute position, a content-defined chunker looks for boundary conditions derived from the data itself.
When bytes are inserted, many downstream boundaries can eventually re-synchronise with the unchanged content.
This improves reuse across versions that have insertions or deletions rather than only in-place edits.
The Chunker Becomes Part of the Corpus Contract
Two repositories can contain identical files and still fail to deduplicate against one another if they cut those files differently.
Chunking parameters therefore become shared state.
Recent restic discussions make the operational point clearly: deduplication depends on compatible chunking parameters because reuse happens at the chunk level, not merely at the whole-file level.
Hashes Give Chunks Stable Names
A content hash can act as an identifier derived from the bytes.
If two independently observed chunks produce the same verified content identity, the system can reuse one stored object.
This is the core idea behind content-addressed storage:
name the object by what it is, not only by where somebody put it.
Hash Equality Must Be Treated as an Integrity Claim
Hashes are designed to make accidental collisions extraordinarily unlikely when appropriate algorithms are used.
But a storage system still needs clear integrity rules:
- which hash algorithm;
- which canonical bytes are hashed;
- whether content is reverified on read;
- how corruption is detected;
- what happens if metadata and content disagree.
Deduplication saves storage by increasing shared dependence. Integrity therefore becomes more valuable, not less.
References Reconstruct Logical Files
A logical file can be represented as an ordered list of chunk identities plus metadata.
The physical store may know nothing about the filename.
Reconstruction follows references and concatenates the required chunks in the correct order.
Metadata and Content Deduplicate Differently
Rename an identical image.
The file content can remain deduplicated while filename metadata changes.
Move it to another directory.
Content identity can remain stable while the directory tree records a different path.
This distinction is why content-addressed systems can preserve reuse across logical reorganisation.
Compression Before Deduplication Can Hide Identity
Take two largely identical files.
Compress each independently with state that reacts to earlier bytes.
A small source difference can alter much of the compressed representation.
Deduplication often works better on stable logical content or chunks before transformations make equivalent structure harder to recognise.
Encryption Can Hide Cross-Object Identity Too
Strong encryption deliberately removes visible structure.
Encrypt identical chunks under different randomised encryption states and the ciphertext can differ completely.
Secure deduplicating systems therefore have to design carefully around the ordering of identity, encryption and storage.
This is a systems-security boundary, not a reason to weaken encryption.
Deduplication Creates Shared Blast Radius
One stored chunk is referenced by 100,000 logical files.
If that chunk is lost or corrupted, 100,000 files may be affected.
The more storage a shared object saves, the more carefully that shared object should be protected.
Reference Counting Is Not Enough by Itself
Garbage collection needs to know whether a chunk is still reachable from any live logical object or snapshot.
Delete one file and the shared chunk may still be needed by thousands of others.
A storage system must remove unreachable data without confusing “not used here anymore” with “not used anywhere anymore.”
Snapshots Turn Deduplication Into Time Compression
Night 1: full filesystem state.
Night 2: mostly the same content.
Night 3: mostly the same again.
Deduplicated snapshots can preserve many logical historical states while paying physically only for new content and changed metadata.
The second pillar, Delta Encoding, owns another way to exploit version similarity.
Deduplication and Delta Encoding Are Not the Same
Deduplication reuses identical content.
Delta encoding stores difference relative to a related base.
Two chunks can be highly similar yet not deduplicate at all.
A delta representation may still make them cheap to store together.
Canonical Form Can Increase Deduplication
Two logically identical documents differ only in irrelevant formatting.
Normalize them into one canonical representation and duplicate structure becomes visible.
Canonical Form owns the general identity-normalisation problem.
Boundaries Decide Which Redundancy Is Visible
File boundary.
Chunk boundary.
Repository boundary.
Tenant boundary.
Each boundary can prevent a system from seeing repetition outside it.
Boundaries owns why where we cut information changes what compression can exploit.
Privacy Can Require a Deliberate Deduplication Boundary
Cross-user deduplication can theoretically reveal whether another user possesses particular content if the system exposes the wrong signals.
Privacy-sensitive systems may therefore limit deduplication scope or hide confirmation channels.
Storage efficiency is not allowed to override confidentiality requirements.
A Better Deduplication Model
logical objects → stable bytes → chunk boundaries → content identity → repository lookup → shared physical object → reference graph → integrity protection → garbage collection → reconstruction
A 28-Lens Deduplication Audit
- Corpus: which objects may share content?
- Scope: file, repository, tenant or global?
- Chunking: whole-file, fixed or content-defined?
- Average chunk: what size target?
- Boundary stability: how do insertions affect later chunks?
- Identity: how is a chunk named?
- Hash: which algorithm and canonical bytes?
- Verification: is content checked on write/read?
- Lookup: how is duplicate discovery indexed?
- Reference: how do logical files point to chunks?
- Order: how is reconstruction sequence preserved?
- Metadata: what cannot deduplicate with content?
- Rename: does path change preserve content reuse?
- Compression order: can transforms hide identity?
- Encryption: where does it occur relative to identity?
- Snapshot: how many historical states reference chunks?
- Reachability: which objects are still live?
- Garbage collection: when is a chunk safe to remove?
- Failure: how many files depend on one chunk?
- Replication: how are shared chunks protected?
- Corruption: how is damage detected?
- Recovery: can one bad pack be repaired?
- Privacy: can dedupe leak possession information?
- Canonicalisation: can irrelevant differences be normalised?
- Delta alternative: is near-duplicate similarity being missed?
- Amortisation: is index cost justified by reuse?
- Drift: is corpus structure changing?
- Return: does reconstructed content exactly match the original?
Laboratory 1: Shift One Byte
Create two long strings that differ by one inserted byte near the start. Compare fixed-size chunks with content-defined chunking conceptually. Which method can regain alignment?
Laboratory 2: One Chunk, Three Files
Build three paper “files” from labelled chunks. Store repeated chunk cards once and reconstruct each logical file with references.
Laboratory 3: Delete One Snapshot
Given a graph of snapshots and shared chunks, identify which chunks become unreachable after one snapshot is removed. This is the logic behind safe garbage collection.
For Primary Readers
If five LEGO models use the same instruction sheet, keep one sheet and let all five models point to it. That is the core intuition.
For Secondary Readers
Distinguish whole-file identity from chunk identity and explain why changing one byte can defeat fixed boundaries.
For Advanced Readers
Model deduplication as content-addressed reuse over a reference graph. Compression gain depends on boundary stability, repeated-chunk frequency and metadata/index overhead; integrity risk scales with shared reachability.
Common Misconceptions
- “Deduplication means compressing repeated bytes inside one file.” It usually means reusing identical content objects across logical objects or versions.
- “Identical files are the only thing that can deduplicate.” Chunk-level systems can reuse parts of different files.
- “Smaller chunks are always better.” Metadata and lookup costs grow as chunks shrink.
- “Hashes remove the need for integrity checks.” Systems still need explicit verification and corruption handling.
- “Shared storage only reduces cost.” It also creates shared failure and privacy boundaries.
Research Corridor
- restic Documentation — References and Design — practical content-defined chunking and deduplicated backup concepts.
- Git User Manual — Efficient Object Storage — content-addressed objects and pack storage provide a useful comparison with deduplication and delta compression.
- eduKateSG — Canonical Form.
- eduKateSG — The Corpus.
Frequently Asked Questions
What is deduplication?
Deduplication is a storage technique that detects identical content and stores one physical copy while allowing multiple logical objects to reference it.
What is content-defined chunking?
It is a chunking strategy that chooses boundaries from the content itself rather than only from fixed byte positions, helping boundaries resynchronise after insertions or deletions.
How is deduplication different from delta compression?
Deduplication reuses exact identical chunks; delta compression can represent a changed object relative to a similar base even when the bytes are not identical.
Final Thought: One Physical Chunk, Many Logical Stories
Copies look separate at the file layer.
The corpus can reveal that their content is shared.
Deduplication works when a storage system stops asking how many names exist and starts asking how many distinct pieces of content actually exist.
THE CORPUS · FOUR PILLAR LEGS
Return to The Corpus, or continue through Delta Encoding, Shared Dictionaries and Corpus Drift. Return to the Information & Representation Hub.