Put two files together. Will they compress better?
Sometimes dramatically.
Sometimes not at all.
If the two files share templates, vocabulary, patterns or statistical relationships, the combined compressor can exploit them. But if the files are genuinely independent, knowing one tells us nothing about the other. There is no shared uncertainty to remove.
Compression has found its relational boundary.
Quick Read
- Correlation creates opportunities for joint compression.
- Independence means knowing one variable does not change the distribution of the other.
- For independent variables, mutual information is zero.
- Their joint entropy is the sum of their individual entropies.
- Combining unrelated files can still help operationally, but not because one statistically predicts the other.
- Apparent patterns in finite data can be accidental and may mislead a compressor.
- A model that invents dependence where none exists can waste bits rather than save them.
The One-Sentence Answer
When two sources are statistically independent, knowledge of one does not reduce uncertainty about the other, so there is no shared information for a compressor to exploit between them.
Two Coins That Do Not Talk
Flip two fair coins independently.
Knowing the first landed heads does not change the second coin’s probability of heads. It remains one half.
The first outcome gives no predictive advantage for the second.
If we encode the pair, we must essentially pay for both uncertainties. There is no hidden discount from correlation.
Joint Entropy Makes the Boundary Precise
For two random variables X and Y, joint entropy H(X,Y) measures the uncertainty in the pair.
When X and Y are independent:
H(X,Y) = H(X) + H(Y)
Nothing is saved by knowing the other variable because conditional entropy does not shrink:
H(X|Y) = H(X)
And mutual information is zero.
These equations are the mathematical form of a simple sentence: the two sources have nothing statistically useful to tell each other.
Contrast With Correlation
Suppose Y is almost always equal to X. Now knowing X makes Y highly predictable. Instead of paying independently for both, we can describe X and then mostly describe the occasions when Y differs.
correlated pair = shared structure + differences
That is the opportunity side information and corpus compression exploit.
Independence removes that opportunity.
Combining Files Does Not Manufacture Correlation
Suppose you concatenate a weather database with cryptographically secure random bytes.
The weather data may be compressible. The random bytes may not be. Putting them in one archive does not cause the weather to explain the random section.
A sensible compressor may still handle each region appropriately, but there is no cross-source relationship to exploit.
A larger corpus is only useful when the added material shares structure relevant to the target.
The Wrong Corpus Can Hurt
Imagine training a text dictionary by mixing legal contracts, encrypted blobs, Mandarin poetry, machine code and random sensor noise.
The corpus is larger, but not necessarily more informative for compressing English legal contracts.
Irrelevant data can consume model capacity, pollute statistics and increase search cost.
More data is not the same as more useful shared information.
False Correlation Is a Compression Trap
Finite datasets contain coincidences.
Flip a fair coin 20 times and you may see HHTHHTHHT… A clever model may invent a rule. But if the process is truly independent, that rule will fail on future flips.
Encoding the invented model costs bits. When the pattern does not persist, residuals also cost bits.
This is overfitting viewed through compression: mistaking accidental coincidence for reusable dependence.
Independence Is Relative to the Variables We Chose
Two raw measurements may look unrelated until we condition on a third variable or transform the representation.
Conversely, an apparent relationship can disappear after accounting for a common cause.
So independence is not a casual visual judgement. It is a statement about a probability distribution and chosen variables.
Compression models can only exploit dependence they represent correctly.
Encryption Tries to Destroy Accessible Dependence
Good encryption aims to make ciphertext reveal no useful information about plaintext without the key, under its security model. The resulting bytes should resist ordinary statistical prediction.
This is why compression is normally performed before encryption. Once accessible redundancy has been concealed, a compressor has little structure left to exploit.
Encryption and compression pursue almost opposite representational goals: compression exposes and exploits predictability; encryption tries to conceal exploitable predictability from unauthorised observers.
Independent Noise Sets a Floor
Suppose a sensor measures a smooth signal plus independent measurement noise.
The smooth signal is predictable. The independent noise is not predictable from the signal or neighbouring measurements under the model.
Lossless compression must still preserve that noise exactly. Once the structured component has been removed, the residual noise can dominate the remaining description.
This is one reason exact scientific data can remain stubbornly large even after excellent modelling.
Do Not Confuse Independence With Difference
Two things can look very different and still be statistically related. Two things can look similar by coincidence and be independent.
Compression cares about predictive relationship, not visual resemblance alone.
If knowing A changes the probabilities for B, there is potentially exploitable dependence. If it does not, A cannot buy us a shorter description of B on average.
Human Reasoning Loves Inventing Connections
People are excellent pattern detectors—and sometimes too excellent.
We see faces in clouds, lucky streaks in random sequences and causal stories in coincidences.
Compression offers a useful discipline: a proposed pattern should earn its complexity by making future or remaining data cheaper to describe.
If the “pattern” costs more than it saves, perhaps it was never structure at all.
Education: Transfer Requires Dependence
A student learns one method and hopes it helps with another problem.
Transfer succeeds when the problems share underlying structure. If they do not, forcing the old method onto the new problem creates error.
Good learners ask what is actually invariant or related rather than assuming every familiar-looking problem belongs to the same family.
Primary School: Does This Clue Help?
Ask a child to guess a hidden number. Offer two clues: “It is greater than 50” and “I am wearing blue socks.”
One clue changes the search space. The other, assuming your socks are unrelated to the chosen number, does not.
That is independence made intuitive.
Secondary School: Correlation Versus Coincidence
Give students small random datasets where accidental patterns appear. Then generate more observations and watch some patterns disappear.
They learn why small samples can tempt us into inventing structure.
JC and Beyond: Mutual Information as the Discount
At higher levels, mutual information I(X;Y) quantifies the reduction in uncertainty about one variable supplied by the other.
When X and Y are independent, I(X;Y)=0. When they share structure, mutual information is positive.
This lets us view mutual information as the potential average description saving created by statistical dependence under the appropriate coding interpretation.
An Independence Checklist
- Does knowing source A change predictions for source B?
- Is the relationship stable or only visible in a small sample?
- Could a hidden variable explain the apparent dependence?
- Does a different representation reveal genuine correlation?
- Is the model spending more bits describing a relationship than it saves?
- Would separate models outperform one joint model?
- Are we accidentally mixing unrelated corpora?
The Deeper Point
Compression is powered by relationships.
Repetition says this resembles something already seen. Prediction says the past constrains the future. Side information says another variable narrows the possibilities. Corpus compression says other files share structure.
Independence is where those discounts stop.
If one thing genuinely tells you nothing about another, no amount of clever bookkeeping can make that nonexistent relationship pay the bill.
Explore the How Compression Works.