Statistical independence works by asserting that the joint probability structure of two events or variables factorises into their separate probability structures. For events A and B, independence means P(A∩B)=P(A)P(B). When P(B)>0, the same statement can be written P(A|B)=P(A): learning B does not change the probability assigned to A. Independence is therefore not “the variables look unrelated” and not “their correlation is zero”. It is a strong probabilistic claim that knowing one carries no information about the other within the model.
Flip two fair coins under a model of separate tosses.
Knowing the first landed heads does not change the probability that the second lands heads.
That is the intuitive heart of independence.
The governing question: after learning one event or variable, does the probability distribution of the other remain unchanged?
Quick Read
EVENTS / VARIABLES → JOINT PROBABILITY → TEST FACTORISATION → P(A∩B)=P(A)P(B) → EQUIVALENT CONDITIONAL FORM → INDEPENDENCE → EXTEND TO MANY VARIABLES → DISTINGUISH PAIRWISE / MUTUAL → CONDITIONAL INDEPENDENCE → CHECK DESIGN / MECHANISM / DATA
1. Independence Is a Property of a Joint Distribution
You cannot decide whether X and Y are independent from their marginal distributions alone. Independence concerns how their values occur together.
2. Event Independence Uses Product Factorisation
Events A and B are independent when:
P(A∩B)=P(A)P(B).
The observed co-occurrence probability equals what would be expected if the events contributed their probabilities separately.
3. Conditional Probability Gives the Information Interpretation
When P(B)>0, independence implies P(A|B)=P(A). B happened, but the probability of A did not move.
See How Conditional Probability Works.
4. Independence Is Symmetric
If A is independent of B, B is independent of A. Product factorisation is symmetric even though conditional probability itself is directional.
5. Independent Does Not Mean Mutually Exclusive
Mutually exclusive non-zero-probability events cannot be independent. If A and B cannot happen together, P(A∩B)=0 while P(A)P(B)>0.
6. Disjointness Is Almost the Opposite Kind of Structure
If learning A tells you B definitely did not occur, A contains strong information about B. Independence says learning A changes nothing about B.
7. Random Variables Are Independent When Joint Probabilities Factorise
X and Y are independent when P(X∈C,Y∈D)=P(X∈C)P(Y∈D) for all appropriate measurable sets C and D.
8. For Discrete Variables, the Joint PMF Factorises
p(x,y)=p_X(x)p_Y(y) for every pair of possible values when X and Y are independent.
9. For Continuous Variables, Joint Density Factorisation Is a Useful Criterion
Where densities exist, independence corresponds to f(x,y)=f_X(x)f_Y(y) almost everywhere.
10. Independence Implies Zero Covariance When Moments Exist
If X and Y are independent with finite required moments, E(XY)=E(X)E(Y), so Cov(X,Y)=0.
11. Zero Covariance Does Not Usually Imply Independence
Let X be symmetric around zero and Y=X². Their covariance can be zero while Y is completely determined by X.
See How Covariance Works.
12. Joint Normality Is a Special Exception
For jointly multivariate-normal variables, zero covariance implies independence. This special theorem should not be exported to arbitrary distributions.
13. Independence Is Stronger Than No Linear Relationship
Correlation and covariance inspect second-order linear co-movement. Independence rules out every probabilistic dependence detectable through the joint distribution.
14. Pairwise Independence Is Not Mutual Independence
Three variables can be independent in every pair and still satisfy a deterministic relationship when considered together.
15. The XOR Example Makes This Visible
Let X and Y be independent fair binary variables and let Z be their XOR. Each pair can be independent, yet knowing any two determines the third. The triple is not mutually independent.
16. Mutual Independence Requires Every Subcollection to Factorise
For several variables, full mutual independence is stronger than checking only pairwise relationships. Every relevant joint event must factor into its marginals.
17. IID Contains Two Separate Claims
Independent and identically distributed means observations share the same distribution and are mutually independent under the model. Identical distribution does not imply independence, and independence does not imply identical distribution.
18. Repeated Measurements on One Person Are Usually Not Independent
Measurements share biology, environment and history. Treating them as IID can make standard errors too small and effective information appear larger than it is.
19. Students in One Classroom Are Usually Not Independent
They share a teacher, peers and school context. Hierarchical and cluster-aware methods exist because row count is not automatically independent-information count.
See How Hierarchical Models Work.
20. Time-Series Observations Commonly Depend on Their Past
Temperature today resembles temperature yesterday. Financial volatility clusters. Demand carries seasonality. Assuming temporal independence can badly misstate uncertainty.
21. Spatial Observations Commonly Depend on Nearby Locations
Neighbouring districts share geography, infrastructure and exposure. Spatial correlation violates simple independence even when observations are recorded on separate rows.
22. Randomisation Can Create Useful Independence
In experiments, treatment assignment is designed to be independent of potential outcomes under the randomisation mechanism. That independence supports causal comparisons.
23. Randomisation Does Not Make Outcomes Independent
Students within a randomised classroom can still have correlated outcomes. Independence of treatment assignment from potential outcomes is different from independence among observations.
24. Conditional Independence Is a Different Claim
X and Y may be dependent overall and independent after conditioning on Z. We write X ⟂ Y | Z under a common notation.
25. Common Causes Can Produce Marginal Dependence
Ice-cream sales and swimming accidents both increase with temperature. Conditional on temperature, their remaining dependence may shrink substantially.
26. Conditioning Can Also Destroy Independence
If X and Y independently influence admission to a programme, conditioning on admission can make X and Y associated among admitted applicants. Collider conditioning creates dependence.
27. Bayesian Networks Are Built From Conditional Independence Claims
A graph represents a factorisation of the joint distribution into local conditional distributions. The graph is useful because conditional independencies allow a huge joint probability object to be represented compactly.
28. Naive Bayes Makes a Strong Conditional Independence Assumption
Naive Bayes classifiers often assume predictors are conditionally independent given the class label. The assumption is rarely literally true, yet the classifier can work well when the resulting probability ranking remains useful.
29. Independence Can Make Computation Dramatically Easier
Joint densities factorise. Likelihoods become products. Variances of independent sums add. Simulation can generate components separately. Independence is mathematically valuable because it decomposes systems.
30. That Convenience Creates Temptation to Assume Too Much
Independence is often adopted because formulas become simple. Convenience is not evidence. The data-generating mechanism, design and residual diagnostics must support the assumption.
31. Sampling With Replacement Can Produce Independent Draws Under an Idealised Model
Each draw restores the population before the next draw. Sampling without replacement from a finite population creates negative dependence among inclusion outcomes.
32. Large Populations Can Make Finite-Population Dependence Negligible
When the sample fraction is tiny, sampling without replacement can be approximated as independent for some calculations. Approximate independence should still be recognised as an approximation.
33. Bootstrap Resampling Uses Conditional IID Draws From the Empirical Distribution
The ordinary nonparametric bootstrap draws observations independently with replacement from the observed empirical distribution, conditional on the original sample.
Dependent time-series or clustered data require different bootstrap architectures.
34. Independence Affects the Variance of Averages
For independent observations with common variance σ², Var(X̄)=σ²/n. Positive dependence adds covariance terms and slows the reduction in uncertainty.
35. Effective Sample Size Shrinks Under Positive Dependence
One thousand highly correlated observations can contain much less information than one thousand independent observations. Effective sample size translates dependence into an approximate independent-information equivalent.
36. Independence Is Not the Same as Exchangeability
Exchangeability says the joint distribution is invariant to permutations of labels. Exchangeable variables can be dependent. Hierarchical models often create exchangeable-but-dependent observations after integrating over shared latent effects.
37. Independence Is Not the Same as Orthogonality
In some mathematical contexts orthogonality means zero inner product or zero covariance. That is weaker than probabilistic independence except in special families.
38. Independence Is Not the Same as Causal Separation
Two variables can be statistically independent in one population despite a causal relationship whose effects cancel across subgroups. Conversely, variables can be dependent through a common cause without either causing the other.