Permutation tests work by specifying what could be rearranged if the null hypothesis were true, repeatedly permuting labels, signs, assignments or residual structures according to that null and the study design, recomputing the chosen test statistic for each permitted rearrangement, and comparing the observed statistic with the resulting randomisation distribution. The method can produce exact finite-sample inference when all valid permutations are enumerated, or a Monte Carlo approximation when only a random subset is used. Its central assumption is exchangeability: the rearrangements must represent outcomes that would be equally plausible under the null.
A permutation test begins with a provocative question:
If the null hypothesis were true, which parts of this dataset could have been relabelled without changing its probability?
That question turns the observed study design into a reference-distribution generator.
NIST describes the general procedure clearly: calculate the statistic in the observed data, repeatedly permute the data or group assignments under the null, calculate the statistic in each permutation, and compare the observed statistic with that empirical reference distribution. NIST also distinguishes randomisation or permutation tests from ordinary parametric tests by noting that the reference distribution comes from rearrangements of the observed data rather than a standard theoretical distribution.
Quick Read
NULL HYPOTHESIS → DEFINE EXCHANGEABLE UNITS → CHOOSE TEST STATISTIC → IDENTIFY ALLOWED PERMUTATIONS → SHUFFLE / REASSIGN → RECOMPUTE STATISTIC → REPEAT → RANDOMISATION DISTRIBUTION → COMPARE OBSERVED STATISTIC → EXACT OR MONTE CARLO P-VALUE → INTERPRET WITH DESIGN
1. A Permutation Test Is a Hypothesis Test Built From Rearrangements
Ordinary hypothesis tests often begin with a mathematical distribution such as t, F, chi-squared or normal.
A permutation test instead constructs the reference distribution by rearranging observed information in ways justified under the null.
The reference distribution is therefore tied directly to the observed values and the permutation rule.
2. The Test Statistic Comes Before the Shuffling
Difference in means.
Difference in medians.
Correlation.
Regression coefficient.
F statistic.
Maximum statistic across many tests.
The statistic should express the scientific contrast of interest before permutations are generated.
3. Exchangeability Is the Core Validity Condition
Under the null, exchanging two labels should not change the joint probability distribution of the data if those labels are allowed to be permuted.
That is the essence of exchangeability.
If some observations are more likely to receive one label because of blocking, time, family structure or cluster membership, unrestricted permutation may be invalid.
4. Exchangeability Is Not the Same as Independence
Independent identically distributed observations are exchangeable.
But exchangeability can also exist within restricted structures such as matched pairs or blocks.
The practical question is not merely “are observations independent?”
It is “which rearrangements preserve the null distribution under this design?”
5. In a Two-Sample Test, Group Labels Are the Typical Permutation Object
Suppose one group has n₁ observations and another has n₂.
Under a null of identical distributions and exchangeability, combine all observations, assign n₁ of them to group A and the rest to group B, then recompute the statistic.
NIST describes this exact structure in its two-sample permutation-test documentation.
6. Enumerating Every Valid Rearrangement Gives an Exact Randomisation Distribution
If only a small number of allocations are possible, calculate the statistic for every one.
The resulting finite set is the exact permutation distribution conditional on the observed values and permutation scheme.
No asymptotic normal approximation is required.
7. The Number of Permutations Grows Combinatorially
For a two-group design, the number of possible allocations is:
(n₁+n₂)! / (n₁!n₂!)
Even moderate samples can produce enormous permutation spaces.
NIST therefore notes that random sampling from the set of all permutations can provide a practical approximation.
8. Monte Carlo Permutation Uses a Random Subset of Permutations
Instead of enumerating every permutation, randomly generate B valid permutations.
Compute the statistic for each.
The fraction at least as extreme as the observed statistic approximates the exact permutation p-value.
More permutations reduce Monte Carlo error but do not change the underlying observed evidence.
9. Monte Carlo P-Values Should Avoid Reporting Impossible Zeros
If none of 999 sampled permutations is as extreme as the observed statistic, reporting p = 0 is wrong.
The exact p-value could still be positive.
A common conservative Monte Carlo convention uses:
(b + 1) / (B + 1)
where b is the number of sampled permutation statistics at least as extreme as the observed one.
10. The Observed Arrangement Belongs in the Reference Logic
The observed labelling is one possible arrangement under the null.
Including it in the Monte Carlo counting logic prevents artificially tiny p-values and aligns the simulation with the finite permutation space.
11. Two-Sided Permutation Tests Need a Definition of “As Extreme”
For symmetric statistics, absolute value may be natural.
For asymmetric discrete distributions, two-sided p-values can be defined in several legitimate ways.
The tail rule should be specified before seeing the result.
12. The Permutation Distribution Is Conditional on the Observed Values
Ordinary two-sample permutation logic often conditions on the pooled observed outcomes and asks how those values could have been assigned to labels under the null.
The randomness comes from assignment or exchangeability, not from imagining entirely new outcome values.
This is a different inferential construction from the bootstrap.
13. Randomisation Tests Are Especially Natural in Randomised Experiments
In an experiment where treatment was randomly assigned, the known assignment mechanism directly generates the null reference distribution.
Under Fisher’s sharp null hypothesis of no treatment effect for any unit, outcomes would have been identical regardless of assignment.
Reassign treatment labels according to the actual randomisation scheme and compare the resulting statistics.
14. Fisher’s Sharp Null Is Stronger Than a Zero Average Effect
“No effect for any individual” is stronger than “average treatment effect equals zero”.
Positive and negative individual effects could average to zero while violating the sharp null.
The null hypothesis determines what potential outcomes can be imputed during randomisation inference.
15. The Randomisation Scheme Must Match the Experiment
Completely randomised assignment allows one permutation set.
Blocked randomisation allows only treatment swaps within blocks.
Cluster randomisation permutes cluster-level treatment assignments, not individuals.
Using unrestricted individual shuffling after cluster randomisation invents assignments that the experiment could never have produced.
16. Matched-Pair Randomisation Permutes Within Pairs
If one treatment is randomly assigned within each matched pair, the valid rearrangement is to swap treatment labels inside a pair or leave them unchanged.
There are 2m possible assignment patterns for m independent pairs.
The pair—not the person—is the randomisation unit.
17. Sign-Flipping Is a Permutation Principle for Symmetric Paired Differences
Calculate one difference per pair.
Under a null where the sign of each difference is exchangeable, randomly flip signs and recompute the mean, median or other statistic.
The resampling object is the sign, not the original observations individually.
18. Permutation Tests Are Often Called Distribution-Free, but the Phrase Needs Care
They avoid specifying a parametric outcome distribution such as normality for many problems.
They still require a valid exchangeability or randomisation structure.
Replacing parametric assumptions with design assumptions is not the same as having no assumptions.
19. Independence Often Remains Necessary at the Permuted Unit
NIST’s general two-sample permutation documentation notes independence of observations as an assumption.
When observations are clustered or repeated, unrestricted row-level permutation is invalid.
The appropriate unit may be a person, family, school, site or time block.
20. Unequal Variances Can Break Naive Two-Sample Permutation Tests
Under a null of equal means but unequal variances, observations from the two groups are not fully exchangeable even when the means match.
A naive permutation of raw group labels can therefore test equality of entire distributions rather than only equality of means.
Studentised statistics can improve robustness to heteroskedasticity.
21. Studentisation Can Make Permutation Tests More Pivotal
Instead of permuting a raw mean difference, use a t-like statistic that divides by an estimated standard error.
The resulting statistic can have a more stable null distribution across heterogeneous variances.
Permutation validity still depends on the theoretical conditions appropriate to the chosen statistic and scheme.
22. Permutation Tests Can Target Correlation
Keep X fixed and randomly permute Y labels across independent units under a null of no association and exchangeability.
Recompute Pearson r, Spearman rho or another association statistic.
The observed correlation is compared with correlations created by broken pairings.
23. Time Series Cannot Usually Be Permuted Arbitrarily
Randomly shuffling daily temperatures destroys autocorrelation, seasonality and trend.
The resulting reference distribution no longer represents the null time-series process.
Circular shifts, block permutations, phase randomisation or model-based residual procedures may be required depending on the null.
24. Spatial Data Need Spatially Valid Permutations
Neighbouring locations share environmental and social structure.
Unrestricted shuffling can erase that dependence.
Restricted permutations, toroidal shifts, spatial blocks or model-based approaches may be needed to preserve the null dependence pattern.
25. Permutation in Regression Requires Careful Null Construction
Simply permuting an outcome can destroy relationships with nuisance covariates that should remain under the null.
Residual-permutation methods such as Freedman–Lane and related schemes permute residual components after accounting for nuisance structure.
Different schemes have different exactness and robustness properties.
26. Freedman–Lane Permutes Residuals From the Reduced Model
Fit the model without the predictor being tested.
Permute residuals from that reduced model under an exchangeability condition.
Add them back to fitted nuisance structure and refit the full model.
The method aims to preserve nuisance effects while randomising the component relevant to the tested predictor.
27. Permutation ANOVA Recreates the F Reference Distribution Empirically
Calculate an F statistic across groups.
Permute group labels under the null.
Recalculate F repeatedly.
NIST documents this k-sample use explicitly.
28. Rank Statistics Can Also Be Permuted
The test statistic need not be parametric.
One can permute labels and calculate rank-sum, Kruskal–Wallis or other robust statistics.
The permutation principle is about generating the null reference distribution; the statistic can be chosen for robustness or scientific relevance.
29. Exact Permutation Tests Can Be Conservative With Discrete P-Values
When only a small number of permutations exists, possible p-values come in coarse steps.
A nominal α = 0.05 test may have an attainable rejection probability below 0.05.
The discreteness is a real finite-sample property rather than a computational failure.
30. Mid-P Adjustments Trade Conservatism for Exactness Guarantees
Mid-p methods count half the probability of permutations tied with the observed statistic.
This can reduce conservatism in discrete tests.
But the resulting procedure does not preserve exact Type I error control in the same strict way as ordinary exact p-values.
31. Ties Matter
Discrete outcomes and rank statistics can generate many identical permutation statistics.
The p-value convention must state how ties are counted.
Ignoring ties can subtly distort finite-sample calibration.
32. Permutation Tests Can Be Inverted to Build Confidence Sets
Test a sequence of candidate parameter values under corresponding sharp nulls.
Retain the values not rejected.
The retained set forms a confidence interval or region under the inversion logic.
Randomisation inference can therefore support estimation as well as testing.
33. Permutation Tests Can Control Multiple Comparisons Through Joint Rearrangements
If many outcomes or features are tested, each permutation can generate the entire vector of test statistics.
The maximum absolute statistic across hypotheses can be recorded.
This builds an empirical null distribution for the most extreme test in the family.
34. Max-T Procedures Preserve Dependence Across Tests
Bonferroni treats tests conservatively without exploiting their dependence.
Permutation max-T methods can use the actual joint dependence structure among test statistics under the permutation null.
Westfall–Young procedures are classic examples for strong family-wise error control under suitable conditions.
35. Permutation Multiple Testing Still Needs a Valid Joint Null Structure
Joint resampling is not magic.
Subset pivotality or related conditions may be required for strong error-control guarantees in some procedures.
See How Multiple Testing Works.
36. Permutation Tests Can Be Powerful for Unusual Statistics
Suppose the scientifically meaningful statistic is a trimmed mean difference, maximum cluster size, network modularity score or custom discrepancy.
Deriving its null distribution analytically may be difficult.
If a valid permutation mechanism exists, the statistic can often be tested directly.
37. Permutation Tests Are Not Automatically More Powerful Than Parametric Tests
When normal-model assumptions hold exactly, a well-designed parametric test can be highly efficient.
A permutation test using the same efficient statistic can have similar power.
A poorly chosen statistic can make permutation testing less powerful.
Resampling does not replace statistical design.
38. The Test Statistic Encodes the Alternative You Care About
A mean difference is sensitive to location shifts.
A variance ratio targets spread.
A Kolmogorov–Smirnov-type statistic detects broader distributional discrepancies.
A test can only be powerful against alternatives its statistic notices.
39. Permutation Is Not the Same as Bootstrap
Bootstrap typically samples with replacement from an empirical or fitted distribution to approximate estimator variability.
Permutation testing usually rearranges existing values or labels without replacement under a null hypothesis to create a test reference distribution.
Bootstrap asks “how would my estimate vary?”
Permutation asks “how extreme is my statistic among arrangements allowed by the null?”
40. Permutation Is Not Cross-Validation
Cross-validation withholds observations to estimate generalisation or tune models.
Permutation deliberately breaks or reassigns structure to generate a null distribution.
Using permutation inside predictive modelling is possible, but the inferential jobs remain separate.
41. Permutation Feature Importance Is a Different Use of Shuffling
In machine learning, permutation feature importance shuffles one predictor in a validation dataset and measures how much predictive performance degrades.
This assesses dependence of predictions on the feature under the chosen shuffling intervention.
It is not automatically a hypothesis test or a causal effect of changing the feature.
42. Correlated Features Complicate Permutation Importance
If two predictors carry nearly the same information, shuffling one may have little effect because the model uses the other.
Alternatively, unconditional shuffling can create unrealistic feature combinations outside the joint distribution.
Conditional permutation methods attempt to respect feature dependence.
43. Permutation Tests Cannot Repair Confounding
A highly significant permutation test of an observational association can still reflect an unmeasured common cause.
Permutation calibrates the statistical test under its exchangeability structure.
It does not create causal exchangeability that the study design lacks.
See How Causal Inference Works.
44. Permutation Tests Cannot Repair Measurement Bias
If the outcome is badly measured, the permutation test evaluates the wrong outcome precisely.
The reference distribution can be exact while the construct is invalid.
Statistical exactness and scientific validity are separate layers.
45. A Very Small Permutation P-Value Does Not Tell You Effect Size
With enough observations, a tiny mean difference can be extraordinarily unusual under the null.
The permutation p-value addresses extremeness.
Magnitude belongs to How Effect Sizes Work.
46. Missing Data Can Break Exchangeability
If dropout differs by treatment and prognosis, the observed cases may no longer be exchangeable according to the original assignment mechanism.
Permuting observed treatment labels among completers can therefore test a selected population rather than the original randomised estimand.
Missingness must be handled as part of the design.
47. Restricted Permutations Preserve Known Design Structure
Block within school.
Permute within age stratum.
Swap treatment inside matched pairs.
Rotate treatments according to a crossover schedule.
The restriction encodes which assignments were genuinely interchangeable under the null.
48. Permutation Can Test Complex Experimental Designs
Factorial experiments.
Split-plot designs.
Cluster-randomised trials.
Stepped-wedge designs.
The key is to permute at the level and within the constraints of the actual randomisation.
49. Reproducibility Requires Reporting the Permutation Scheme
“We used a permutation test” is incomplete.
Report:
- what was permuted;
- what was held fixed;
- the null hypothesis;
- the test statistic;
- the number of permutations;
- whether all permutations or Monte Carlo sampling were used;
- the tail definition;
- how ties were handled;
- the random seed where simulation was used.
50. The Hostile Test: Unrestricted Shuffling in a Blocked Trial
A treatment was randomised separately within ten schools.
The analyst shuffles treatment labels across all students in all schools.
The permutation distribution includes assignments that the trial could never have produced.
The test is calibrated to the wrong experiment.
51. The Second Hostile Test: Raw Mean Permutation Under Severe Heteroskedasticity
Two groups have equal means but radically different variances and sample sizes.
Raw observations are not exchangeable under the equal-mean null.
An unrestricted mean-difference permutation test can have distorted Type I error.
A studentised statistic or alternative method may be needed.
52. The Third Hostile Test: Time Series Shuffled Like IID Rows
A strongly seasonal temperature series is randomly permuted day by day.
The null reference distribution destroys the very dependence that generated the data.
The resulting p-value answers an artificial question.
53. The Fourth Hostile Test: 999 Permutations and p = 0
No sampled permutation exceeds the observed statistic.
The report prints p = 0.000.
The Monte Carlo process only shows the p-value is smaller than the simulation resolution, not exactly zero.
54. The Fifth Hostile Test: Significant Association Presented as Intervention Effect
Study time and grades show a permutation p-value below 0.001.
Prior attainment and motivation affect both variables.
The association is strong.
The causal intervention effect is not identified.
55. Primary School: Permutation Begins as “What If the Labels Had Been Swapped?”
Two small groups of plants receive two treatments.
Students observe a difference in growth.
They write the treatment labels on cards and repeatedly reassign the labels to the same measured plants.
How often does a difference this large appear just from relabelling?
A permutation test asks whether the observed arrangement looks unusual among the arrangements the null hypothesis says should have been interchangeable.
56. Secondary School: The Null Determines the Legal Moves
Students can compare unrestricted shuffling with paired shuffling.
They see that statistics is not merely “shuffle many times”.
The scientific design determines which moves are legal.
57. JC and University: Permutation Testing Becomes Conditional Randomisation Inference
At higher levels, learners should reconstruct:
- null hypothesis;
- exchangeability;
- randomisation unit;
- test statistic;
- allowed permutation group;
- exact versus Monte Carlo enumeration;
- tail definition;
- studentisation;
- restricted permutations;
- joint multiple-testing permutations;
- design dependence.
58. Where Permutation Tests Fit in the eduKateSG “How Works” Landscape
- How P-Values Work — owns interpretation of the resulting tail probability.
- How Randomisation Works — owns the assignment mechanism that powers exact randomisation tests.
- How Bootstrap Works — distinct resampling method for sampling variability rather than null rearrangement.
- How Multiple Testing Works — uses joint permutation distributions for max-T and related procedures.
- How Correlation Works — association statistics that can be permutation-tested.
- How Regression Works — nuisance-adjusted permutation schemes and regression statistics.
Permutation Tests own one precise canonical job: construct a hypothesis-test reference distribution by rearranging labels, signs or values according to the exact exchangeability or randomisation structure implied by the null and study design.
59. What This Article Does Not Claim
- Permutation tests are not assumption-free.
- Exchangeability must be justified at the permuted unit.
- Unrestricted shuffling is invalid for many blocked, paired, clustered, temporal and spatial designs.
- A permutation p-value does not measure effect size.
- A significant permutation test does not prove causality.
- Raw mean permutations can be invalid under some unequal-variance nulls.
- Monte Carlo permutation p-values have simulation error and should not be reported as exact zeros.
- Permutation testing and bootstrap resampling solve different inferential problems.
- Permutation feature importance is not automatically a hypothesis test or causal effect.
- Exact statistical calibration cannot repair invalid measurement or target definition.
60. A Compact Permutation-Test Audit
- What null hypothesis is being tested?
- What statistic represents the alternative of interest?
- What is the exchangeable unit?
- What exactly is being permuted?
- Which features of the data must remain fixed?
- Does the permutation scheme match the original randomisation or sampling design?
- Are blocks, pairs, clusters or strata respected?
- Are time or spatial dependencies preserved?
- Are unequal variances compatible with the chosen raw permutation?
- Would studentisation improve validity?
- Are all possible permutations being enumerated?
- If Monte Carlo sampling is used, how many permutations?
- Is the observed arrangement included in the p-value calculation?
- How are ties handled?
- How is a two-sided tail defined?
- Could the statistic be insensitive to scientifically important alternatives?
- Are many hypotheses being tested jointly?
- Would a max-T or Westfall–Young procedure preserve dependence?
- Does missingness alter exchangeability?
- Does the result support association, randomisation inference or a causal estimand?
61. Frequently Asked Questions
What is a permutation test?
A permutation test constructs a null reference distribution by rearranging labels, signs or values in ways that are exchangeable under the null hypothesis, recomputing a statistic for each arrangement and comparing the observed statistic with that distribution.
Is a permutation test nonparametric?
Often yes in the sense that it may avoid specifying a parametric outcome distribution, but it still relies on exchangeability, randomisation or other structural assumptions. It is not assumption-free.
What is the difference between permutation and bootstrap?
Permutation rearranges existing labels or values under a null hypothesis to build a test reference distribution. Bootstrap usually samples with replacement from an empirical or fitted distribution to approximate estimator uncertainty.
Can a permutation test be exact?
Yes. If all valid permutations under the null are enumerated and the exchangeability or randomisation assumptions hold, the resulting finite-sample test can have exact conditional Type I error properties.
62. Authoritative Research Corridor
- NIST Dataplot — Two-Sample Permutation Test
- NIST Dataplot — Fisher Two-Sample Randomisation Test
- NIST Dataplot — K-Sample Permutation Test
- NIST SP 800-90B — General Structure of Permutation Testing
- eduKateSG — How Randomisation Works
- eduKateSG — How Bootstrap Works
Final Thought: The Null Hypothesis Writes the Rules of the Shuffle
Permutation tests look computational.
Underneath, they are intensely conceptual.
What is allowed to move?
What must stay together?
Which assignments could really have occurred?
Which rearrangements would still describe the same null world?
The computer can generate millions of permutations.
It cannot decide which permutations are scientifically legal unless we tell it.
A permutation test becomes exact only after the reasoning is exact: the right units, the right null, the right legal moves and the right statistic. The shuffle is merely the computation that follows.