THE GOLD STANDARD · PROBABILITY
Exact tests are discrete decisions, not smooth approximations
A reader-first guide to exact rank-test power, attainable p-values, lattice corrections and the assumptions that make a finite reference distribution valid.
Choose a reading route
Chapter contents
- 1. Exact tests live on a finite grid
- 2. From ranks to the exact null distribution
- 3. Attainable significance levels and conservative size
- 4. Worked example: six paired observations
- 5. Exact power is a probability over the rejection region
- 6. Why lattice corrections appear
- 7. Changed-context transfer: eight pairs and ties
- 8. Randomisation, mid-p and two-sided choices
- 9. A reproducible exact-power workflow
- 10. Limits, misconceptions and frequently asked questions
- 11. Evidence checks before trusting a result
- 12. Sources and connected reading
1. Exact tests live on a finite grid
An exact rank test does not begin with a smooth bell curve. It begins with a finite reference set: the rankings or sign changes that would be possible if the null hypothesis were true. Each allowable rearrangement contributes probability mass. The resulting tail probability can therefore move only in steps.
That simple fact changes how we interpret a nominal level such as 5%. If the attainable tail probabilities jump from 3.125% to 7.8125%, there is no non-randomised rejection rule whose size is exactly 5%. A defensible test uses the largest rejection region whose null probability does not exceed 5%, so it is conservative by construction.
- State the null symmetry or exchangeability that makes the rearrangements valid.
- Enumerate or sample the reference distribution without losing the observed arrangement.
- Choose a rejection boundary from attainable probabilities, not from a continuous approximation.
- Report the achieved size when it differs materially from the nominal level.
2. From ranks to the exact null distribution
For paired continuous differences, the Wilcoxon signed-rank statistic orders the absolute non-zero differences and then sums the ranks carrying a positive sign. Under a symmetric null distribution, each sign pattern is equiprobable conditional on the observed absolute ranks. With n non-zero untied pairs there are 2n sign patterns.
The exact one-sided p-value is the fraction of those patterns whose statistic is at least as large as the observed statistic. This is a conditional statement: it is exact for the randomisation structure being used. It does not mean that every rank procedure remains distribution-free after arbitrary missingness, dependent sampling, data-driven subgrouping or a null that fails to justify sign exchange.
Permutation rank tests use the same architecture. The group action defines the orbit of rearranged data, the statistic orders the orbit, and a tail count gives a valid level no greater than the target. Hemerik and Goeman formalise this group-invariance argument for complete and randomly sampled permutation sets.
3. Attainable significance levels and conservative size
Suppose M equally likely reference outcomes are available and r of them are at least as extreme as the observation. The basic exact tail probability is r/M. Its resolution is therefore 1/M before ties or stabilisers merge outcomes. A nominal α is attainable only when a rejection set has null probability exactly α.
When α is not attainable, the ordinary non-randomised rule rejects on a smaller set and has actual size below α. That is not a defect hidden by the word exact. Exactness means the type-I error is controlled under the stated null; it does not promise a continuously adjustable threshold.
A useful report gives both numbers: nominal level 0.05, achieved conditional level 0.046875. This lets a reader distinguish principled conservatism from a coding error or a power shortfall.
4. Worked example: six paired observations
Take six non-zero paired differences with distinct absolute values. Their ranks are 1 through 6, so the largest possible positive-rank sum is W+ = 21. Under the symmetric null there are 64 equiprobable sign patterns.
For an upper-tail test, W+ ≥ 19 corresponds to negative-rank subsets with total at most 2: no negative ranks, rank 1 alone, or rank 2 alone. Exactly three patterns qualify, so the achieved size is 3/64 = 0.046875. Moving the boundary down to W+ ≥ 18 admits five patterns and raises the size to 5/64 = 0.078125.
| Rule | Qualifying patterns | Exact size |
|---|---|---|
| W⁺ = 21 | 1 of 64 | 0.015625 |
| W⁺ ≥ 20 | 2 of 64 | 0.031250 |
| W⁺ ≥ 19 | 3 of 64 | 0.046875 |
| W⁺ ≥ 18 | 5 of 64 | 0.078125 |
The 5% rule is therefore W+ ≥ 19, not a critical value obtained by rounding a normal quantile. Its exact power against a specified alternative is the probability, under that alternative, that the data land in this three-pattern rejection region.
5. Exact power is a probability over the rejection region
Null calibration and power answer different questions. Calibration counts or weights reference outcomes under the null. Power integrates the same fixed rejection rule under an alternative distribution. For a rank test, the alternative changes both the signs and the ordering of absolute observations, so exact power usually requires a fully specified alternative or a careful numerical calculation.
A simulation estimate of power is trustworthy only when it regenerates complete datasets under the alternative, recomputes ranks and ties, and applies the already chosen exact critical rule. Reusing the observed ranks while changing only signs estimates a conditional quantity, not the unconditional power of the test.
- Lock the sample size, sidedness, tie rule and rejection boundary first.
- Generate data from a stated alternative, including dependence and censoring if present.
- Re-rank every simulated sample.
- Count rejections and attach a Monte Carlo uncertainty interval.
- Repeat near the decision boundary to detect numerical or implementation instability.
6. Why lattice corrections appear
A normal approximation replaces a stepped distribution with a continuous one. A continuity correction shifts the boundary by roughly half a lattice spacing, while a lattice-aware Edgeworth expansion adds periodic terms that reflect where the threshold sits relative to the grid.
Kolassa’s analysis of rank-sum and signed-rank statistics explains why ordinary Edgeworth terms can appear surprisingly accurate in some settings and why lattice effects cannot simply be dismissed. The safe lesson is practical: an asymptotic series is an approximation to a discrete distribution, and its quality depends on sample size, the statistic’s span, the evaluation point and tie structure.
Lattice correction is not permission to manufacture an unattainable exact p-value. It improves a continuous approximation. Exact enumeration remains the reference when it is computationally feasible and its null assumptions are appropriate.
7. Changed-context transfer: eight pairs and ties
Now increase the example to eight untied non-zero pairs. There are 256 sign patterns, so the smallest one-sided tail probability is 1/256 = 0.00390625 and the grid is four times finer than with six pairs. More sample information has improved threshold resolution, but 0.05 is still not automatically attainable.
Change the context again by introducing equal absolute differences or zeros. Midranks alter the statistic’s support, zero-handling conventions change the effective sample, and several sign patterns may produce the same value. The denominator may still count sign patterns, while the set of distinct statistic values is smaller. Software defaults now matter.
Transfer check: if a program reports p = 0.050000 for a tiny sample, ask whether it used exact enumeration, a normal approximation, a mid-p rule, randomisation at the boundary or rounded output. The number alone does not identify the method.
8. Randomisation, mid-p and two-sided choices
A randomised boundary test can achieve an otherwise unattainable α by rejecting with a specified probability when the statistic lands exactly on the boundary. This is mathematically clean but often awkward in one-off educational or clinical decisions because two identical datasets can receive different decisions.
A mid-p value assigns half the boundary mass to the tail. It often reduces conservatism, but it is not guaranteed to preserve the same finite-sample type-I bound as the ordinary exact p-value. It should be labelled, not silently substituted.
Two-sided exact p-values also require a definition. Doubling the smaller tail, ordering outcomes by probability, and ordering them by an absolute statistic need not agree in an asymmetric discrete distribution. The method must be chosen before seeing the result.
9. A reproducible exact-power workflow
A dependable analysis separates the combinatorics, the probability model and the decision rule. That separation makes it possible to audit where a discrepancy entered.
| Stage | Question | Evidence to retain |
|---|---|---|
| Null structure | Why are the transformations exchangeable? | Design and assumptions |
| Support | Which statistic values and masses are attainable? | Enumeration or validated recursion |
| Critical rule | What achieved size does the boundary produce? | Tail table |
| Alternative | What data-generating model defines power? | Parameters and code version |
| Sensitivity | Do ties, zeros or dependence change the result? | Changed-context reruns |
For large permutation spaces, random permutations can still give an exact level when the observed transformation is handled correctly and the randomisation scheme follows the proven construction. A naive Monte Carlo proportion without that design can be slightly anti-conservative.
10. Limits, misconceptions and frequently asked questions
Does exact mean the p-value equals the true probability that the null is correct?
No. A p-value is a tail probability computed under a null model. It is not the posterior probability of the null hypothesis.
Is a conservative exact test always preferable?
Not automatically. Conservatism protects type-I error but can reduce power. The right comparison considers the scientific question, design, attainable thresholds and pre-specified alternatives.
Can continuity correction make a small-sample rank test exact?
No. It can improve a normal approximation, but it does not reproduce the complete finite reference distribution in every case.
Are ties a minor implementation detail?
No. Ties change ranks, support and sometimes the validity of the nominal reference distribution. The tie convention belongs in the method report.
A misconception worth removing
Rank-based does not mean assumption-free. Exactness rests on a transformation invariance—such as exchangeability or sign symmetry—and on a statistic and sampling plan fixed independently of the observed outcome.
11. Evidence checks before trusting a result
- Verify a tiny case by hand, as in the 64-pattern example.
- Check that every reference outcome has the intended weight; duplicate statistic values do not imply duplicate probabilities.
- Compare enumeration with a second implementation or recursion.
- Confirm that the reported p-value lies on the attainable grid for the stated method.
- Run a null simulation and ensure the empirical rejection rate does not exceed the designed level beyond Monte Carlo uncertainty.
- Recompute power after plausible tie, variance and distribution-shape changes.
A result passes the Gold Standard only when a reader can reconstruct the rejection region, not merely reproduce a rounded p-value.
12. Sources and connected reading
Primary and specialist sources: Hemerik and Goeman, Exact testing with random permutations; Kolassa, Edgeworth approximations for rank sum test statistics; Elliffe, rank-permutation tests and exact probabilities; and Troendle, power approximations for Wilcoxon rank-sum tests.
Continue through The Gold Standard of Probability, the Gold Standard of Education and the How X Works hub.
Specialist route: asymptotic relative efficiency and local power, efficiency bounds, and second-order rank efficiency and finite-sample reversals.