Find your next learning step
A discrete test cannot usually land on every requested significance level; boundary randomisation fills the missing probability carefully, but it must be reported as part of the procedure.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The teaching chapters remain open; these grouped routes form the index.
Route 1 · See the lattice
Route 2 · Work the boundary mixture
Route 3 · Transfer the rule
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · SEE THE LATTICE
1. Why a discrete test has missing significance levels
An exact test starts from the null distribution of a statistic rather than a large-sample approximation. That sounds as if any significance level can be selected, but a finite reference set makes the rejection probability jump in steps. With 20 equally likely outcomes, a non-randomised upper-tail rule can have size 0, 0.05, 0.10 and so on. It cannot have size 0.07 by choosing an ordinary cut-off.
This is a lattice problem. The available tail probabilities are the sums of null masses attached to whole outcomes. If the requested level lies between two attainable sums, rejecting only above the boundary is conservative; rejecting at and above the boundary is too liberal. The missing probability must not be disguised by calling either rule “exact at 7%”.
The practical habit is simple: write the null masses, identify the strict tail, identify the boundary mass, and state whether randomisation is allowed. The arithmetic of the boundary is part of the test definition, not an afterthought.
CHAPTER 2 OF 12 · SEE THE LATTICE
2. Boundary mixing as a probability budget
Suppose outcomes more extreme than a boundary have total null probability p>, while outcomes exactly on the boundary have probability p=. A test that always rejects in the strict tail and rejects on the boundary with probability γ has size p> + γp=. To reach target α, choose γ = (α − p>)/p=, provided p> ≤ α ≤ p> + p=.
This formula is a budget. The strict tail has already spent p> of the allowed Type I error. Boundary randomisation spends only the remaining α − p>. The division by p= converts that remaining mass into the fraction of boundary cases to reject.
Randomisation is not permission to move probability anywhere. It applies only to the boundary needed to bridge the adjacent attainable levels. Values of γ below zero or above one indicate that the chosen boundary is wrong or that α lies outside the bracket.
CHAPTER 3 OF 12 · WORK THE BOUNDARY MIXTURE
3. Worked example: reach 7% exactly
Imagine a null reference distribution with a strict upper tail of 0.04 and a boundary event of 0.06. The target size is α = 0.07. The mixing probability is γ = (0.07 − 0.04)/0.06 = 0.50. Reject every strict-tail outcome and half of the boundary outcomes, using a pre-specified independent random device.
| Component | Null probability | Rejection rule | Contribution to size |
|---|---|---|---|
| Strictly beyond boundary | 0.04 | Always reject | 0.040 |
| At boundary | 0.06 | Reject with probability 0.50 | 0.030 |
| Below boundary | 0.90 | Do not reject | 0.000 |
| Total | 1.00 | Combined rule | 0.070 |
The non-randomised strict rule has size 0.04. The inclusive rule has size 0.10. Only the stated mixture has size 0.07. This is exact size under the assumed null distribution; it is not a claim that the model itself is perfectly true.
CHAPTER 4 OF 12 · WORK THE BOUNDARY MIXTURE
4. A reproducible implementation needs one more input
The rule must specify how the random decision is produced. A practical implementation can draw U uniformly from 0 to 1 and reject a boundary case when U ≤ γ. A simulation should record its seed. A regulatory or scientific procedure should state the randomisation mechanism before seeing the result.
| Observation region | Decision without U | Decision with U |
|---|---|---|
| Strict tail | Reject | Reject |
| Boundary | Undetermined | Reject when U ≤ γ |
| Interior | Do not reject | Do not reject |
If two analysts use different random draws, a boundary observation can receive different decisions even though both follow the same exact-size rule. That is a genuine cost. Reporting γ and the realised draw makes the procedure auditable; concealing the draw turns a designed random decision into an unexplained discrepancy.
CHAPTER 5 OF 12 · WORK THE BOUNDARY MIXTURE
5. Ties and unequal boundary masses
In permutation tests, several transformed data sets can share the boundary statistic. The relevant quantities are the number or probability of transformations strictly more extreme and the number or probability tied at the boundary. Equal counting is justified only when the transformations are equally weighted under the null.
For an equally weighted group G, one useful expression is γ = (α|G| − M+)/M0, where M+ counts values strictly beyond the boundary and M0 counts boundary ties. This is the same budget written in counts. If transformations have unequal probabilities, use probability masses instead.
Ties are not merely a computational nuisance. They determine the width of the jump that randomisation must cross. A large tie block can make the boundary decision common, so the reproducibility trade-off becomes more visible.
CHAPTER 6 OF 12 · TRANSFER THE RULE
6. Changed context: a 5% target on a 32-point reference set
Now change the context. There are 32 equally likely null outcomes, two strictly beyond the selected boundary and one at it. The strict-tail probability is 2/32 = 0.0625, already above 0.05. Randomising at that boundary cannot repair the rule because γ would be negative.
Move the boundary outward. Suppose one outcome is now strictly beyond and two are tied at the new boundary. Then p> = 1/32 and p= = 2/32. The required mix is γ = (0.05 − 1/32)/(2/32) = 0.30. Reject the single strict outcome and 30% of cases in the two-outcome boundary block.
The transfer check is to recompute the bracket before reusing γ. The original 0.50 mixture belonged to a different null lattice. Boundary mixing is not a universal correction factor.
CHAPTER 7 OF 12 · TRANSFER THE RULE
7. One-sided, two-sided and ordering choices
An exact test still needs an ordering that says which outcomes are at least as incompatible with the null as the observation. For a one-sided statistic, that ordering may be direct. For a two-sided discrete test, several defensible conventions can produce different boundaries and different attainable p-values.
Randomisation does not resolve an ambiguous ordering. First define the statistic, tail convention and treatment of ties; then calculate the strict and boundary masses. A test can have exact size for its chosen ordering while another exact test answers a different question.
This is why “exact” should never be read as “choice-free”. It describes probability calibration under a fully specified null procedure. It does not choose the scientific alternative, the statistic or the evidential direction for the analyst.
CHAPTER 8 OF 12 · TRANSFER THE RULE
8. Randomised p-values and decision equivalence
The same idea can be expressed through a randomised p-value. Add the strict-tail probability to a uniform fraction of the boundary mass. Under suitable conditions, the resulting p-value is uniform under the null, so rejecting when it is at most α gives exact size.
This representation can help compare decisions across α values because one realised auxiliary variable generates a position within the boundary interval. Yet the p-value is partly random. It should not be presented as if it were determined by the observed data alone.
A useful record separates the data-derived interval from the auxiliary position. Report the strict-tail probability, boundary mass, generated U and final randomised p-value. Then another reader can see which part came from the sample and which part came from the pre-specified random device.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. What exact size does and does not guarantee
Exact size controls the null rejection probability at the stated α under the stated null distribution and randomisation rule. It does not guarantee high power, a meaningful effect, a correct data-generating model or freedom from selection bias. It also does not make a Monte Carlo approximation exact unless its own sampling procedure is calibrated.
The decision cost matters. In a simulation study, boundary randomisation may be natural. In a high-stakes single case, a decision-maker may prefer a transparent conservative rule to a coin-flip boundary, even though its size is below α. That choice should be stated rather than hidden behind terminology.
Randomised and conservative tests answer the same calibration problem with different priorities: exact long-run size versus deterministic repeatability. Neither priority erases the need for substantive judgement.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Evidence and implementation checks
Hemerik and Goeman show why permutation tests can be conservative on a discrete lattice and give the boundary-randomisation formula for exact size. Their paper, Exact testing with random permutations, also makes clear that the preferred rule depends on context because randomisation reduces reproducibility.
Before using the method, verify exchangeability or the relevant invariance assumption, the weighting of transformations, the statistic and ordering, the tail convention, the handling of ties, and the independence of the auxiliary random draw. Test code on a tiny reference set where every null outcome can be enumerated and the final size can be summed directly.
For the wider subject map, return to The Gold Standard of Probability. Compare the preceding lattice calculations with exact rank-test power and attainable significance levels and with random arrivals and the inspection paradox.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Is a randomised exact test arbitrary?
No. The probability of randomisation is calculated from the strict-tail mass, boundary mass and target α. The realised boundary decision is random, but the long-run rule is specified exactly.
Why not always use the inclusive boundary?
Because including the entire boundary can make the null rejection probability exceed α. Exactness requires the complete rule, not simply an exact null distribution.
Can I reuse the same γ for another sample size?
Not safely. The reference lattice and tie structure can change with sample size, statistic and observed configuration. Recompute the bracket and mixture.
Is the conservative test wrong?
No. It controls size at or below α and is deterministic. It trades some rejection probability, and often some power, for repeatability.
Should the random seed be reported?
Yes when computation uses a pseudo-random generator. Report enough detail to reproduce the realised decision, while distinguishing reproducibility of the draw from validity of the statistical assumptions.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. A final transfer check
Take a new hypothetical null distribution: p> = 0.018, p= = 0.050 and target α = 0.025. The required mix is γ = (0.025 − 0.018)/0.050 = 0.14. The test rejects all strict-tail outcomes and 14% of boundary outcomes, giving 0.018 + 0.14 × 0.050 = 0.025.
Now change only the target to 0.01. The strict tail already contributes 0.018, so the same boundary cannot produce the smaller size. The rule must move to a more extreme boundary or change its ordering. That single check prevents the common mistake of forcing γ into the interval zero to one after choosing an unsuitable cut-off.
Continue through the Gold Standard of Education to see why changed-context checks matter, or return to the verified How X Works hub for another system.
Top · Contents · Previous · How X Works