Find your next learning step
When only a few independent scenes exist, the honest reference distribution may be a short list of allowed reassignments—not thousands of pixels pretending to be independent.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Frame the problem
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · FRAME THE PROBLEM
1. Why small audits need design-based logic
A remote-sensing audit may contain millions of pixels but only a handful of independently selected scenes, acquisitions or field campaigns. A conventional large-sample test can look precise because it counts pixels. Randomisation inference instead asks which assignments or label swaps were allowed by the audit design under a sharp null hypothesis.
If every allowed assignment is enumerated, the resulting test can be exact for that design. Exact does not mean universally correct. It means the rejection probability is controlled under the stated randomisation mechanism, statistic and null.
CHAPTER 2 OF 12 · FRAME THE PROBLEM
2. The permutation unit is the scientific unit
Permuting pixels across scenes usually destroys cloud, season, sensor and landscape structure. If workflows were assigned to whole scenes, scenes are the permutation units. If each scene pair received opposite workflows at random, the allowed operation is a within-pair label swap. Exchangeability follows the assignment design, not the number of rows in a raster.
Write the allowed group of permutations before computing the statistic. Restricted designs may permit swaps only within biome, season or sensor strata. Convenience permutations that were impossible in the real audit do not generate the correct reference distribution.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. A five-pair exact sign ledger
Suppose five matched scene pairs compare workflow A with workflow B, with the workflow label randomly assigned within each pair. The observed accuracy differences A minus B are +4, +3, +2, +1 and −1 percentage points. The observed sum is 9. Under the sharp null, each pair difference can have either sign, giving 2^5=32 equally likely sign patterns.
| Pair | Absolute difference | Observed sign | Contribution |
|---|---|---|---|
| 1 | 4 | + | +4 |
| 2 | 3 | + | +3 |
| 3 | 2 | + | +2 |
| 4 | 1 | + | +1 |
| 5 | 1 | − | −1 |
| Observed sum | — | — | 9 |
A one-sided statistic at least as large as 9 occurs when the total flipped absolute weight is at most 1: no flip, or either one-point pair alone. That is 3 of 32 assignments, so the exact one-sided p-value is 0.09375. The audit does not support a five-percent claim, even though four of five differences are positive.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. P-values are granular when scenes are few
With five paired scenes, the smallest possible one-sided p-value is 1/32 if only the most extreme assignment qualifies. No modelling trick creates finer randomisation evidence from the same design. This granularity is information: it shows that the audit was built to detect only large, consistent effects.
Report the full randomisation distribution or at least its attainable tail probabilities. A Monte Carlo sample of permutations is unnecessary when all assignments can be enumerated. If enumeration is impossible, sample the allowed permutations and include the observed assignment correctly in the p-value calculation.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Valid and invalid permutation schemes
The test changes when the unit changes. Pixel shuffles ignore spatial and scene dependence. Unrestricted scene shuffles can violate matching or stratification. Within-pair swaps respect the five-pair design, while biome-restricted permutations respect a blocked scene design. The statistic may be a mean difference, median, area-weighted loss or policy cost, but the permutation set must remain tied to assignment.
| Scheme | Nominal units | Allowed rearrangement | Audit meaning |
|---|---|---|---|
| Pixel shuffle | Millions of pixels | Any pixel label | Usually invalid for a scene-assigned workflow |
| Unrestricted scene shuffle | 10 scenes | Any 5 versus 5 split | Valid only for an unrestricted scene randomisation |
| Matched-pair sign flip | 5 pairs | Swap labels within each pair | Exact for the paired randomisation |
| Stratified scene shuffle | Scenes within strata | Shuffle only inside each stratum | Preserves blocked assignment |
Before analysing, draw the hierarchy: pixel, object, scene, acquisition, campaign. Mark where assignment or selection occurred. That diagram usually reveals the defensible randomisation unit.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. From a sharp null to a useful effect estimate
The sharp null says every scene would have the same outcome under either workflow. Rejecting it does not supply the magnitude of improvement. Pair the exact test with the observed scene-level effect, the distribution of pair differences and an interval obtained by inverting an appropriate randomisation test when the design permits.
For operational decisions, report how many scenes favour each workflow and whether the sign changes across biome or cloud strata. A non-significant exact test may still reveal a practically important but underpowered pattern; label that as uncertainty, not proof of no effect.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: a wildfire alert audit
Transfer the method to six regions comparing an old and new wildfire-alert threshold. If thresholds are randomly assigned within three matched region pairs during a trial, the valid exact test has only 2^3=8 sign patterns. Pixel-level false alarms remain outcomes aggregated to the region-period unit.
The changed context adds rare incidents and unequal exposure. Predefine a region-level loss combining missed incidents and false alerts, or analyse them separately. Do not invent exchangeability by swapping individual alerts when the threshold operated on an entire region.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Relation to cluster-robust and wild-bootstrap inference
The preceding few-cluster inference article uses cluster-robust variance and wild-bootstrap sensitivity. Randomisation inference is different: its validity comes from known assignment symmetries, not from an asymptotic approximation to a clustered regression.
Use spatially correlated reference-error diagnostics when scenes are observational clusters. Use exact scene permutations only when the selection or assignment design justifies the permitted swaps.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
Permutation tests are not assumption-free. They require invariance or exchangeability under the null for the chosen transformation. Observational scenes selected by convenience do not become randomised because labels can be shuffled in software. Spatial blocking repairs some dependence but cannot repair unmeasured confounding.
A p-value is not the probability that the workflows are equal, and an exact test is not automatically powerful. Multiple metrics, thresholds and subgroups create multiplicity. The reference labels and response design can also be wrong even when the assignment test is exact.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive the sampling frame, scene inclusion rules, randomisation seed, blocking variables, assignment probabilities, analysis unit, statistic and every allowed permutation. Preserve scene-level outcomes before pixel aggregation so another analyst can reproduce the test.
Show the observed assignment inside the reference distribution, count all equally or use known unequal assignment probabilities, and report attainable p-values. Repeat with alternative defensible scene summaries and leave-one-scene-out checks. Connect statistical results to reference-label quality and area-estimation requirements.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can I permute scenes if the workflow was not randomised?
Only if a defensible invariance or exchangeability argument exists under the null. Convenience shuffling is not randomisation inference.
Is an exact p-value always better than a bootstrap p-value?
It is exact for its design and null. A bootstrap may target a different sampling process; neither substitutes for defining the independent unit.
Why not use every pixel for power?
Pixels supply within-scene detail, but they do not create more independent scene assignments. Aggregate or model the hierarchy without miscounting the experimental units.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Olofsson and colleagues set out good practices for land-change accuracy assessment and area estimation. Roberts and colleagues explain why spatial structure must shape resampling and validation. Hemerik and Goeman develop exact testing under group invariance. The five-pair and scheme ledgers above are original teaching examples.
Continue through the Remote Sensing root, few-cluster inference and wild bootstrap, defensible reference labels and spatial validation and nearby pixels. The Gold Standard of Education and How X Works provide the wider route.
Top · Contents · Previous · How X Works