Find your next learning step
A permutation is valid because it mirrors the design, not because software can shuffle labels. Biome blocks protect comparisons from mixing scenes that were never exchangeable in the first place.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Frame the problem
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · FRAME THE PROBLEM
1. Why unrestricted shuffling can answer the wrong question
Remote-sensing audits often span forests, croplands, wetlands and drylands. These biomes differ in class prevalence, seasonal behaviour, cloud contamination and sensor response. If workflow labels were assigned within biome, an unrestricted scene shuffle mixes assignments that the design never allowed.
Constrained randomisation keeps each label rearrangement inside its permitted block. The resulting p-value is design-sensitive: it measures how surprising the statistic is among assignments that could actually have occurred. Fewer permutations are a truthful consequence of the design, not a defect to hide.
CHAPTER 2 OF 12 · FRAME THE PROBLEM
2. Blocks encode assignment, not post-hoc convenience
A block is defined before outcomes are inspected and usually reflects the randomisation or sampling plan. Biome, sensor track, acquisition month or matched scene quality may form blocks. The valid choice is the factor that restricted assignment or supports a defensible exchangeability argument under the null.
Creating blocks after seeing performance can tune the reference distribution. A scientifically plausible biome label is not sufficient if assignment ignored it and observational confounding remains. Archive when, why and by whom each block was defined.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. A three-biome scene ledger
Consider twelve audit scenes: four forest, four cropland and four wetland. Within each biome, two scenes receive workflow A and two receive workflow B. The outcome is scene-level accuracy difference after a common reference process. The assignment count is six choices per biome, or 6³=216 constrained assignments.
| Biome | Scenes | A assignments | B assignments | Allowed within-block allocations |
|---|---|---|---|---|
| Forest | 4 | 2 | 2 | 6 |
| Cropland | 4 | 2 | 2 | 6 |
| Wetland | 4 | 2 | 2 | 6 |
| Combined | 12 | 6 | 6 | 216 |
An unrestricted six-versus-six shuffle would contain 924 allocations, most of which violate the design. Its larger reference set is not more informative; it answers a different hypothetical experiment.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. A weighted blocked statistic
Suppose the observed A-minus-B mean differences are +4 points in forest, +1 in cropland and −1 in wetland. An equal-biome statistic is (4+1−1)/3=1.33 points. An area-weighted statistic could differ if forest represents most of the target population.
| Biome | A−B difference | Target-area share | Area-weighted contribution |
|---|---|---|---|
| Forest | +4 | 0.50 | +2.00 |
| Cropland | +1 | 0.30 | +0.30 |
| Wetland | −1 | 0.20 | −0.20 |
| Total | — | 1.00 | +2.10 points |
For each of the 216 allowed assignments, recompute the same statistic from scene outcomes. The exact p-value counts assignments at least as extreme as observed, with the observed assignment included.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Granularity and unequal assignment probabilities
With only 216 equally likely assignments, attainable p-values are multiples of 1/216. Additional pixels inside a scene do not refine this randomisation grid. They improve the scene outcome measurement but do not create new scene assignments.
Some adaptive designs assign workflows with unequal probabilities. Then allocations should not be counted equally; the reference distribution uses their known design probabilities or an appropriate conditional test. Calling such a test “permutation” does not erase the weighting rule.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Blocking can increase or decrease power
Good blocks remove predictable biome variation from the comparison and can sharpen evidence. Too many tiny blocks may leave very few assignments, while weak blocks can add complexity without reducing noise. Power depends on outcome variation within blocks, allocation balance and the statistic.
Choose blocks for design validity first, then assess power prospectively. If an important biome has only one scene per workflow, show its pair difference and the coarse attainable inference rather than borrowing millions of pixels as independent replicates.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: post-fire mapping
Transfer the method to a post-fire burn-severity audit across low-, medium- and high-canopy strata. If algorithms are assigned within matched fire regions and canopy strata, preserve both restrictions. Region may be the true assignment unit even when each region contains many image tiles.
The changed context adds rare severe burns and unequal mapped area. Predefine whether the statistic weights strata equally, by area or by policy loss. Do not choose the most favourable weighting after observing the algorithm comparison.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connecting to the small-audit route
The scene-permutation guide establishes the design-based logic and paired sign-flip example. This article extends it to several biome blocks and explicit target-population weighting.
Use few-cluster inference and wild-bootstrap sensitivity when assignment probabilities are not known and inference rests on a clustered model. Use spatially correlated reference-error diagnostics to separate label dependence from assignment design.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and misconceptions
Blocking does not repair observational confounding merely by grouping similar scenes. Exactness applies to the stated randomisation design and sharp null. Reference-label error, biome misclassification, missing scenes and outcome multiplicity remain separate threats.
A non-significant blocked test is not proof of equal workflows. A significant result does not identify which biome benefits or how large the operational gain is. Multiple statistics, block definitions and subgroups require a predeclared analysis or multiplicity control.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive the scene frame, assignment unit, block variables, allocation ratios, assignment probabilities, random seed, exclusions, outcome aggregation and statistic. Preserve the observed allocation and enumerate all allowed allocations when feasible.
Report the number of assignments, attainable p-values, blockwise effects, target-population weights and leave-one-block-out sensitivity. Compare with an unrestricted shuffle only as a demonstration of design error, not as a competing valid result. Link inference to reference quality and area-estimation practice.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can biome be added as a block after the audit?
It may support a labelled sensitivity analysis, but it is not the original randomisation distribution unless the design or a prespecified exchangeability argument used it.
Do more pixels create more permutations?
No. The number of assignments comes from the scene or cluster design. Pixels refine outcomes within those units.
Should every biome have equal weight?
Only if the estimand values biomes equally. Area, population or decision-loss weights may be appropriate, but they must be justified before seeing the result.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Hemerik and Goeman develop exact testing under group invariance. Olofsson and colleagues set out good practices for land-change accuracy assessment and area estimation. Roberts and colleagues explain why spatial structure must shape resampling and validation. The biome ledgers above are original teaching examples.
Continue through the Remote Sensing root, scene permutations and exact tests and reviewer disagreement and defensible labels. The Gold Standard of Education and How X Works provide the wider route.
Top · Contents · Previous · How X Works