Find your next learning step
A sharp null says every scene has zero effect. A weak null says the average effect is zero even though individual scenes may benefit or worsen. Those are different questions and need different randomisation logic.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Frame the problem
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · FRAME THE PROBLEM
1. Sharp and weak nulls are not interchangeable
Under a sharp no-effect null, workflow A and workflow B produce identical potential outcomes for every scene. That statement fills in all missing potential outcomes, so the assignment distribution of any statistic can be calculated exactly.
A weak null constrains only an average, such as the mean scene-level accuracy effect. It permits forest scenes to improve and wetland scenes to worsen. Missing potential outcomes are not determined, so an ordinary Fisher randomisation test with an unstudentised statistic need not have the intended size.
CHAPTER 2 OF 12 · FRAME THE PROBLEM
2. Studentisation stabilises unequal scene variation
A studentised statistic divides the estimated average effect by a design-consistent standard-error estimate. This adjustment matters when outcome variation or effect heterogeneity differs between assignment groups.
The resulting randomisation test remains exact for the compatible sharp null used to impute outcomes and can be asymptotically conservative for the weak average-effect null under suitable conditions. That is not finite-sample exactness for every heterogeneous design.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. A twelve-scene heterogeneity ledger
Suppose twelve scenes have workflow effects measured in accuracy points. The values are hidden in practice but useful for understanding the null.
| Scene stratum | Scene effects | Stratum mean | Pattern |
|---|---|---|---|
| Forest | +6, +4, +2, 0 | +3.0 | Mostly beneficial |
| Cropland | +3, +1, −1, −3 | 0.0 | Mixed |
| Wetland | −2, −4, −5, −1 | −3.0 | Mostly harmful |
| All scenes | 12 effects | 0.0 | Weak null true, sharp null false |
A sharp-null test targets a false statement in this example. A weak-null procedure asks whether the zero average is compatible with the randomised design while allowing the scene effects to differ.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Worked unstudentised and studentised statistics
Imagine one allowed assignment yields an A-minus-B mean difference of 2.4 points. The estimated standard error is 1.2, so the studentised statistic is 2.0. Another assignment yields a mean difference of 2.0 but standard error 0.6, producing a statistic of 3.33.
| Assignment | Mean difference | Estimated SE | Studentised value |
|---|---|---|---|
| Observed | 2.4 | 1.2 | 2.00 |
| Permutation P1 | 2.8 | 2.0 | 1.40 |
| Permutation P2 | 2.0 | 0.6 | 3.33 |
| Permutation P3 | −2.5 | 1.5 | −1.67 |
A permutation with a larger raw difference is not necessarily more extreme after accounting for its variability. The same studentisation rule must be recomputed for every allowed assignment.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Respect blocks, clusters and assignment probabilities
If workflows were assigned within biome blocks, permutations must stay within those blocks. If whole regions were assigned, the region is the unit for both the statistic and its variance. Pixel labels do not create extra assignments.
Unequal assignment probabilities require design weights. Studentisation cannot repair a reference distribution built from impossible shuffles. The design comes first; the statistic makes that design useful for the weak null.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Heterogeneity should be displayed, not averaged away
Report biome or scene-level contrasts alongside the overall weak-null test. A zero average can hide offsetting operational consequences, and a nonzero average can hide a harmed subgroup.
Predefine whether heterogeneity is descriptive, a formal interaction test or a decision-weighted estimand. Searching many subgroups and reporting only the most extreme one changes the error rate.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: post-fire algorithms
Transfer the workflow to two burn-severity algorithms assigned across matched fire regions. Region effects can vary with canopy, terrain and severity. A studentised blocked randomisation test can target the average region effect while preserving the matched assignment.
The transfer still needs common reference mapping and a declared outcome scale. Area-weighted and region-weighted averages answer different policy questions; studentisation does not choose the estimand.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connecting weak-null tests to earlier guides
The scene-permutation guide develops the sharp-null test. The confidence-set guide inverts sharp effect hypotheses. This article changes the target to an average effect under heterogeneity.
Use biome-block randomisation to ensure every shuffle matches the original assignment design before any studentised statistic is interpreted.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and misconceptions
Studentisation is not a universal small-sample cure. With very few clusters, unstable variance estimates and coarse assignment sets can still produce weak calibration. Asymptotic validity should not be described as finite-sample exactness.
A weak-null rejection does not prove every scene benefits. A failure to reject does not establish practical equivalence. Measurement error, missing scenes and reference-label uncertainty remain outside the assignment calculation unless explicitly incorporated.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive the assignment unit, blocks, probabilities, scene frame, outcome, estimand, variance estimator, studentisation rule, allowed allocations and random seed. State the sharp imputation null and the weak null of scientific interest.
Report raw and studentised statistics, attainable p-values, blockwise effects, influence of each scene or cluster, alternative variance estimators and policy conclusions under plausible weighting schemes. Separate design uncertainty from reference-data uncertainty.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Why not test the weak null with the ordinary mean difference?
Under heterogeneity and unequal variances, its permutation distribution may not approximate the weak-null sampling distribution. Studentisation addresses that mismatch under stated conditions.
Is the result exact?
It is exact for the sharp null used to complete potential outcomes. For the weak null, the guarantee is generally asymptotic and often conservative.
Can I test each biome separately?
Yes when prespecified and supported by the design, with multiplicity and small-block limitations reported.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Wu and Ding develop randomisation tests for weak null hypotheses using studentised statistics. Chung and Romano study exact and asymptotically robust permutation tests. Ding explains a paradox separating Fisher and Neyman randomisation inference. The scene ledgers above are original teaching examples.
Continue through the Remote Sensing root, randomisation confidence sets and design-sensitive p-values. The Gold Standard of Education and How X Works provide the wider route.
Top · Contents · Previous · How X Works