Find your next learning step
An average accuracy gain can hide scenes that become worse. Effect quantiles ask how the workflow difference is distributed across scenes, but each scene reveals only one potential outcome, so honest analysis needs randomisation-compatible bounds.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Frame the problem
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · FRAME THE PROBLEM
1. Average effects and scene risk are different questions
If workflow A improves forest scenes by six points but harms wetland scenes by four, an overall mean can still be positive. A procurement decision may care about the average, the worst credible biome, or the proportion of scenes harmed more than a tolerance.
Effect quantiles describe the distribution of scene-level causal differences. They are not the quantiles of observed accuracy under A minus the quantiles under B. That subtraction compares two marginal distributions and generally does not recover the distribution of paired individual effects.
CHAPTER 2 OF 12 · FRAME THE PROBLEM
2. Why individual scene effects are not observed
For scene i, define Yi(A) and Yi(B), with effect τi=Yi(A)−Yi(B). A randomised audit assigns only one workflow to the scene, so only one potential outcome is observed. Randomisation identifies average effects readily, but the pairing of the two potential outcomes remains missing.
Bounds and tests for effect quantiles use the assignment mechanism and finite-population logic without pretending the missing pair is known. Additional rank or structural assumptions can tighten results, but those assumptions must be visible.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. A hidden scene-effect ledger
The following complete schedule is a teaching device. In a real audit one column per scene would remain unobserved.
| Scene | Accuracy under A | Accuracy under B | Effect A−B |
|---|---|---|---|
| F1 | 91 | 85 | +6 |
| F2 | 88 | 84 | +4 |
| C1 | 82 | 80 | +2 |
| C2 | 79 | 80 | −1 |
| W1 | 73 | 77 | −4 |
| W2 | 70 | 75 | −5 |
| U1 | 86 | 85 | +1 |
| U2 | 84 | 82 | +2 |
The mean effect is +0.625 points, but the lower quarter contains harmful effects. A decision based only on the mean misses that operational asymmetry.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Worked effect quantiles and a risk rule
Sort the eight effects: −5, −4, −1, +1, +2, +2, +4, +6. Using simple order statistics, the 25th percentile lies around −4 to −1, the median around +1 to +2, and the 75th percentile around +2 to +4.
| Summary | Teaching value | Decision reading |
|---|---|---|
| Mean | +0.625 | Small positive average |
| Lower quartile | Between −4 and −1 | A material harmed tail exists |
| Median | Between +1 and +2 | Typical scene improves modestly |
| Maximum | +6 | Best-case scene is not the policy target |
A risk rule might require a lower confidence bound for the 20th effect percentile to exceed −2 points. If the randomisation-compatible bound crosses −2, the audit does not support that safety claim even when the mean is positive.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Invert randomisation tests to bound quantiles
One route tests hypotheses about how many units can have effects above or below a threshold, then inverts those tests into simultaneous confidence bounds for the effect distribution. Stratified designs restrict permutations within the original scene blocks.
The result can be conservative because the unobserved pairing is adversarial within the allowed design. Conservative is not the same as useless: it distinguishes evidence supported by randomisation from a smooth but assumption-heavy estimate.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Connect quantiles to biome and loss functions
Effect quantiles across all scenes do not automatically protect every biome. Report block-specific effects or impose a decision loss that gives high consequence scenes suitable weight. Scene-weighted and area-weighted distributions answer different questions.
Choose the unit before analysis. Pixel-level quantiles can overwhelm small but important scenes and do not create pixel-level randomisation when assignment occurred by scene or region.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: flood-map workflows
Transfer the method to two flood-mapping workflows randomised across held-out catchments. The average intersection-over-union difference may be positive while steep urban catchments worsen.
A lower effect-quantile bound can support a risk statement about the fraction of catchments with unacceptable degradation. The transfer still needs common reference maps, matched assignment and a declared catchment or area weighting rule.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Connect quantiles to the randomisation branch
The weak-null guide tests an average effect while allowing scenes to differ. The confidence-set guide explains inversion under sharp effect hypotheses.
Use constrained biome-block randomisation to ensure the reference assignments match the actual audit design.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and misconceptions
Randomisation does not reveal every individual effect. Quantile bounds can be wide with few scenes, coarse assignments or weak overlap. Narrow model-based estimates may rely on assumptions that the design alone does not justify.
A lower-tail bound is not a guarantee for an unobserved future scene outside the audit frame. Reference-map error, missing scenes and post-selection of thresholds require separate treatment.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive the scene frame, assignment unit, blocks, probabilities, outcome definition, treatment labels, effect threshold, target quantiles and decision loss. State whether inference is simultaneous across thresholds or selected after inspection.
Report average effects, quantile bounds, attainable p-values, biome contrasts, alternative unit weights and the conclusion under nearby harm tolerances. Keep causal uncertainty separate from reference-label uncertainty.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Why not subtract the two accuracy quantiles?
Because units occupying the same rank under A and B need not be the same scenes. The individual-effect distribution depends on the missing pairing.
Are effect quantiles identified by randomisation?
Randomisation supports tests and bounds, often without point identification of the full distribution. Additional assumptions can narrow them.
Can one harmed scene overturn the decision?
Only if the decision rule assigns that type of harm enough consequence. The loss rule should be set before viewing the result.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Ding, Feller and Miratrix develop randomisation inference for treatment-effect variation and later decompose systematic and idiosyncratic variation. Su and Li give randomisation-based inference for treatment-effect quantiles in stratified experiments. The scene ledgers above are original teaching examples.
Continue through the Remote Sensing root, partial-identification and audit-sample guide and robust scene decisions under accuracy sets. The Gold Standard of Education and How X Works provide the wider route.
Top · Contents · Previous · How X Works