Find your next learning step
A randomised audit can estimate how workflow A performs across scenes and how workflow B performs across scenes. It still does not reveal which A outcome would pair with which B outcome for the same scene, and that missing pairing controls the distribution of individual scene effects.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the decision, or read from Chapter 1. The chapters remain open; these grouped routes are a navigable index.
Route 1 · Define the evidence
Route 2 · Work the ledgers
Route 3 · Transfer the method
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE EVIDENCE
1. Marginals are not a joint distribution
For scene i, the causal difference is Yi(A)−Yi(B). Assignment reveals one potential outcome. Across many scenes, randomisation identifies the marginal distributions of A and B, but not the dependence between the two potential outcomes within a scene.
Average effects avoid this problem because the difference of marginal means equals the mean of the individual differences. Variances, harm fractions and effect quantiles depend on the pairing and generally are only partially identified without extra assumptions.
CHAPTER 2 OF 12 · DEFINE THE EVIDENCE
2. Fréchet and Makarov logic
Fréchet–Hoeffding limits describe which joint distributions are compatible with two known marginals. Makarov-type formulas translate that compatibility into pointwise lower and upper bounds for the distribution of a sum or difference.
The bounds ask an honest worst-case question: among every within-scene pairing consistent with the observed marginal outcome distributions, how small or large could the probability of an effect below threshold d be? They do not choose a scientifically preferred pairing.
CHAPTER 3 OF 12 · WORK THE LEDGERS
3. Two outcome marginals
Consider four teaching scenes. Sorted accuracy scores under A and B are visible as marginals, while row pairings are intentionally absent.
| Order position | Workflow A | Workflow B |
|---|---|---|
| 1 | 70 | 68 |
| 2 | 75 | 74 |
| 3 | 80 | 81 |
| 4 | 85 | 83 |
| Mean | 77.5 | 76.5 |
The table must not be read as matched rows. It is only two ordered lists. Matching equal ranks is one dependence assumption, not a consequence of randomisation.
CHAPTER 4 OF 12 · WORK THE LEDGERS
4. Worked best-looking and worst-looking pairings
Pair equal ranks and the effects are +2, +1, −1 and +2. Pair the smallest A with the largest B, then continue in reverse order, and the effects are −13, −6, +6 and +17. Both pairings preserve exactly the same A and B marginals and the same +1 mean.
| Compatible pairing | Scene effects | Fraction below 0 |
|---|---|---|
| Equal-rank | +2, +1, −1, +2 | 25% |
| Reverse-rank | −13, −6, +6, +17 | 50% |
| Mean in either case | +1 | Not enough to recover harm |
Fréchet or Makarov bounds formalise the full set of compatible pairings rather than reporting only these two examples.
CHAPTER 5 OF 12 · WORK THE LEDGERS
5. Do not subtract marginal quantiles
The median of A minus the median of B is a contrast of two marginal locations. It is not generally the median of Yi(A)−Yi(B), because the scenes at the two medians need not be the same units.
Likewise, a quantile–quantile curve assumes a rank pairing. Rank invariance can be useful as a sensitivity scenario, but it must be named and compared with design-compatible bounds rather than presented as observed heterogeneity.
CHAPTER 6 OF 12 · TRANSFER THE METHOD
6. Interpret pointwise bounds carefully
A lower and upper bound at each effect threshold can be sharp pointwise: no tighter value is possible at that threshold using only the marginals. The functions that attain different pointwise limits need not be the same joint distribution everywhere.
Therefore do not stitch every favourable endpoint into one imagined effect distribution. Uniform confidence bands and finite-sample uncertainty add another layer beyond the population identification bounds.
CHAPTER 7 OF 12 · TRANSFER THE METHOD
7. Changed-context transfer: flood-map workflows
Suppose catchments are randomised between two flood-mapping pipelines. The marginals may show A has higher mean intersection-over-union, but decision-makers want the fraction of catchments harmed by more than five points.
Use pairing bounds at −5, retain the catchment as the assignment unit and repeat within declared biome or urbanisation blocks. Pixel-level pairing cannot be inferred when treatment was assigned by catchment.
CHAPTER 8 OF 12 · TRANSFER THE METHOD
8. Tighten bounds with defensible information
Pre-treatment covariates and randomisation blocks can tighten bounds by comparing more homogeneous scene groups. Panel outcomes, monotone response restrictions or scientifically justified dependence constraints can also help.
Every tightening assumption excludes compatible pairings. Show what it rules out and why. Narrowness purchased by an invisible copula choice is not stronger design evidence.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Connect the heterogeneity branch
The randomisation-compatible heterogeneity guide frames scene-risk decisions. The partial-identification guide links bounds to targeted audits.
Use robust decisions under accuracy sets when the action must remain defensible across every distribution in the identified set.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Limits and evidence checks
Pairing bounds can be wide with heterogeneous scenes, small blocks or coarse outcomes. That width is information: the audit does not support a detailed individual-effect story. Sampling uncertainty, reference-label error and missing scene coverage remain separate problems.
Archive assignment probabilities, scene frame, blocks, marginal outcome estimators, thresholds, bound formulas and confidence method. Report results under no dependence assumption and under each explicitly justified restriction.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Why is the mean effect identified?
Linearity makes the mean of paired differences equal the difference of marginal means; nonlinear features depend on pairing.
Are equal-rank pairings conservative?
Not generally. They impose positive rank dependence and can either narrow or move the harm distribution.
Can a larger audit identify the pairing?
It estimates marginals more precisely, but each scene still reveals only one potential outcome unless the design supplies additional within-unit information.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Fan and Park develop sharp bounds and inference for treatment-effect distributions. Firpo and Ridder discuss partial identification of effect distributions and functionals. Ding, Feller and Miratrix study randomisation inference for treatment-effect variation. The scene ledgers above are original teaching examples.
Continue through the Remote Sensing root, the Gold Standard of Education and How X Works.
Top · Contents · Previous · How X Works