Find your next learning step
Accuracy uncertainty matters most when it can change the action, so the next audit should target the decision—not merely the widest interval.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Translate accuracy into loss
Route 2 · Work the decision ledgers
Route 3 · Design and stop audits
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · TRANSLATE ACCURACY INTO LOSS
1. Accuracy uncertainty becomes a decision problem
A remote-sensing accuracy assessment often ends with estimated class accuracies and confidence intervals. A downstream user still has to act: monitor, commission an audit, protect an area or delay a release. When reference labels are imperfect, the admissible accuracy values may form a set rather than one trusted matrix.
Robust decision analysis asks which action performs acceptably across that set. It does not pretend the true accuracy is known, and it does not reduce every uncertainty to the widest possible worst case without examining consequences.
CHAPTER 2 OF 12 · TRANSLATE ACCURACY INTO LOSS
2. From accuracy sets to losses
Start with a set of confusion matrices or class-accuracy vectors consistent with the sample, response-design assumptions and reference-label uncertainty. Then define a loss for each action under each admissible state. Loss may include missed habitat, unnecessary field work, delayed response or false alarms.
Units must be explicit. Overall accuracy alone cannot determine the action because two maps with the same overall score can have very different omission error in the class that matters.
CHAPTER 3 OF 12 · WORK THE DECISION LEDGERS
3. An original three-action ledger
Suppose a habitat map supports a monitoring decision. Three admissible accuracy states survive the reference-data analysis. The following losses are hypothetical decision units, with lower values preferred.
| Action | Optimistic accuracy state | Central state | Pessimistic rare-habitat state | Worst loss |
|---|---|---|---|---|
| Monitor from the map | 4 | 12 | 35 | 35 |
| Run a targeted audit first | 8 | 10 | 18 | 18 |
| Protect immediately | 20 | 20 | 20 | 20 |
The table does not say audits are always best. It says that the preferred action depends on the admissible accuracy set, the loss scale and the cost of delay.
CHAPTER 4 OF 12 · WORK THE DECISION LEDGERS
4. Minimax loss and minimax regret
Minimax loss chooses the action with the smallest maximum loss. Minimax regret instead compares each action with the best action in each state, then minimises the largest shortfall. These rules answer different questions.
| Accuracy state | Best attainable loss | Monitor regret | Audit regret | Protect regret |
|---|---|---|---|---|
| Optimistic | 4 | 0 | 4 | 16 |
| Central | 10 | 2 | 0 | 10 |
| Pessimistic | 18 | 17 | 0 | 2 |
Use the rule that matches the institutional commitment. A safety-critical release may prioritise worst loss; a reversible choice may care more about regret or expected utility.
CHAPTER 5 OF 12 · WORK THE DECISION LEDGERS
5. Designing the targeted audit
An audit should sample where additional labels can change the decision, not merely where uncertainty is visually interesting. Candidate strata include rare mapped classes, disagreement zones, geographic domains with weak reference coverage and cells that drive the loss difference between actions.
Keep probability sampling when population-level accuracy or area inference is required. A purely convenience-based targeted sample can diagnose a boundary yet fail to support design-based estimates for the whole map.
CHAPTER 6 OF 12 · DESIGN AND STOP AUDITS
6. A decision-based stopping rule
Let D be the loss of monitoring minus the loss of auditing. If the current identified interval for D crosses zero, either action may be better. Continue sampling while a feasible audit can materially narrow that interval at acceptable cost.
A practical stopping condition is reached when one action dominates across the admissible set, or when the maximum plausible decision improvement is smaller than the cost of more reference data. This is a decision rule, not a universal statistical theorem; predeclare its loss scale, interval construction and audit cost.
CHAPTER 7 OF 12 · DESIGN AND STOP AUDITS
7. Changed-context transfer: regional deployment
Suppose a national accuracy set is narrow, but one wetland region has few reference observations and different seasonal appearance. A national minimax action can conceal a local vulnerability. Rebuild the admissible set by decision domain or use a hierarchical model that retains regional uncertainty.
The transfer check is to compare national and regional action rankings, then stress-test reference-label error. This connects partial identification and targeted audit samples with probabilistic reference labels.
CHAPTER 8 OF 12 · DESIGN AND STOP AUDITS
8. Value of information before sampling
Estimate what each candidate label can change. A point in a large, well-measured class may barely affect the action. A point in a rare class with high omission cost may reduce the worst loss substantially. Rank audit strata by expected or robust value of information per unit cost.
Do not calculate value of information from the same model being challenged without sensitivity analysis. Include alternative response-error assumptions and spatial clusters.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
Robust does not mean immune to all misspecification. If the true process lies outside the accuracy set, the guarantee does not apply. Extremely broad sets can also force a permanently conservative action that no realistic data would support.
Minimax is not automatically ethical or risk appropriate. Losses encode priorities and may distribute harms unevenly. Decision makers should inspect class-specific and regional consequences, not only the scalar maximum.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Retain the sampling design, inclusion probabilities, response protocol, disagreement records and the construction of every admissible accuracy state. Save class-specific area weights, loss functions, action constraints and audit costs.
Recompute decisions under alternative label-error bounds, spatial domains and loss ratios. Record when the selected action changes. For audit stopping, preserve the predeclared rule and all interim looks so that optional stopping is not hidden.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can confidence intervals be used as an accuracy set?
Sometimes, but a collection of separate intervals may include impossible joint combinations. Preserve dependence or use a simultaneous region when the decision depends on several entries.
Is the worst case always one corner of the set?
Only for suitable linear losses and convex sets. Nonlinear downstream models can place the worst case elsewhere.
Should the audit stop when overall accuracy is precise?
Not necessarily. Stop when the decision-relevant uncertainty is resolved or more information is not worth its cost.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Olofsson and colleagues set out good practices for probability sampling, response design and uncertainty reporting. Stehman examines key issues in rigorous land-cover accuracy assessment. Obradović develops decision methods under partial identification. The loss and regret ledgers above are original teaching examples.
Continue through the Remote Sensing root, partial identification, probabilistic reference labels and review queues and value of information. The Gold Standard of Education and How X Works provide the wider route.
Top · Contents · Previous · How X Works