Find your next learning step
When imperfect reference labels cannot identify one true accuracy, a defensible range shows what the evidence supports and where the next audit has value.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Define the identified set
Route 2 · Work the bound ledger
Route 3 · Target and transfer audits
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE IDENTIFIED SET
1. When the honest answer is a range
Remote-sensing accuracy is often reported as one number even when reference labels are imperfect. Partial identification takes a different position: if the available assumptions do not determine one true accuracy, report the set of accuracies still compatible with the evidence.
This is not giving up. A defensible bound can rule out poor performance, expose overconfidence and show exactly what an additional audit must learn. It is often more useful than a precise estimate built on an unverified “ground truth”.
CHAPTER 2 OF 12 · DEFINE THE IDENTIFIED SET
2. From point identification to an identified set
A parameter is point identified when the observed data and assumptions imply one value. It is partially identified when they imply an interval or more general set. For map accuracy, uncertainty about reference-label error can turn one apparent agreement rate into a range of possible true agreement rates.
The width of the range belongs to the claim. Narrowing it requires evidence or stronger assumptions. A solver cannot manufacture identification by returning more decimal places.
CHAPTER 3 OF 12 · WORK THE BOUND LEDGER
3. A simple label-error bound
Suppose 100 sampled units compare a map label with a reference label. The two agree on 82 units. An independent quality review concludes that no more than five reference labels can be wrong, but cannot identify which five.
Each corrected reference label can change the map-reference agreement count by at most one. Therefore true map agreement can be as low as 77 or as high as 87. Under this bounded-error assumption, overall accuracy lies in [0.77, 0.87].
| Quantity | Observed or assumed value | Consequence |
|---|---|---|
| Sample size | 100 | One unit equals one percentage point |
| Apparent agreements | 82 | Apparent accuracy 0.82 |
| Maximum reference errors | 5 | At most five agreements can flip |
| Identified accuracy set | 0.77 to 0.87 | Honest bound before sampling uncertainty |
The bound is simple because it makes few claims about where errors occur. Class-specific information can tighten it, while correlated or spatially concentrated errors can make naive tightening unsafe.
CHAPTER 4 OF 12 · WORK THE BOUND LEDGER
4. Sharpness and worst-compatible cases
A bound is sharp when every value inside it can arise under the stated assumptions and observed data. To test sharpness, construct label configurations at the lower and upper endpoints. The lower case changes five apparent agreements into disagreements; the upper case changes five apparent disagreements into agreements.
If a proposed endpoint cannot be produced without violating class totals or known audit results, the bound can be tightened. Keep the construction or optimisation that attains each endpoint so readers can see what the range means.
CHAPTER 5 OF 12 · WORK THE BOUND LEDGER
5. Class accuracies need more structure
Overall accuracy can be bounded by counting flips, but producer’s and user’s accuracies contain uncertain denominators. An incorrect reference label can change both a numerator and a class total. Bounds should therefore be derived from the full error matrix, not by adding and subtracting five percentage points from every metric.
Stratified samples also require inclusion probabilities and area weights. The identified set for a sample agreement rate is not automatically the identified set for population area or class-specific accuracy.
CHAPTER 6 OF 12 · TARGET AND TRANSFER AUDITS
6. Targeted audits can narrow the useful cells
An audit budget should focus on labels that most affect the decision: rare classes, high-cost confusions, uncertain boundaries or cells that determine the widest bound. Auditing an observed agreement may lower the upper endpoint; auditing a disagreement may raise the lower endpoint.
| Audit choice | Likely information gain | Design risk |
|---|---|---|
| Random sample of all units | Broad, design-based update | Few rare errors reviewed |
| Oversample disagreements | Tightens uncertain confusion cells | Needs known inclusion probabilities |
| Review ambiguous boundaries | Tests difficult support | Cannot represent easy interiors alone |
| Independent field audit | Breaks shared-image dependence | Higher cost and temporal mismatch |
This links naturally to review queues, triage budgets and value of information: the best next label is the one expected to change a relevant decision or bound.
CHAPTER 7 OF 12 · TARGET AND TRANSFER AUDITS
7. Changed-context transfer: spatially clustered error
Suppose the five possible reference errors all lie near one wetland boundary. A bound based only on the total count remains valid for the sample, but a random-error model used to tighten it may fail. Nearby units share imagery, interpretation cues and geolocation problems.
The transfer check groups possible errors by spatial support and repeats the endpoint calculation under cluster-level constraints. Then validate on separated locations. This continues latent class reference models without pretending conditional independence has been established.
CHAPTER 8 OF 12 · TARGET AND TRANSFER AUDITS
8. Sampling uncertainty and identification uncertainty
The identified interval [0.77,0.87] describes ambiguity from imperfect labels in the observed sample. A probability sample adds sampling uncertainty because another sample would contain different units. These two layers should not be collapsed into one unexplained interval.
One approach forms confidence regions for moment inequalities or computes design-based limits at each admissible reference-error configuration. Report the assumptions and method rather than attaching a conventional standard error to the apparent accuracy.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
An upper bound on the number of reference errors must itself come from evidence. Choosing five because it gives a comfortable answer is not calibration. Bounds can also be wide when assumptions are weak; that accurately describes what the data do not determine.
Partial identification does not automatically reveal which map is better. Two products may have overlapping sets. A decision can still be robust if every admissible value leads to the same choice; otherwise, further audit may be valuable.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive the sample design, inclusion probabilities, spatial units, map and reference labels, class definitions and audit protocol. State every restriction used in the bound: total error cap, class-specific caps, dependence rules and temporal matching.
Save endpoint constructions, not only final numbers. Recompute bounds after each audit wave and predeclare the stopping rule—for example, when every admissible accuracy supports the same deployment decision.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Is the midpoint the best estimate?
Not without an additional model or loss rule. The midpoint is a summary of the interval, not an identified truth.
Can expert confidence scores replace an audit?
They can inform constraints if calibrated against independent outcomes. Uncalibrated confidence is not a verified error rate.
Should bounds always be made narrower?
Only with defensible evidence or assumptions. A narrow but false interval is worse than a wide, honest one.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Foody demonstrates how imperfect ground reference data can distort land-cover change accuracy in Remote Sensing of Environment. Olofsson and colleagues set out good practices for probability sampling, accuracy and area estimation. Obradović develops a formal partial-identification approach for imperfect reference tests. The five-label bound and audit ledger above are original teaching examples.
Continue through the Remote Sensing root, latent class reference models, probabilistic reference labels and reviewer adjudication. The Gold Standard of Education and How X Works connect bounds to transferable evidence.
Top · Contents · Previous · How X Works