Find your next learning step
A validation sample can support more than one class. Recording that uncertainty can improve honesty, but only when the response design, sampling weights and propagation rules remain explicit.
Open the chapter contents · Remote Sensing root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Define the response
Route 2 · Work the matrix ledger
Route 3 · Propagate and transfer
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE RESPONSE
1. A reference label can be a distribution
A land-cover sample may not belong cleanly to one visible class. Mixed pixels, seasonal change, image timing, occlusion and ambiguous legend rules can leave several defensible interpretations. Forcing every sample into one hard label hides that uncertainty and makes the accuracy matrix look more certain than the evidence.
A probabilistic reference label records a vector q=(q₁,…,qK), with non-negative entries that sum to one. The entries describe documented support for the reference classes under a declared response design. They are not automatically calibrated probabilities simply because decimals were written down.
CHAPTER 2 OF 12 · DEFINE THE RESPONSE
2. Response design comes before the numbers
The response design defines the spatial unit, temporal window, evidence sources, legend and decision rules used to obtain reference labels. Independent reviewers, adjudication and provenance remain essential. Probability vectors should be grounded in repeatable evidence such as reviewer votes, calibrated interpretation models or subpixel composition estimates.
This extends, rather than replaces, the guide to reviewer disagreement and defensible reference labels. A vector without an audit trail can be more misleading than an honest “uncertain” flag.
CHAPTER 3 OF 12 · WORK THE MATRIX LEDGER
3. Expected contributions to a confusion matrix
Suppose the map assigns sample s to class i and the reference vector is qₛ. Instead of adding one count to a single cell, add qₛj to each reference-class column j in row i. Each sample still contributes total mass one. Survey weights can be applied afterward according to the sampling design.
This expected matrix answers: what agreement would be credited if the reference uncertainty vector were treated as fractional membership or belief mass? It does not reveal the true unknown label, and it does not remove sampling variance.
CHAPTER 4 OF 12 · WORK THE MATRIX LEDGER
4. A four-sample worked ledger
Consider map classes Forest (F), Cropland (C) and Water (W). Four validation samples have equal design weight for this teaching example.
| Sample | Map class | Reference vector (F, C, W) | Matrix contribution |
|---|---|---|---|
| S1 | F | (0.90, 0.10, 0.00) | row F: 0.90, 0.10, 0.00 |
| S2 | F | (0.55, 0.45, 0.00) | row F: 0.55, 0.45, 0.00 |
| S3 | C | (0.20, 0.75, 0.05) | row C: 0.20, 0.75, 0.05 |
| S4 | W | (0.00, 0.10, 0.90) | row W: 0.00, 0.10, 0.90 |
The diagonal mass is 0.90+0.55+0.75+0.90=3.10, giving an expected agreement of 3.10/4=0.775. Hard adjudication could instead produce 1.00 if every maximum-probability class matches the map. The 0.225 difference is not necessarily map error; it reflects unresolved reference support that the hard score discarded.
CHAPTER 5 OF 12 · WORK THE MATRIX LEDGER
5. A response matrix for imperfect labelling
Another model uses a response matrix R, where R_ab is the probability that the response process records label b when latent class a is present. This is a measurement-error model for the reference process. If R is estimated from trusted repeats or expert panels, an observed accuracy matrix can sometimes be adjusted or analysed jointly with latent labels.
R must not be guessed from the same map being evaluated. Identifiability can be weak: several combinations of map error and reference error may explain the same observed table. Report the assumptions and uncertainty in R, not only a corrected point estimate.
CHAPTER 6 OF 12 · PROPAGATE AND TRANSFER
6. Propagating uncertainty
There are at least three uncertainty layers: sampling design, uncertain reference response and model or map variation. A defensible interval should state which layers it includes. For the teaching ledger, one can draw a reference class from each q vector many times, rebuild the matrix and examine the resulting accuracy distribution. With survey weights, resampling must respect strata or clusters.
| Layer | Example source | Suitable check |
|---|---|---|
| Sampling | finite stratified validation sample | design-based variance or design-respecting bootstrap |
| Response | ambiguous imagery or reviewer process | draws from q, repeated interpretation or response model |
| Map | training variation or temporal mismatch | refits, independent dates or model ensemble |
Monte Carlo draws propagate the declared q values; they do not validate them. Calibration needs samples with stronger evidence or repeated outcomes.
CHAPTER 7 OF 12 · PROPAGATE AND TRANSFER
7. Changed-context transfer: rare classes
Suppose Water is rare in the landscape but oversampled in validation. Equal weights no longer estimate population accuracy. Apply design weights based on inclusion probabilities or mapped-area strata, following the sampling plan. A few uncertain rare-class samples may then have large influence.
The transfer check is to compare unweighted and design-weighted expected matrices, then inspect effective sample size and influential samples by class. Probability vectors address response ambiguity; they do not cancel the need for probability sampling.
CHAPTER 8 OF 12 · PROPAGATE AND TRANSFER
8. Calibration and proper questions
Group samples by a stated reference probability, then compare with stronger follow-up evidence where available. If samples given 0.8 Forest support are later confirmed as Forest only about half the time under the target definition, the vectors are overconfident.
Calibration is class- and context-dependent. A reviewer may distinguish open water well but confuse shrub and cropland during a dry season. Preserve class-pair and season information rather than reporting one global confidence score.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
Fractional matrix entries do not create ground truth. They encode a response model or graded membership whose meaning must be explicit. Mixed composition, epistemic uncertainty and annotator disagreement are not identical; they may require different q construction.
Do not train and validate on the same uncertain labels without accounting for dependence. Do not estimate a response matrix from tiny cells and then invert it as if exact. Matrix correction can amplify noise when R is ill-conditioned. Keep negative or impossible corrected counts as a warning, not something to clip silently.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Practical evidence checklist
Archive the sampling frame, inclusion probabilities, spatial support, imagery dates, legend version, reviewer records, adjudication notes and q-generation rule. Check every vector sums to one and that missing evidence is not encoded as false certainty.
Publish the hard matrix, expected matrix, design-weighted estimates and sensitivity analysis under plausible alternative q values. Flag cells with small effective sample size. If a correction model is used, report R, its estimation sample, uncertainty, condition and validation.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Is q a class probability or a fraction of the pixel?
It can be either, but those meanings differ. State whether q represents belief about one latent class, physical class composition, reviewer proportions or another quantity. Do not combine them without a model.
Can expected accuracy be compared with ordinary accuracy?
Yes, as a sensitivity comparison when denominators and weights match. Explain that fractional agreement answers a different question from a hard adjudicated label.
Should uncertain samples be discarded?
Usually not automatically. Exclusion can bias the assessment toward easy homogeneous areas. Preserve them, improve the evidence where possible and show sensitivity to the response design.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Sources and next reading
Olofsson and colleagues’ good-practice framework grounds sampling, response design and analysis. Stehman and Foody discuss key issues in rigorous land-cover accuracy assessment. McRoberts examines imperfect reference data and class-proportion estimators. The worked vector ledger is an original teaching example.
Continue through the Remote Sensing root, review queues and value of information and reviewer disagreement and adjudication. The Gold Standard of Education and How X Works provide the wider transfer route.
Top · Contents · Previous · How X Works