A satellite classifier can score perfectly on held-out pixels and still fail at a new site. Holding out a row in a data table does not necessarily hold out the landscape, image patch or local signature that helped the model make its prediction. The split must resemble the prediction task you want to trust.
This article uses a deliberately simplified four-site example with a fully specified classifier. Its 100% and 50% results are constructed calculations, not reported satellite performance or universal penalties for random splitting. The aim is to identify which claim each evaluation can support.
Find the question you need
Choose a route below or read in order. All teaching chapters remain open; the index groups let you jump directly to a worked example or check.
Route 1 · Define the place where predictions are needed
Chapters 1–2: Define the place where predictions are needed.
Route 2 · Work the four-site counterexample
Chapters 3–4: Work the four-site counterexample.
Route 3 · Read denominators and split units
Chapters 5–6: Read denominators and split units.
Route 4 · Build a test that matches deployment
Chapters 7–8: Build a test that matches deployment.
Route 5 · Audit, practise and continue
Chapters 9–12: Audit, practise and continue.
Full chapter index · Try the transfer checks · Remote Sensing subject root
Chapter contents
Define the place where predictions are needed · Chapters 1–2
Work the four-site counterexample · Chapters 3–4
Read denominators and split units · Chapters 5–6
Build a test that matches deployment · Chapters 7–8
Audit, practise and continue · Chapters 9–12
CHAPTER 1 OF 12 · Define the place where predictions are needed
1. Define the transfer claim before splitting pixels
There are several legitimate remote-sensing prediction tasks. One model fills unlabelled pixels among well-sampled fields in a known region. Another predicts an entirely unvisited catchment. A third must work on future images after weather, land management or sensor conditions have changed.
These tasks expose a model to different distances from its training evidence. A random pixel split can place a test pixel immediately beside a training pixel from the same field and acquisition. That may resemble local interpolation while saying little about transfer to a distant new region.
Write the intended prediction population in ordinary language first. For example: “classify land cover at fields in districts absent from model development, using only imagery available by the prediction date.” That statement tells you what must be kept out of training and tuning.
The subject root’s demand for independent relevant evidence becomes concrete here. Independence is not established by giving rows different labels such as train and test. Ask what common information remains across the boundary and whether the intended deployment would supply it.
CHAPTER 2 OF 12 · Define the place where predictions are needed
2. See how local similarity crosses a row-level split
Neighbouring pixels can share land cover, soil, illumination, atmospheric conditions and image-processing history. Image patches can overlap physically. Several labelled pixels may also inherit a class from the same reference polygon or site interpretation.
A model may exploit a local signature that is predictive within those sites but not portable elsewhere. It need not be handed the true test label directly. Dependence can make the evaluation easier because the training examples contain close analogues of the supposedly new test cases.
Reserve the strongest use of “leakage” for information crossing a boundary that the claimed deployment would not allow. Spatial dependence by itself is not proof of a flawed application. If future predictions are genuinely interspersed among a representative training sample, local similarity may be part of the intended task.
The practical question is therefore not whether a split is called random or spatial. It is whether the held-out predictions reproduce the information, geography and timing of the target use. Study the existing cross-validation mechanism guide for the wider training, tuning and test distinction.
CHAPTER 3 OF 12 · Work the four-site counterexample
3. Construct a four-site teaching dataset
Make four fictional sites A, B, C and D, each with one hundred labelled pixels. All pixels at A and B belong to class zero; all at C and D belong to class one. Every pixel also has a site signature that identifies its site. In this toy dataset there is no additional portable feature that predicts class at an unseen site.
Define the learning rule completely. For each site represented in training, remember its observed class. At prediction time, return that remembered class if the site was seen. Otherwise return class zero. This simple rule exposes the information boundary without needing a complex classifier.
We assume labels and site identities are correct, the class is constant within each site, and the rule never uses a test label. These artificial conditions deliberately make within-site prediction easy. They are not a claim that real landscapes have constant classes or that a realistic model must behave this way.
| Site | Pixel count | True class | Within-site training/test allocation |
|---|---|---|---|
| A | 100 | 0 | 80 / 20 |
| B | 100 | 0 | 80 / 20 |
| C | 100 | 1 | 80 / 20 |
| D | 100 | 1 | 80 / 20 |
There are four hundred pixel rows but only four site contexts. Creating additional pixels inside A does not create another independent example of transfer to a new site. Keep both counts visible before deciding what the performance estimate means.
CHAPTER 4 OF 12 · Work the four-site counterexample
4. Work the same-site pixel test
Assign eighty pixels from each site to training and twenty from each site to testing. Training contains all four site signatures and their classes. The memorising rule therefore predicts the correct class for all twenty test pixels at every site.
The held-out test contains eighty pixels, and all eighty predictions are correct. Pixel accuracy is 80/80 = 100%. No test labels were copied into training. Nonetheless, this experiment has tested new pixels at familiar sites rather than an unseen geographic context.
If the claimed use is transfer to unvisited sites, the result is insufficient. The rule used site-class information learned from other pixels at each test site. That information would be missing at the proposed new sites.
If the stated use is precisely to fill in pixels within these already represented sites, the same evaluation answers a narrower and potentially relevant question under our toy assumptions. The calculation is not numerically false. Its interpretation becomes false when the report silently expands the target population.
Now train on all pixels from B and C, and hold out all pixels from A and D. The classifier learns class zero at B and class one at C. It has no remembered class for A or D, so its specified fallback predicts zero at both.
At A, one hundred predictions are correct. At D, all one hundred are wrong. Whole-site holdout accuracy is 100/200 = 50%. The changed split removed the information that made the first evaluation easy.
The test-set confusion counts are: true zero predicted zero, 100; true zero predicted one, 0; true one predicted zero, 100; true one predicted one, 0. Class-one recall is zero. The overall 50% result needs this class account because the task may care especially about class one.
| Evaluation | Training sites represented | Test context | Correct / tested | Accuracy |
|---|---|---|---|---|
| Within-site pixel holdout | A, B, C, D | 20 pixels at each familiar site | 80 / 80 | 100% |
| Whole-site holdout | B, C | All pixels at unseen A and D | 100 / 200 | 50% |
| Leave D out | A, B, C | All pixels at unseen D | 0 / 100 | 0% |
The observed gap is fifty percentage points for our constructed rule and dataset. Do not convert that into a general correction factor for satellite models. Different features, sampling, landscapes and deployment questions can produce different gaps or little gap at all.
CHAPTER 6 OF 12 · Read denominators and split units
6. Count test contexts as well as pixels
Imagine copying each held-out pixel ten times. The accuracy is unchanged, and the number of evaluated rows increases tenfold. Nothing about the classifier’s performance at an additional independent site has been learned.
Closely related observations raise a similar issue without being literal copies. A pixel-level standard error based on independent trials can be misleading if errors move together within sites. The relevant uncertainty calculation needs a sampling model that represents that dependence.
Report the number of sites, scenes, dates, polygons and pixels involved, as appropriate. A dataset can be large at one level and small at another. With four sites, fold-to-fold differences are informative but cannot guarantee precise knowledge of performance across an entire region.
Keep aggregation explicit. A pixel-weighted average gives larger sampled sites more influence, while an equally weighted site mean answers a different question. For the statistical distinction, see equal sample sizes and different cluster precision. Do not invent an effective sample size from the pixel count alone.
CHAPTER 7 OF 12 · Build a test that matches deployment
7. Choose grouped, blocked or buffered splits for a reason
Group by site when the target is an unseen site and the site is the relevant shared context. Group by acquisition or image strip when a common observation process is the concern. Keep overlapping patches and their source observations in the same group when their overlap would reveal information across the evaluation boundary.
Spatial blocks divide a region into geographic units, some held out for evaluation. Buffers can remove training observations close to test observations. Neither label guarantees the right separation: adjacent blocks can still place training and test pixels a few metres apart at a boundary.
Choose distances using the intended deployment, known spatial structure and suitable diagnostics. A larger buffer can reduce short-distance dependence but also leave less training data and make the problem more extrapolative. That change should be acknowledged when interpreting lower performance.
There is no universal block size for all sensors, targets and regions. Explore a defensible range and show how the claim changes. A new-region task and a densely sampled local interpolation task may reasonably require different evaluations of the same model.
CHAPTER 8 OF 12 · Build a test that matches deployment
8. Keep preprocessing and tuning inside the boundary
A good geographic split can still leak information through model development. If labels from every region are used to choose features, thresholds or model settings before cross-validation, the held-out sites have already influenced the pipeline.
For an outer training/test split, fit data-dependent preprocessing and tune the model using only the outer training portion. Inner splits should also respect the grouping required by the task. Refit the selected pipeline on outer training data before predicting the untouched outer test group.
Some transformations are fixed by independent physical calibration or product documentation, rather than estimated from the evaluation dataset. Record that distinction. Do not call every correction leakage; identify which information it uses and whether that information would be available at real prediction time.
For future-date use, check the acquisition timeline too. A feature built with later observations can invalidate a forecasting claim even if its geographic split is clean. A valid spatial test cannot repair an unavailable future input or a reference label that describes the wrong date.
CHAPTER 9 OF 12 · Audit, practise and continue
9. Separate model transfer from map-area estimation
A blocked predictive test asks how a modelling process behaves under specified geographic separation. Estimating the accuracy or class area of a particular finished map is another job, often requiring a probability sampling design and suitable reference evidence.
Convenient held-out blocks are not automatically a representative sample of the mapped area. Conversely, a well-designed sample for estimating a fixed map’s area need not answer how a newly trained model transfers to a distant unsampled region. Explain which inference is intended.
Use stratified validation and area-adjusted estimates for unequal sampling fractions and paired validation of net change area for uncertainty in a temporal contrast. Those calculations require their own sampling assumptions; this spatial-split illustration does not replace them.
Reference errors remain possible after splitting. Check spatial alignment, label definitions and reference timing. A geographically distant validation point with an unreliable label is not automatically better evidence than a nearby point with a clear, task-matched interpretation.
CHAPTER 10 OF 12 · Audit, practise and continue
10. Audit the complete evidence path
Save the site or block assignments, random seeds where relevant, coordinates or reproducible geographic definitions, buffer rules and the dates of every input. Record how many training observations were removed by a buffer and whether every class remains learnable in each fold.
Inspect the closest training-to-test distances rather than relying on a coloured block diagram. Also inspect shared polygons, source scenes and duplicated patch footprints. Two nominally separate records may still represent the same underlying observation.
Preserve fold-level predictions and confusion counts. Show geographic variation alongside any pooled summary, and explain weighting. If one held-out district fails, a large successful district should not make that failure disappear from a report about dependable regional transfer.
Finally compare the validation geometry with actual prediction locations. A model can pass a carefully executed test that answers the wrong deployment question. The final evidence statement should name both the places checked and the places to which the conclusion is being transferred.
CHAPTER 11 OF 12 · Audit, practise and continue
11. Practise a new split and explain what changed
Check one: train the toy rule on A, B and C, then test only D. Every D pixel receives fallback zero and is wrong. Accuracy is zero. This does not contradict the earlier 50%: the test population now contains only the unseen class-one site.
Check two: use leave-one-site-out testing across all four sites with the same rule. Held-out A and B are predicted zero correctly; held-out C and D are predicted zero incorrectly. Pooled accuracy is 200/400 = 50%, with two successful and two failed sites.
Check three: suppose deployment is restricted to additional pixels within the four already labelled sites under the same stable signature assumptions. The 100% same-site result addresses that artificial task more directly than the distant-site holdout. The correct evaluation follows the intended use, not a preference for the lower score.
For transfer, design a split for a new-year flood classifier. Ask what must be separated by location, by flood event and by acquisition time. A student who can name those boundaries has learned more than the instruction to click a spatial-cross-validation option.
CHAPTER 12 OF 12 · Audit, practise and continue
12. Questions, connected reading and sources
Is a random split always wrong?
No. Its suitability depends on sampling and the intended prediction population. The problem is reporting local interpolation evidence as proof of unsupported geographic or temporal transfer.
Does spatial blocking make observations independent?
Not automatically. Dependence can cross block boundaries, and blocks can differ in class or environment. Check separation, target use and available training data instead of treating the word blocked as a guarantee.
Can I choose the buffer that gives the best test score?
That would tune the evaluation to the desired answer. Choose and justify the design from the task and evidence. If several plausible designs are explored, report their roles and keep final assessment independent of model selection.
Why keep a class confusion table?
A pooled percentage can conceal complete failure on the target class. Rare change and the reliability of mapped alerts develops the separate prevalence and denominator questions.
Choose the next question
Design and interpret validation evidence
- From Training Folds and Held-Out Predictions to Model Assessment, Hyperparameter Tuning, Leakage Control, Nested Evaluation and Better Generalisation
- Equal Sample Sizes, Different Cluster Precision
- Stratified Validation and Area-Adjusted Class Estimates
- Paired Validation and Uncertainty in a Net Change-Area Estimate
- Rare Change, False Alarms and the Reliability of a Mapped Alert
- A Shared Baseline and the Uncertainty in Comparing Two Changes
Check the observations behind each label
- Co-Registration Offsets and False Change at a Boundary
- Why Bilinear Resampling Can Invent Land-Cover Classes
- Why Averaging NDVI Differs from Averaging Reflectance
- Coverage Thresholds and the Footprint Behind a Regional Mean
- Cloud Masks and the Seasonal Population Behind a Clear-Sky Mean
- Revisit Gaps, Cloud Cover and Missed Short Events
- Emissivity and Brightness-Temperature Bias
Sources and scope
Roberts and colleagues’ cross-validation review explains evaluation with structured data. Andy Stock’s 2025 marine remote-sensing block study examines block choices and their relationship to prediction tasks. Mahoney and colleagues’ spatial cross-validation simulation study provides further comparisons. The four-site classifier and all percentages here are original, deliberately constructed examples.
Sources checked 22 October 2026. Continue through The Gold Standard of Remote Sensing, The Gold Standard of Education and How X Works to practise checking whether evidence survives a change of context.