Your validation set is next door

A model predicts something that varies across geography — yield, footfall, price, risk. You hold out a fifth of the rows at random, fit on the rest, and score R² 0.91. Then you point it at the next metro and it performs worse than the mean.

Nothing in the code was wrong. The validation design answered a question nobody was going to ask.

Holding out rows assumes the rows are interchangeable

Cross-validation estimates how a model will behave on data it has not seen by treating the held-out rows as a stand-in for that future data. The estimate is honest exactly to the degree that the held-out rows are independent of the ones used for fitting.

Spatial observations are not. Tobler’s first law — everything is related to everything else, but near things more than distant things — is normally quoted as the reason spatial prediction works at all. It is equally the reason spatial validation is hard. If nearby observations resemble each other, then a randomly chosen test point almost certainly has a training point a short distance away, and predicting it is not prediction. It is interpolation between values you were handed.

What the difference is worth

Ploton and colleagues put a number on this in Nature Communications in 2020. Working with large-scale ecological mapping models — the kind that get published with impressive held-out scores — they re-evaluated under spatial cross-validation and found that models reporting strong predictive performance under random splits showed close to none once the folds were spatially separated. The models had not been fitted badly. They had been graded on interpolation and deployed as prediction.

Roberts and colleagues had set out the methodology three years earlier in Ecography, in what remains the clearest treatment: any structure in the data — spatial, temporal, hierarchical, phylogenetic — creates dependence between folds, and blocking along that structure is what restores the meaning of the estimate.

How to tell whether it is happening to you

Three checks, cheapest first.

Measure the distance from each held-out point to its nearest training point. This takes one line and it is the most direct diagnostic there is. If the median of that distribution is much smaller than the distance over which your target actually varies, your holdout is scoring interpolation.

Fit an empirical variogram. The range — the separation beyond which semivariance flattens out — is the scale over which observations carry information about each other. Cressie is the standard reference. Any block smaller than the range is not really a block.

Compute Moran’s I on the residuals. If what the model got wrong is itself spatially clustered, there is structure left in the data that the model has not captured, and a random holdout will let it borrow that structure from its neighbours rather than predict it.

Blocking, and how big the blocks should be

The fix is to make the folds spatially separate. Partition the study area into contiguous blocks and assign whole blocks to folds, so that a held-out point’s neighbours are held out with it. Valavi and colleagues packaged the common strategies as blockCV, which is a reasonable place to start even if you are not in R.

Block size is the parameter that matters and the one most often chosen by eye. Blocks should be at least as large as the autocorrelation range; below that, the corners of adjacent blocks still leak into each other. Above it, you start trading away training data for nothing.

The stricter variant is buffered leave-one-out: hold out a single observation and exclude every training point within a radius of it. It is expensive and it wastes data, but it gives the cleanest reading of prediction at a chosen distance, which is occasionally the exact number you want.

The objection, which is a good one

Spatial cross-validation is not the settled answer, and pretending otherwise would be dishonest. Wadoux, Heuvelink, de Bruin and Brus argued in Ecological Modelling in 2021 that spatial cross-validation is the wrong tool for estimating map accuracy, and that it is systematically pessimistic: by forcing an artificial separation between training and validation locations, it evaluates the model under conditions that do not obtain when the map is used over the sampled region. Their alternative is design-based inference from a probability sample, which is unbiased and needs no modelling assumptions at all.

I think both papers are right, about different questions.

• How accurate is this map over the region I sampled? A probability sample answers this cleanly. Spatial cross-validation will understate it.
• How will this model do somewhere I have no data? Spatial cross-validation answers this. A probability sample over the old region cannot.
• Random cross-validation answers neither. It reports the accuracy of filling gaps between known points, which is a real quantity and almost never the one being asked about.

Meyer and Pebesma’s area-of-applicability work is the most useful reconciliation I know: rather than compressing the answer into one number, delineate where the model is operating within the range of what it was trained on and where it is extrapolating, and report accuracy only inside that region.

Why this shows up in commercial work

Property and market models are trained on the metros where the data is good and sold on the promise of working in the ones where it is not. A random split across a national training set will place held-out records on the same street as training records, and the resulting score describes a capability nobody is buying.

The structural point is the same one I made about grouped data: the holdout is only meaningful if it is genuinely independent of the training set. There, the group was an entity ID and the leak came from splitting by row. Here the group is a neighbourhood and it has no ID at all — membership is defined by distance, which is why the leak survives every check that looks for duplicate keys.

The number worth reporting is the one that answers the question you are actually going to be asked. Usually that question is about somewhere you have never been.

References

1. W. R. Tobler. “A Computer Movie Simulating Urban Growth in the Detroit Region.” Economic Geography, 46, 1970, pp. 234–240. The first law, in its original context.
2. D. R. Roberts et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, 40(8), 2017, pp. 913–929. The methodological reference.
3. P. Ploton et al. “Spatial validation reveals poor predictive performance of large-scale ecological mapping models.” Nature Communications, 11, 2020, article 4540.
4. A. M. J.-C. Wadoux, G. B. M. Heuvelink, S. de Bruin, D. J. Brus. “Spatial cross-validation is not the right way to evaluate map accuracy.” Ecological Modelling, 457, 2021, 109692. The counter-argument, and worth reading in full.
5. H. Meyer, E. Pebesma. “Predicting into unknown space? Estimating the area of applicability of spatial prediction models.” Methods in Ecology and Evolution, 12(9), 2021, pp. 1620–1633.
6. R. Valavi, J. Elith, J. J. Lahoz-Monfort, G. Guillera-Arroita. “blockCV: An R package for generating spatially or environmentally separated folds.” Methods in Ecology and Evolution, 10(2), 2019, pp. 225–232.
7. A. Brenning. “Spatial cross-validation and bootstrap for the assessment of prediction rules in remote sensing.” IGARSS, 2012.
8. N. Cressie. Statistics for Spatial Data, revised ed. Wiley, 1993. Variograms, ranges, and what they mean.
9. P. A. P. Moran. “Notes on Continuous Stochastic Phenomena.” Biometrika, 37(1/2), 1950, pp. 17–23.
10. P. Legendre. “Spatial autocorrelation: trouble or new paradigm?” Ecology, 74(6), 1993, pp. 1659–1673.