The Gold Standard of Archaeology | Posterior Calibration, Coverage Tests and Synthetic-Scene Stress Tests

1. Define the object whose uncertainty is being calibrated

Back to contents

A tracker may estimate peak positions, complete paths, association decisions, birth and death times or the number of active tracks. These are not interchangeable targets. Pointwise intervals can cover peak location at each frequency while the joint path is wrong; a high-probability association can coexist with uncertain geometry. Before simulating, name the target, the unit of repetition and the rule for declaring coverage. For a complete archaeological interpretation, keep the computational target separate from the physical claim. Coverage of a spectral ridge is not coverage of a wall, void or occupation phase unless the measurement model justifying that translation is also tested.

2. Separate calibration, sharpness and archaeological accuracy

Back to contents

Calibration asks whether probabilities behave like their labels across repeated compatible cases. Sharpness asks whether uncertainty is concentrated. Accuracy asks how close estimates are to known truth under a chosen loss. A tracker can be calibrated but unhelpfully broad, or sharp and badly under-covering. It can also be computationally calibrated for a misspecified generative model while producing poor archaeological interpretations in the field. Report all three dimensions. Do not reward a method for narrow bands until coverage is established, and do not treat nominal coverage as proof that the synthetic scenes resemble the site.

3. Build synthetic scenes from an explicit generative story

Back to contents

Generate latent tracks first, then produce observations through the stages that create real ambiguity: finite array aperture, variable signal strength, peak merging, missed detections, false peaks, band limits and correlated noise. Retain the complete truth for every scene, including associations and truncated tracks. Vary numbers of sources, crossing angles, curvature, dwell time and clutter. A simulation that draws data from the fitted code with identical assumptions is useful for implementation checks but too friendly as the only stress test. Keep a second scene family with plausible departures from the analysis model.

4. Use repeated simulation without hiding Monte Carlo uncertainty

Back to contents

For each scene, fit the tracker exactly as it will be used, including pruning, tuning and convergence rules. Estimate coverage over many independent scenes and attach uncertainty to the estimated rate. With only twenty scenes, observing eighteen successes does not establish precise 90% coverage. Stratify scene generation in advance so difficult but plausible cases are represented rather than left to chance. Preserve random seeds and failed fits. Excluding non-convergent or empty-track cases after seeing results makes coverage look better precisely where the system is weakest.

5. Measure global and conditional coverage

Back to contents

Overall coverage can average away structural failures. Report results by event type: isolated ridges, crossings, births, deaths, low signal, band edges, peak mergers and clutter bursts. For continuous quantities, examine rank or posterior-quantile diagnostics in addition to interval coverage. For discrete associations, compare stated posterior mass with empirical correctness in probability bins, while retaining dependence among decisions. A model that assigns 0.8 probability to many hypotheses should be right about eight times in ten within the tested population, not merely on the easy subset.

6. Worked example: two crossing dispersion tracks

Back to contents

A team simulates 600 frequency–slowness scenes with two latent tracks, realistic dropout and recorded ground truth. The nominal 90% joint path region covers the full generating history in 86% of scenes. That summary seems close, but conditional coverage is 95% for well-separated tracks, 63% within two bins of a crossing and 71% when a track is born near the lower band edge. Posterior association probabilities above 0.9 are correct only 78% of the time at crossings. Expanding every interval would hide the mechanism. The team revises the merger and clutter model, then repeats the locked benchmark with new seeds and reports both versions.

7. Choose diagnostics that can see the relevant failure

Back to contents

Simulation-based calibration ranks are useful for scalar parameters and selected functions, but a tracker contains structured latent objects. Test path length, curvature, birth time, number of tracks, association entropy and log likelihood as separate quantities. A marginal position rank cannot detect every wrong joint dependency. Data-dependent quantities can expose problems that parameter-only summaries miss, yet many diagnostics create their own multiplicity and interpretation burden. Predefine a compact diagnostic suite and include plots, numerical thresholds and failure-handling rules rather than choosing the most reassuring view after simulation.

8. Changed-context transfer check: a magnetometer survey edge

Back to contents

Transfer the workflow from synthetic dispersion tracks to magnetic lineaments near the edge of a survey. The same steps apply: generate known latent features, simulate the measurement and processing chain, fit the reconstruction and test coverage. The changed context adds spatial truncation, ferrous clutter and uncertain background removal. A model calibrated on centred, complete lineaments may under-cover endpoints at the boundary. Edge-specific synthetic scenes and held-out field analogues are needed. Passing the original tracker benchmark cannot be carried over merely because both outputs are called paths.

9. Stress-test assumptions rather than only the code

Back to contents

After within-model calibration, vary noise tails, clutter rate, missed-detection dependence, source count, prior smoothness and observation resolution. Introduce features absent from the fitting model, such as brief splitting, nonstationary velocity or coherent artefacts. These tests do not estimate universal coverage; they map how rapidly confidence fails under plausible mismatch. Distinguish implementation error from model inadequacy. If coverage collapses only under an implausible extreme, say so. If a mild change produces severe undercoverage, widen claims, redesign the model or require a diagnostic gate before field interpretation.

10. Audit the simulation-to-claim chain

Back to contents

Archive the generative code, parameter distributions, scene labels, random seeds, tracker version, priors, pruning rules, convergence diagnostics, failed cases and scoring functions. Keep the locked benchmark separate from exploratory scenes. Report overall and conditional denominators, Monte Carlo intervals and every planned diagnostic. Link each tested condition to a field observable: signal-to-noise ratio, band edge, clutter density or path separation. An independent analyst should be able to regenerate the scenes, recover the known truth and verify that the public uncertainty claim matches the benchmark that was actually run.

11. Frequently asked questions

Back to contents

Does 90% coverage mean every real path has a 90% chance of being correct? No; it is a repeated-case property under the tested data-generating conditions. Is simulation-based calibration a posterior predictive check? No. They answer related but different questions about computation and model–data behaviour. Can one synthetic scene validate a tracker? No; coverage needs repeated known truths. Does passing within-model calibration prove field validity? No; the real acquisition process may differ. Should undercoverage always be fixed by wider intervals? No; structural failures at crossings or edges usually require a better model or narrower claim.

12. The gold standard makes posterior confidence earn its label

Back to contents

A posterior is useful when its probabilities survive an observable test. The gold standard defines the path object, generates varied known scenes, repeats the complete pipeline, measures conditional coverage and then perturbs the assumptions that matter in the field. It keeps calibration distinct from narrowness and from archaeological truth. When confidence fails, the benchmark shows where: crossings, births, edges, clutter or model mismatch. That turns uncertainty from decorative shading into a documented performance claim with clear limits.