GOLD STANDARD · RADAR · HOW X WORKS
Find the decision hidden inside the model
Reference truth is often too expensive or unavailable for every radar alarm. A two-phase design can choose a representative verification sample, while doubly robust calibration and coverage stress tests reveal which conclusions survive imperfect selection and outcome models.
This guide connects the Gold Standard of Radar, Gold Standard of Education, and How X Works hub.
Chapter contents and reading routes
Choose the route closest to your present task. Every chapter remains expanded, so the article also reads naturally from start to finish.
Orient two-phase truth
Design the sample
Calibrate robustly
Transfer and govern
Use evidence and next routes
CHAPTER 1 OF 12 · ORIENT TWO-PHASE TRUTH
1. Phase one sees alarms; phase two establishes truth
In phase one, the radar system records all eligible alarm opportunities, scores and operating covariates. In phase two, a selected subset receives expensive reference sensing, expert association or physical confirmation. The verified cases must represent the operational target through a known or estimable selection mechanism.
This design is stronger than verifying only convenient alarms because selection is part of the study. Preserve every phase-one row, its phase-two inclusion probability and whether truth was actually resolved. An unverified case is missing truth, not a false alarm.
CHAPTER 2 OF 12 · ORIENT TWO-PHASE TRUTH
2. Define the operational target before sampling
The target may be all alarm clusters, all scans in a specified envelope, or events under a set of covariate regimes. Range, weather, clutter, geometry and reference-sensor uptime determine whether the target and the verifiable population coincide.
Write inclusion and exclusion rules before looking at labels. If an outage stratum can never receive truth, no weighting method identifies its calibration without extra evidence. Coverage gaps need new data or explicit bounds.
CHAPTER 3 OF 12 · DESIGN THE SAMPLE
3. Choose phase-two strata that protect the tails
Sample across CFAR score, predicted uncertainty, range, clutter regime, weather and reference-sensor condition. Oversample rare or high-consequence combinations, then retain the probabilities required to return estimates to the target population.
A purely random subset may waste expensive truth on easy cases. A purely risk-based subset may have poor coverage. Two-phase design balances precision and representation by guaranteeing base coverage while enriching strata that drive tail calibration or operational decisions.
CHAPTER 4 OF 12 · DESIGN THE SAMPLE
4. Work through a two-phase truth sample
Suppose 4,000 alarm clusters enter phase one. The design verifies 400: 200 high-score clusters sampled at probability 0.40, 120 medium-score clusters at 0.15 and 80 low-score clusters at 0.05. Their inverse selection weights are 2.5, 6.67 and 20.
A naive verified-only reliability curve treats the three groups equally represented. Weighted calibration restores their operational influence, but the low-score stratum can become unstable because a few observations carry large weights. Precision and positivity must be checked before interpretation.
| Phase-one stratum | Phase-two sample | Selection probability | Inverse weight |
|---|---|---|---|
| High score | 200 | 0.40 | 2.50 |
| Medium score | 120 | 0.15 | 6.67 |
| Low score | 80 | 0.05 | 20.00 |
CHAPTER 5 OF 12 · DESIGN THE SAMPLE
5. Separate selection and outcome models
The selection model estimates the probability that a phase-one case receives resolved truth. The outcome model estimates truth or calibration conditional on observed radar and environmental features. Each model answers a different missing-data question.
Use variables observed for every phase-one case, and avoid features created only after verification. Diagnostics should test overlap, weight concentration and outcome-model residuals by the same regimes used for CFAR calibration.
CHAPTER 6 OF 12 · CALIBRATE AND STRESS
6. Use a doubly robust calibration estimator carefully
A doubly robust estimator combines inverse-probability weighting with an outcome-regression correction. Under its stated conditions, it can remain consistent if either the selection model or the outcome model is correctly specified, though not when both fail.
Doubly robust does not mean assumption-free. Extreme weights, unmeasured causes of verification and extrapolation beyond observed support can dominate the answer. Fit both models transparently and compare with weighted-only, regression-only and verified-only results.
CHAPTER 7 OF 12 · CALIBRATE AND STRESS
7. Stress coverage rather than trusting one curve
Recompute calibration after trimming or stabilising weights, altering selection-model features, restricting to shared support and perturbing unresolved strata within credible bounds. Also inspect range–clutter–weather cells where truth density is low.
A conclusion is robust when the operational decision remains stable across these checks. If a threshold passes overall calibration but fails in a sparse long-range regime, report the local limitation and collect more truth rather than averaging it away.
| Stress test | What changes | Warning sign |
|---|---|---|
| Weight cap sensitivity | Influence of rare verified cases | Decision depends on cap |
| Shared-support restriction | Removes extrapolation | Large target population excluded |
| Outcome-model alternatives | Tests regression dependence | Calibration curve moves materially |
| Coverage perturbation | Bounds unresolved truth | Threshold no longer acceptable |
CHAPTER 8 OF 12 · TRANSFER AND GOVERN
8. Changed-context transfer: medical test verification
Move from radar alarms to a diagnostic test where only a subset receives the definitive reference procedure. The stable structure is phase-one measurements, selective verification, inverse probabilities, outcome regression and target-population calibration.
The clinical loss and covariates differ, but the warning is identical: accuracy among verified cases is not automatically target accuracy. Transfer succeeds when the verification pathway and zero-probability strata are rebuilt.
CHAPTER 9 OF 12 · TRANSFER AND GOVERN
9. Design adaptive truth collection without losing inference
A later phase may target regimes where stress tests are weakest. Preserve the original probabilities and treat the adaptive additions as another sampling wave with documented selection. Pooling waves without design information can reintroduce the bias the study was built to avoid.
Reserve some random verification in every important regime. It supports surveillance for unmodelled failures and gives the selection model evidence outside the current uncertainty score.
CHAPTER 10 OF 12 · TRANSFER AND GOVERN
10. Limits and common misconceptions
Large phase-one data do not compensate for almost no truth in a consequential stratum. A doubly robust estimator can be biased when both working models are wrong, when verification depends on unobserved truth after conditioning, or when positivity fails.
Weight trimming trades variance for bias rather than eliminating a problem. Reference sensors also make errors and associations can be ambiguous. Include truth uncertainty and association error where they matter.
CHAPTER 11 OF 12 · CHECK AND CONTINUE
11. Questions radar teams often ask
Two-phase verification is useful when full truth is infeasible but phase-one covariates cover the target and a probability-based phase-two sample can be implemented. The sampling plan belongs in the calibration specification, not in an appendix added later.
How large should the truth sample be?
Size follows the required precision by regime, expected verification yield, weight variation and the decision margin. A single percentage of phase-one alarms is not a universal answer.
Does double robustness justify a complex outcome model?
No. Complexity can increase extrapolation and instability. Use defensible features, cross-fitted or held-out diagnostics where appropriate, and show the simpler estimators beside it.
CHAPTER 12 OF 12 · CHECK AND CONTINUE
12. Evidence, specialist routes and the next check
FDA guidance on partial and two-stage verification explains why naive verified-only calculations are biased. primary research on doubly robust ROC estimation under verification bias develops the correction, while research on efficient two-phase validation sampling shows how phase-two design affects precision.
Continue through reference-sensor coverage and verification bias, cluster calibration and truth uncertainty, and conditional coverage and calibration drift. The next check is to plot phase-two selection probability and calibration error on the same operating-regime map.