The Gold Standard of Experimental Design | Detector Ensemble Dependence, Multiplicity and False-Discovery Control

1. Define the discovery and the error rate

Back to contents

State what a positive result means. Is the audit identifying contaminated benchmark items, contaminated datasets, clean items to retain, or models whose scores appear affected? The direction matters because false discoveries change meaning. Define the family over which error is controlled: all items in one benchmark, all model–benchmark pairs or every detector output examined. False discovery rate is the expected proportion of false positives among reported discoveries; it is not the chance that each flagged item is wrong and not a guarantee for one realised run. Pair the target error rate with a power goal and a minimum usable clean set.

2. Map dependence before combining detectors

Back to contents

Detectors can share token probabilities, prompts, models, reference corpora and calibration items. Perplexity and Min-K-style scores derived from the same likelihood vector will be strongly related. String overlap and near-duplicate search may also share preprocessing. Draw a dependency map identifying common inputs and transformations. Measure empirical correlations on negative controls, but do not treat low linear correlation as independence. Dependence can concentrate in the tail where decisions are made. The ensemble design should explain what distinct contamination mechanism each detector can see; adding several versions of one score usually increases apparent consensus more than genuine coverage.

3. Calibrate every base detector on named controls

Back to contents

A raw score is not a comparable probability. Build negative controls that are plausibly absent from training and positive controls with known exposure mechanisms, spanning length, domain, difficulty and model family. Estimate score distributions or conformal p-values under the null. Keep calibration separate from the items used to learn ensemble weights where the guarantee requires it. If a detector has no sensitivity to paraphrased exposure, state that limit instead of compensating with an optimistic threshold. Calibration drift across models or decoding settings should trigger recalibration. Retain control identities, construction dates and provenance so future audits can reproduce the claimed operating point.

4. Prespecify how evidence is fused

Back to contents

Fusion can average calibrated scores, combine p-values, vote at thresholds or learn weights. Each choice implies an evidence model. A maximum rule is sensitive but multiplies false-alarm opportunities. Majority voting can fail when detectors share the same blind spot. Learned weights can overfit a small control set and become dependent on the p-values they later combine. Use sample splitting, cross-fitting or a theory that accommodates adaptive weights when needed. Document normalisation, missing scores, clipping and tie handling. Compare the ensemble with every base detector so improved performance is demonstrated rather than assumed from complexity.

5. Control multiplicity at both detector and item levels

Back to contents

Searching many detectors, thresholds, prompt variants and benchmark slices creates a garden of forking alarms. Decide which searches belong to the confirmatory family before seeing outcomes. If each item receives one valid combined p-value, apply a false-discovery procedure suited to its dependence structure. If investigators first select the best detector and then report its p-values, the selection must be included in calibration. Dataset-level tests and item-level flags answer different questions and should not share one denominator. Report the number of hypotheses considered, including analyses that produced no discoveries. A nominal 5 percent threshold on each of hundreds of tests is not 5 percent family control.

6. Worked example: three detectors over 200 items

Back to contents

An audit runs a likelihood-tail score, a zlib-normalised score and a near-duplicate search on 200 benchmark items. The two likelihood detectors correlate at 0.92 on clean controls; duplicate search is largely distinct. Flagging an item when any detector crosses its 5 percent threshold yields 31 positives, many driven by one noisy likelihood tail. The team converts each score to a control-calibrated p-value, combines the two correlated likelihood signals as one prespecified family, then adds duplicate evidence using a held-out weighting rule. Applying false-discovery control to the 200 combined item p-values returns 14 discoveries. Positive controls show lower raw recall but much better realised precision. The audit reports both power loss and error control.

7. Use negative controls to test the entire ensemble

Back to contents

Run the finished pipeline—preprocessing, detector scoring, weight estimation, fusion and multiplicity correction—on benchmark-like negative controls. Counting false alarms only before fusion misses errors created by tuning and selection. Use repeated control batches when possible to examine realised false-discovery proportion and threshold variability. Controls should challenge shortcuts: vary text length, rarity, formatting and domain without introducing exposure. A detector that flags unusual but clean items needs recalibration or a narrower claim. Freeze the pipeline before applying it to the target benchmark, and record any post-hoc change as exploratory rather than silently updating the confirmatory result.

8. Changed-context transfer check: plagiarism screening

Back to contents

A school combines phrase matching, stylometry and AI-text detection to flag submitted essays. The transferable principles are the same: detectors share features, multiple students are tested and a flag is not a verdict. Calibrate on known authentic and copied work, define the family, preserve due process and keep human review separate from statistical screening. The context changes the harm: false discoveries can affect students directly, so conservative thresholds, explanation and appeal are essential. An ensemble that agrees because all tools penalise unusual vocabulary does not provide independent evidence of misconduct. The example shows why dependence is an ethical as well as statistical design concern.

9. Know what false-discovery control does not solve

Back to contents

Valid false-discovery control depends on valid p-values and the assumptions of the chosen procedure. It cannot detect contamination mechanisms absent from every base detector. It does not correct a biased control set, unknown benchmark provenance, selective publication or a model that exposes only coarse outputs. Average FDR control allows the realised false-discovery proportion to exceed the target in a particular run. Adaptive fusion theory may be asymptotic and weak in small samples. Report confidence or sensitivity where available. Do not translate a controlled list into certainty about individual items; downstream confirmation and provenance work remain necessary.

10. Audit the ensemble as one statistical instrument

Back to contents

Archive detector versions, prompts, model endpoints, score definitions, control sets, calibration splits, random seeds, weight-learning code, fusion rule, tested families, error target and every threshold considered. Re-run with each detector removed to reveal dominance and fragility. Compare independent, positively dependent and more conservative correction assumptions where relevant. Stress-test control-set shift and alternate sample splits. Publish the full item-level score matrix when licensing and privacy allow, not only the final flags. A dependency graph and analysis plan should make clear which choices were fixed before target outcomes became visible.

11. Frequently asked questions

Back to contents

Do three agreeing detectors mean three independent confirmations? Not when they share inputs or score construction. Is majority vote enough? No; voting needs calibration and can preserve shared bias. Does Benjamini–Hochberg always work? Its guarantee depends on valid p-values and an allowed dependence setting; choose the procedure deliberately. Can weights be learned on the audit data? That can invalidate calibration unless the design accounts for adaptivity. Does FDR control prove each flagged item is contaminated? No. It controls an average proportion across discoveries under assumptions; each item still needs careful interpretation.

12. The gold standard earns confidence at the ensemble level

Back to contents

A contamination ensemble should be treated as a single instrument assembled from dependent parts. The gold standard defines discoveries and families, calibrates base scores on credible controls, fuses evidence under an explicit dependence model, corrects multiplicity and validates the complete frozen pipeline. It reports both false-discovery control and power, along with mechanisms no detector can see. The result is not a louder vote from similar signals. It is a transparent, testable screening system whose confidence matches the information actually added.