The Gold Standard of Experimental Design | Contamination Audit Sensitivity, Negative Controls and Detector Calibration

1. Define the contamination mechanism and audit estimand

Back to contents

Specify whether the concern is verbatim test-item exposure, answer-only exposure, paraphrased derivatives, task templates, benchmark discussion, supervised fine-tuning or repeated evaluation feedback. These mechanisms can change behaviour differently.

Define the quantity the audit targets: fraction of exposed items, score inflation, public-form advantage or probability that a model saw a specified record. A detector tuned to exact strings cannot support a claim about semantic exposure in general.

2. Separate detector score, decision rule and substantive claim

Back to contents

A detector may produce a similarity, likelihood gap, confidence difference or test statistic. The threshold converts that continuous score into a flag, while the audit claim interprets flags at benchmark level. Keep these steps distinct.

Choose thresholds on calibration data or by a predeclared error criterion, not by inspecting the unknown model until the result looks convincing. Report the score distribution and uncertainty even when a binary summary is required.

3. Build positive controls with known exposure

Back to contents

Create controlled-exposure models or fine-tuning conditions in which the exact items, paraphrases, answers or templates are inserted at known doses. Match training budget and legitimate data as closely as possible. Repeat across seeds because exposure effects can vary.

Positive controls should include mechanisms the audit claims to detect and realistic weak exposures, not only an extreme memorisation condition. If the detector misses planted exposure at the relevant dose, a clean verdict on an unknown model is not supported.

4. Build negative controls that share difficulty without exposure

Back to contents

Use fresh items or matched alternative forms that preserve task, topic, format and difficulty while remaining access-controlled. Include a model trained on the legitimate training split without test leakage. Negative controls reveal ordinary confidence and wording differences that can resemble contamination.

A mismatched easy control gives false reassurance, while a harder control can manufacture a public-form advantage. Document item construction and use human or model-independent checks to test equivalence without exposing the bank.

5. Estimate sensitivity, specificity and operating conditions

Back to contents

Sensitivity is the probability of a flag under the calibrated exposure mechanism; specificity is the probability of no flag under the clean control. Both depend on threshold, sample size, exposure dose, model family and item design. Report confidence intervals and the calibration domain.

Prevalence changes the meaning of a flag. Even a detector with useful sensitivity and specificity can have a low positive predictive value when contamination is rare. Avoid translating a flag rate directly into an exposed-item count without a validated measurement model.

6. Worked example: a promising detector with weak positive value

Back to contents

A detector is calibrated on 100 exposed and 100 clean items. At the chosen threshold it flags 72 exposed items and 18 clean items, giving estimated sensitivity 0.72 and specificity 0.82. Applied to a benchmark where only 10 percent of 1,000 items are contaminated, the same rates imply about 72 true flags and 162 false flags. Only about 31 percent of flags would be true under that prevalence.

The team reports the operating curve and benchmark-level interval instead of deleting every flagged item. It also tests a stricter threshold and increases the audit sample before drawing conclusions.

7. Use placebo labels and pipeline checks

Back to contents

Randomly permuted exposure labels, duplicated clean items and deterministic toy cases can verify scoring, pairing and significance code. A placebo should behave as expected under the null; otherwise the pipeline may contain leakage, indexing mistakes or a one-sided test chosen in the wrong direction.

Negative-control outcomes are useful when they share nuisance structure with the benchmark but cannot plausibly be affected by exposure. They test whether the audit is reacting to format, verbosity or model confidence rather than contamination.

8. Changed-context transfer check: a code benchmark

Back to contents

Move from multiple-choice questions to programming tasks. Exact text may be absent while canonical function names, tests or solution idioms were exposed. Paraphrasing the prompt can leave the code signature untouched, and fresh tasks may differ in library demands.

The transferable standard is mechanism-specific calibration. Plant exact prompts, signatures and semantically equivalent solutions separately; create clean controls with matched language and test complexity; then state which channel the detector can and cannot see.

9. Plan power before treating non-rejection as evidence

Back to contents

Use calibrated effect distributions to simulate the deployed test across sample sizes and contamination fractions. Predeclare the minimum exposure or score inflation the audit is designed to detect. Check whether calibration transports to the target model and benchmark.

A non-significant result with low power is not evidence of cleanliness. When the planned audit cannot distinguish a meaningful contamination level, abstain or report a wide bound rather than issuing a confident certificate.

10. Audit detector selection and multiplicity

Back to contents

Trying many prompts, transformations, thresholds and statistics creates another adaptive leaderboard. Register the primary detector and decision rule, retain exploratory analyses as exploratory and correct or hierarchically control multiple tests.

Archive calibration data provenance, control construction, model checkpoints, seeds, prompts, scoring code, thresholds and all attempted variants. Report detector failures and model families outside the validated range.

11. Frequently asked questions

Back to contents

Does a detector flag prove an item was seen? No; clean items can produce the same signal. Does no flag prove the benchmark is clean? No. Are paraphrases sufficient positive controls? Only for mechanisms they reproduce. Should contaminated-looking items be deleted automatically? Not without considering false positives and the benchmark estimand. Can one calibration transfer to every model? Usually not without evidence.

12. The gold standard calibrates the audit before judging the benchmark

Back to contents

A credible contamination audit defines the exposure mechanism, separates scores from claims, uses controlled positive and matched negative controls, reports operating characteristics, plans power and protects against adaptive detector selection. It makes abstention possible when calibration does not transport.

The objective is not a clean-or-dirty label. It is a bounded statement about which contamination effects the audit could have detected, which it could have missed and how those limits change the reported benchmark result.