The Gold Standard · Experimental Design
Make the decision path visible
Cutover drift begins after the formal release date. One notebook pins the new preference-weight uncertainty set, an API still serves the old controls and a dashboard combines both without showing their identifiers. Each component may look healthy, yet the resulting decision has no single reproducible method.
The gold standard makes the controlling version visible at every boundary, validates compatibility, detects mixed-version fingerprints continuously and defines how to quarantine, recompute and communicate affected outputs without overwriting their history.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · DEFINE CONTROL
1. Define the controlling version at the decision boundary
Record the method package, input snapshot, control totals, environment, decision rule and effective period that jointly define an output.
A release number alone is insufficient when current code can load old controls or cached inputs.
CHAPTER 2 OF 12 · DEFINE CONTROL
2. Give every run a portable version fingerprint
Emit identifiers and hashes for code, configuration, weights, uncertainty set, inputs and dependencies into reports, APIs and audit records.
The fingerprint should travel with the result so downstream users can detect incompatible combinations without guessing from dates.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · DEFINE CONTROL
3. Map compatibility before migration
List which old and new components can exchange outputs safely, which require transformation and which must never be mixed.
Distinguish representational compatibility from scientific equivalence. Two files may parse correctly while embodying different populations or loss assumptions.
| Component | Identifier |
|---|---|
| Code | Commit or package hash |
| Controls | Version and checksum |
| Inputs | Snapshot ID |
| Decision rule | Rule-set version |
CHAPTER 4 OF 12 · DETECT MIXING
4. Monitor for cutover drift continuously
Compare expected and observed fingerprints across pipelines, notebooks, dashboards, exports and decision stores. Alert on missing, stale or impossible combinations.
Do not stop monitoring after launch. Delayed jobs, cached services and manual workbooks often reveal drift days or weeks later.
CHAPTER 5 OF 12 · DETECT MIXING
5. Use cross-version invariants and decision comparisons
Check conserved totals, admissible regions, calibration summaries, extreme weights and decisions expected to remain stable.
An unexplained agreement can be as suspicious as a disagreement when both paths accidentally share a stale dependency.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · DETECT MIXING
6. Worked example: a dashboard joins Version 2 weights to Version 1 decisions
Suppose the new service publishes Version 2 weights, but a cached API continues returning Version 1 decision thresholds. The dashboard joins them because the schema is unchanged. A treatment option appears robust even though no approved version produces that combination.
Fingerprint reconciliation detects the mixed pair, quarantines the dashboard output and reruns affected cases using the complete Version 2 package. The original display and correction remain linked. Changed-context check: a planned bridge analysis may intentionally compare Version 1 thresholds with Version 2 weights, but it must use a separate analysis label, declared purpose and non-operational status.
| Pair | Treatment |
|---|---|
| New-new | Operational |
| Old-old | Historical or bounded |
| Old-new compatible | Validated bridge |
| Old-new incompatible | Quarantine |
CHAPTER 7 OF 12 · RECONCILE CONTINUOUSLY
7. Quarantine incompatible outputs before interpretation spreads
Stop downstream promotion, preserve the exact mixed output and identify all recipients and decisions that used it.
Quarantine is not deletion. Investigators need the original evidence to determine scope, cause and whether the discrepancy changed a decision.
CHAPTER 8 OF 12 · RECONCILE CONTINUOUSLY
8. Recompute material cases under an approved package
Choose the controlling version from the effective-date and governance record, then rerun affected cases with frozen inputs where possible.
Report both technical differences and decision consequences. Not every numeric change warrants reversal, but every material decision flip needs review.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · RECONCILE CONTINUOUSLY
9. Run reconciliation as an operational control
Schedule fingerprint scans, dependency inventories, sample reproductions and decision-difference reports throughout the migration window.
Assign owners and response times. Continuous reconciliation fails when alerts exist but nobody has authority to contain or repair mixed outputs.
| Signal | Action |
|---|---|
| Missing fingerprint | Stop promotion |
| Stale dependency | Refresh and rerun |
| Mixed pair | Quarantine and bound scope |
| Decision flip | Independent review |
CHAPTER 10 OF 12 · CLOSE THE CUTOVER
10. Close legacy paths deliberately
Disable unauthorised calls, expire caches, update templates, archive executable packages and document approved historical or research-only exceptions.
A legacy path that remains convenient will be reused. Make the safe current route easier and the old route visibly non-operational.
CHAPTER 11 OF 12 · CLOSE THE CUTOVER
11. Preserve audit trails through correction and rollback
Retain who changed what, old and new values, timestamps, reasons, approvals, affected intervals, notifications and replacement outputs.
Corrections should add records and links, not obscure the sequence of events that produced the mixed-version decision.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · CLOSE THE CUTOVER
12. Test that the cutover remains reproducible
Periodically sample decisions, rebuild them from retained packages, verify fingerprints at system boundaries and rehearse containment of an injected mixed-version result.
The gold standard treats migration as a monitored state, not a launch event. Continuous reconciliation ends only when every operational path identifies and reproduces the same controlling method.
| Measure | Question |
|---|---|
| Legacy calls | Are old paths still used? |
| Mixed outputs | Is drift declining? |
| Reproduction rate | Can decisions be rebuilt? |
| Repair latency | How fast are users corrected? |
Frequently asked questions
What is cutover drift?
It is the gradual emergence of mixed old and new components after a formal method transition.
Can two versions share the same output schema?
Yes, but technical compatibility does not prove scientific equivalence or permission to combine them.
Should mixed-version outputs be deleted?
No. Quarantine and preserve them so scope, cause and decision consequence can be reconstructed.
When is continuous reconciliation complete?
When every operational dependency is on an approved path, legacy use is bounded and sampled decisions reproduce from one controlling package.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Version-reconciliation audit trails, reproducible cutovers and rollback
- Version divergence, parallel decisions and reconciliation
- Uncertainty-set re-expansion after population change
- Uncertainty-set contraction and avoiding overconfident robustness