The Gold Standard · Experimental Design
Make the decision path visible
After old and new preference-weight sets are compared, teams still need to move live decisions from one version to the other. A cutover can fail even when the new method is sound: the wrong controls may be loaded, one dashboard may keep old code or a repair may overwrite the evidence needed to reproduce a historical choice.
The gold standard packages every version, records who changed what and why, validates parallel outputs, maps dependencies and defines rollback before release so reconciliation remains an auditable scientific transition rather than a one-way software event.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · PACKAGE THE VERSIONS
1. Create an immutable release package
Bundle method specification, source code, environment, control totals, input snapshot, tests, uncertainty-set output and decision results.
Use a unique identifier that appears in every dependent report. A filename alone is not a reliable version contract.
CHAPTER 2 OF 12 · PACKAGE THE VERSIONS
2. Record a complete change rationale
State the defect or improvement, evidence, expected effect, affected population and authorised approver.
Separate methodological change from deployment configuration so later audits can explain whether a decision moved for scientific or operational reasons.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · PACKAGE THE VERSIONS
3. Preserve the prior version in executable form
Keep the software, dependencies, inputs and instructions needed to rerun representative results.
A PDF description is insufficient when implementation details determine trimming, convergence or feasibility.
| Artifact | Purpose |
|---|---|
| Method and code | Define computation |
| Inputs and controls | Define evidence |
| Tests and outputs | Validate result |
| Authority and date | Define governance |
CHAPTER 4 OF 12 · VALIDATE THE CUTOVER
4. Predefine acceptance criteria
Include conserved totals, calibration residuals, coverage, extreme weights, set geometry, decision-flip review, reproducibility and performance.
Acceptance should distinguish expected methodological change from unexplained discrepancy.
CHAPTER 5 OF 12 · VALIDATE THE CUTOVER
5. Run parallel and independent verification
Compare old and new versions on frozen inputs, then have an independent reviewer reproduce key outputs from the release package.
A parallel run executed by one misconfigured pipeline can make two outputs agree for the wrong reason.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · VALIDATE THE CUTOVER
6. Worked example: the new code uses last quarter’s controls
Suppose Version 2 passes unit tests and produces plausible weights, but one deployment loads last quarter’s population controls. A live dashboard shifts while the validation notebook does not. The audit trail links the dashboard run to the wrong control checksum.
The release is rolled back to Version 1, affected decisions are flagged and Version 2 is redeployed only after configuration validation. Changed-context check: if the old controls were intentionally required for a historical backcast, the same checksum difference is correct; the run purpose and effective period decide whether it is an error.
| Check | Pass condition |
|---|---|
| Invariants | Required totals conserved |
| Geometry | Expected set change explained |
| Decisions | Flips reviewed |
| Reproduction | Independent run matches |
CHAPTER 7 OF 12 · OPERATE SAFELY
7. Map every dependency before cutover
List data pipelines, notebooks, dashboards, reports, APIs, operational rules and archived reproductions that consume the set.
Assign owners and required version behaviour. An unowned spreadsheet can keep a superseded method alive long after formal transition.
CHAPTER 8 OF 12 · OPERATE SAFELY
8. Stage deployment with observable checkpoints
Use shadow output, limited users, a subset of decisions or a reversible effective date before full release.
Record checkpoints and discrepancies. Staging should generate evidence, not merely delay responsibility.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · OPERATE SAFELY
9. Define rollback triggers and recovery steps
Triggers may include failed invariants, unexplained decision flips, missing audit events, incompatible downstream output or inability to reproduce the release.
Name the last trusted package, data restoration path, communication owner and rules for decisions made during the failed interval.
| Dependency | Control |
|---|---|
| Pipeline | Pinned package ID |
| Dashboard | Version displayed |
| Report | Method citation |
| Archive | Executable retention |
CHAPTER 10 OF 12 · PRESERVE MEANING
10. Repair dependent outputs without overwriting history
Publish corrected results, link them to superseded versions and identify which decisions materially changed.
Do not erase the original record. Historical validity and current control can coexist when scope and effective dates are explicit.
CHAPTER 11 OF 12 · PRESERVE MEANING
11. Retire versions only after evidence and retention duties are met
Disable unauthorised new use, retain reproducible archives and document approved exceptions such as historical backcasts.
Retirement is different from deletion. Prior versions may be essential to explain research, policy or appeals.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · PRESERVE MEANING
12. Audit the cutover as a scientific result
Sample runs, verify hashes and logs, reproduce decision flips, inspect rollback readiness and test whether users can identify the controlling version.
The gold standard makes method change recoverable. A version that cannot be reproduced or rolled back cannot safely govern consequential decisions.
| Item | Purpose |
|---|---|
| Trigger | Start recovery |
| Trusted version | Restore control |
| Affected interval | Bound repair |
| Communication | Reach decision users |
Frequently asked questions
Is source-code versioning enough for reproducibility?
No. Inputs, controls, environment, configuration and decision rules also need immutable identifiers.
When should a cutover be rolled back?
When predefined integrity, reproducibility or decision-safety criteria fail and the discrepancy cannot be safely contained.
Should the old method be deleted after transition?
No. Retain an executable archive for historical reproduction while preventing unauthorised current use.
How are decisions during a failed cutover handled?
Bound the affected interval, rerun material cases under the trusted package and preserve both the original and corrected records.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Version divergence, parallel decisions and reconciliation after method change
- Uncertainty-set re-expansion after population change
- Uncertainty-set contraction and avoiding overconfident robustness
- Uncertainty-set misspecification and coverage diagnostics