The Gold Standard · Experimental Design
Make the decision path visible
A mixed-version output is difficult enough when its fingerprints are visible. The harder case has missing logs, overwritten configuration or a spreadsheet export with no package identifier. The team knows that old and new components overlapped, but cannot immediately prove which combination produced a decision.
The gold standard reconstructs provenance from independent records, labels what remains uncertain and avoids inventing precision. Material cases are rerun under an approved package, original and repaired decisions remain linked and unresolved ambiguity becomes a governed limitation.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · SECURE EVIDENCE
1. Freeze the available evidence before investigating
Preserve files, logs, database snapshots, caches, emails, notebooks, reports and access records with dates and hashes.
Do not rerun or clean the system first; ordinary operations can overwrite the traces needed to reconstruct the original decision.
CHAPTER 2 OF 12 · SECURE EVIDENCE
2. Define the decision object and material question
Identify the subject, input data, output, decision rule, actor, time window and consequence.
The investigation does not need to reconstruct every system event. It needs enough provenance to determine which approved method could have produced the material decision.
Evidence check
Check whether the scope, owner and decision consequence are explicit before continuing.
CHAPTER 3 OF 12 · SECURE EVIDENCE
3. Rank provenance sources by reliability
Prefer immutable audit events, signed releases and database histories over filenames, memory or screenshots. Note clock quality, custody and whether records can be altered.
Use multiple independent sources. Agreement between weak copies does not equal one strong source if they share the same origin.
| Source | Typical strength |
|---|---|
| Immutable audit event | High |
| Release manifest | High |
| Database history | Context dependent |
| Filename or memory | Low |
CHAPTER 4 OF 12 · RECONSTRUCT THE RUN
4. Build an event timeline with explicit uncertainty
Place deployments, configuration changes, cache refreshes, runs, exports and decisions in time. Use ranges where clocks or timestamps are unreliable.
Separate known, inferred and unknown facts so later readers do not mistake a plausible story for a proven sequence.
CHAPTER 5 OF 12 · RECONSTRUCT THE RUN
5. Generate and test candidate execution packages
Enumerate feasible combinations of code, controls, inputs, environment and decision rules. Reproduce outputs and compare intermediate values, formatting and side effects.
Do not stop at the first matching final number. Different versions can converge accidentally while producing different behaviour elsewhere.
Evidence check
Ask a second reviewer to reconstruct the example from the preserved evidence and stated assumptions.
CHAPTER 6 OF 12 · RECONSTRUCT THE RUN
6. Worked example: a spreadsheet has outputs but no package identifier
Suppose an analyst saved weighted results during a migration week, then copied values into a decision workbook. The service logs expired, and the workbook records neither control version nor uncertainty-set release. Two candidate packages reproduce the headline total, but only Version 2 reproduces the extreme-weight pattern and row ordering.
The team records Version 2 as the best-supported reconstruction, labels remaining uncertainty and reruns the case using the approved complete Version 2 package. Changed-context check: if both versions reproduce every material intermediate and produce the same decision, provenance remains incomplete, but the repair may focus on documentation and control rather than reversing the outcome.
| Component | Evidence |
|---|---|
| Code | Build or commit |
| Controls | Version or checksum |
| Inputs | Snapshot or export |
| Decision rule | Effective record |
CHAPTER 7 OF 12 · REPAIR DECISIONS
7. Quantify residual provenance uncertainty
State which versions remain possible, what evidence distinguishes them and whether decision conclusions differ across candidates.
Do not assign arbitrary probabilities. Use bounded scenarios and clear confidence language tied to evidence.
CHAPTER 8 OF 12 · REPAIR DECISIONS
8. Rerun material cases under one approved package
Freeze the best available inputs, choose the controlling version from governance records and reproduce the decision with independent verification.
If original inputs are unavailable, state substitutions and test sensitivity rather than presenting the rerun as exact.
Evidence check
Verify that the control still reaches the intended decision under a changed but plausible context.
CHAPTER 9 OF 12 · REPAIR DECISIONS
9. Repair decisions without obscuring the original
Link the original output, reconstruction record, approved rerun, materiality review and corrected decision.
A correction should preserve who saw what and when. The audit trail grows; it does not erase inconvenient history.
| Finding | Action |
|---|---|
| One supported version | Verified rerun |
| Several same outcome | Document uncertainty |
| Several different outcomes | Reconsider decision |
| Inputs missing | Sensitivity and limitation |
CHAPTER 10 OF 12 · PREVENT RECURRENCE
10. Notify users according to consequence
Tell decision owners, participants and downstream systems which outputs are affected, what is known, what changed and which version now controls.
Avoid overstatement. An unresolved provenance gap is not proof that every result was wrong, but it is a reason to limit unsupported reliance.
CHAPTER 11 OF 12 · PREVENT RECURRENCE
11. Close the controls that allowed provenance loss
Require portable fingerprints, longer retention, immutable audit events, configuration capture, controlled exports and visible version display.
Target the actual loss path. More logging is not useful if the same job can overwrite or omit it.
Evidence check
Confirm that versions, notices and unresolved limits remain visible in the final record.
CHAPTER 12 OF 12 · PREVENT RECURRENCE
12. Test reconstruction readiness with drills
Select historical decisions, rebuild them from retained records and inject a missing-event scenario to see whether teams can bound and repair the result.
The gold standard treats provenance as operational evidence. A consequential method is trustworthy only when its decisions can be reconstructed honestly, including the limits of what is known.
| Measure | Question |
|---|---|
| Fingerprint coverage | Can runs identify themselves? |
| Retention | Do logs outlive decisions? |
| Rebuild success | Can outputs be reproduced? |
| Gap closure | Was the loss path fixed? |
Frequently asked questions
Can provenance be reconstructed without complete logs?
Sometimes. Independent manifests, databases, exports and reproducible intermediate patterns can bound the likely execution package.
Should investigators choose the first version that matches the output?
No. Test intermediate values, alternative candidates and whether the match is unique.
What if two versions produce the same decision?
Record the provenance gap and compare material intermediates; decision reversal may be unnecessary, but control repair still is.
May the original decision record be overwritten?
No. Preserve it and link the reconstruction, rerun and corrected decision as additional records.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Cutover drift, mixed-version outputs and continuous reconciliation
- Version-reconciliation audit trails and rollback
- Version divergence and parallel decisions
- Uncertainty-set misspecification and coverage diagnostics