The Gold Standard · Experimental Design
Make the decision path visible
A provenance investigation can reconstruct several plausible stories. One timeline points to the approved preference-weight package; another fits a cached control file; both explain part of the output, and neither is fully proven. Choosing the more convenient story would turn missing evidence into false certainty.
The gold standard builds competing narratives from traceable evidence, tests their predictions, states reconstruction confidence and sends material disputes to an independent adjudicator with a defined authority and record.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · BUILD NARRATIVES
1. Define the exact disputed provenance claim
State which code, control set, input snapshot, environment or decision rule is contested and why the difference could matter.
Do not ask an adjudicator to choose the whole history when only one component is uncertain.
CHAPTER 2 OF 12 · BUILD NARRATIVES
2. Separate observed facts from interpretation
List immutable events, signed manifests, database histories, exports and physical records; label timing estimates, witness recollections and inferred dependencies separately.
A neat narrative can still be weak when every supporting item derives from one mutable source.
Evidence check
Check whether the scope, owner and decision consequence are explicit before continuing.
CHAPTER 3 OF 12 · BUILD NARRATIVES
3. Construct the smallest set of plausible narratives
Include each execution package consistent with material evidence and eliminate only those contradicted by reliable records.
Avoid multiplying speculative stories that make no distinct prediction about the decision.
| Layer | Example |
|---|---|
| Observed | Signed manifest |
| Inferred | Cache likely active |
| Unknown | Exact input snapshot |
| Material | Decision could change |
CHAPTER 4 OF 12 · TEST RIVALS
4. Derive testable predictions for every narrative
Predict intermediate values, row order, rounding, metadata, timestamps, warnings and side effects.
A narrative earns confidence by explaining evidence it was not invented merely to match.
CHAPTER 5 OF 12 · TEST RIVALS
5. Score evidence quality without false precision
Consider source integrity, independence, custody, temporal resolution, reproducibility and susceptibility to alteration. Use structured confidence bands rather than arbitrary percentages.
Document which evidence would change the assessment if later recovered.
Evidence check
Ask a second reviewer to reconstruct the example from the preserved evidence and stated assumptions.
CHAPTER 6 OF 12 · TEST RIVALS
6. Worked example: timestamps support different package histories
A workbook modification time suggests the old weight file, while a signed deployment manifest says the new package was active. A database extract contains an extreme-weight pattern unique to the new controls, but its export time is later than the decision.
Investigators test both packages against saved intermediates and find that only the new package reproduces row order and the extreme pattern; they rate that narrative best supported but not proven because input custody is incomplete. Changed-context check: if both packages produce the same material intermediates and decision, adjudication may record unresolved provenance without forcing a winner, while still repairing retention controls.
| Dimension | Question |
|---|---|
| Integrity | Could it be altered? |
| Independence | Same source? |
| Timing | Does it cover the run? |
| Reproduction | Does it predict details? |
CHAPTER 7 OF 12 · ADJUDICATE
7. Define reconstruction confidence in decision terms
State whether one narrative is uniquely supported, better supported, tied, or unresolvable, and whether alternatives change the conclusion.
Confidence concerns the provenance claim, not the investigator’s general confidence or the desirability of an outcome.
CHAPTER 8 OF 12 · ADJUDICATE
8. Use an independent adjudicator for material disputes
Choose reviewers outside the original build and decision chain, disclose conflicts, provide equal access to evidence and specify who has final authority.
The adjudicator should test reasoning and evidence, not merely vote between team preferences.
Evidence check
Verify that the control still reaches the intended decision under a changed but plausible context.
CHAPTER 9 OF 12 · ADJUDICATE
9. Record the adjudication and any dissent
Preserve the question, narratives, excluded alternatives, evidence weights, tests, limitations, decision and minority reasoning.
A later artifact may change the result; the record must make reconsideration possible.
| Finding | Statement |
|---|---|
| Unique match | Strongly supported |
| One better fit | Best supported |
| Material tie | Unresolved dispute |
| Same decision | Gap remains; outcome stable |
CHAPTER 10 OF 12 · REPAIR AND LEARN
10. Repair the material decision under a controlled package
Rerun the case with preserved inputs or bounded substitutes, compare outcomes across surviving narratives and issue a linked correction where needed.
Do not overwrite the original output or imply that adjudication reconstructed unavailable facts exactly.
CHAPTER 11 OF 12 · REPAIR AND LEARN
11. Close the evidence gaps that produced narrative conflict
Retain audit trails, bind exports to manifests, preserve input hashes, synchronise clocks and require portable package identifiers.
Fix the missing relation between data and metadata, not only the latest investigation file.
Evidence check
Confirm that versions, notices and unresolved limits remain visible in the final record.
CHAPTER 12 OF 12 · REPAIR AND LEARN
12. Run blind reconstruction and adjudication drills
Give an independent team a historical decision with selected evidence gaps; test whether it identifies alternatives, avoids confirmation bias and reaches a reproducible bounded conclusion.
The gold standard is honest reconstruction: the strongest supported account, visible uncertainty and a repair path when certainty is unavailable.
| Measure | Warning |
|---|---|
| Late conflict disclosure | Independence weak |
| Untested alternative | Confirmation bias |
| Missing dissent | Record incomplete |
| Nonreproducible ruling | Reasoning opaque |
Frequently asked questions
Must adjudication always select one narrative?
No. A tied or unresolvable result is valid when alternatives remain materially plausible.
Should confidence be expressed as a percentage?
Only when a defensible model supports it; otherwise use evidence-linked categories and describe what remains unknown.
Who counts as independent?
A reviewer outside the original build and decision chain, with disclosed conflicts and authority defined before review.
What if competing narratives produce the same decision?
Record the provenance gap and repair controls; reversal may be unnecessary when material intermediates and conclusions are stable.
Continue through verified routes
Core routes
Experimental Design subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Mixed-version provenance gaps, reconstruction and decision repair
- Version-reconciliation audit trails, reproducible cutovers and rollback
- Version divergence, parallel decisions and reconciliation
- Uncertainty-set misspecification and coverage diagnostics