The Gold Standard · Anthropology
Make the decision path visible
A linkage threshold is a decision rule, not a permanent fact. It turns a comparison score into an action: accept the pair as the same person, reject it, or send it for review. The rule may be defensible when it is set, yet become unreliable when the population, source systems or patterns of missing data change.
The difficult part is that a threshold can keep producing plausible-looking outputs while its error balance quietly moves. A growing mobile population may create more address disagreement. A new transliteration service may change name similarity. A digital form may reduce missing dates of birth while increasing copied contact details. None of these changes automatically proves that the threshold is wrong. They do mean that yesterday's validation cannot be treated as today's evidence.
Choose the route closest to your decision
Chapter contents
Chapters 1–3
Chapters 4–6
Chapters 7–9
Chapters 10–12
CHAPTER 1 OF 12 · DEFINE THE DRIFT
1. State what the threshold decides
Begin with the decision unit. A threshold may decide whether two rows are the same person, whether several records belong to one identity cluster, or whether a case can proceed without human review. Those are different decisions. A pairwise score of 0.92 does not carry the same meaning when it is used to join two records as when it helps merge a whole household cluster.
Write the action beside the threshold: auto-link, clerical review or reject. Then name the consequences of a wrong action. A false link can disclose information, transfer authority to the wrong person or contaminate later analysis. A missed link can fragment a history, duplicate a case or deny continuity. The acceptable balance depends on that purpose, not on a generic desire for a high match rate.
A sound specification therefore records the score, the cut-off, the eligible population, the variables used, the blocking rules and the date range. Without that bundle, later teams cannot tell whether the same threshold is being applied to the same problem.
CHAPTER 2 OF 12 · DEFINE THE DRIFT
2. Build a dated baseline
Drift can only be measured against a baseline that was honestly evaluated. Preserve a labelled comparison set from the period when the rule was accepted. It should include clear matches, clear nonmatches and difficult cases near the decision boundary. If only easy cases were labelled, the baseline will make every threshold look better than it is.
Report the baseline by meaningful slices: source system, language or script, age band, mobility pattern, record completeness and case type. The purpose is not to rank groups. It is to discover whether one overall error rate hides concentrated harm. Small slices need uncertainty intervals and careful suppression; a percentage calculated from five reviewed pairs is not a stable performance estimate.
Keep the old feature definitions and preprocessing code with the labels. Recomputing old scores with today's normalisation can erase the very change the audit is trying to observe.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 3 OF 12 · DEFINE THE DRIFT
3. Separate population drift from pipeline drift
Population drift means the mix of people or cases has changed. Pipeline drift means the machinery has changed: a parser, standardisation rule, candidate generator, comparison function, model or source feed. Both can move score distributions, but they call for different repairs.
A useful diagnostic table compares the old and new periods at three levels. First compare raw fields: missingness, value frequency, name length and address turnover. Then compare comparison features: exact agreement, edit distance and date tolerances. Finally compare scores and decisions. If raw fields are stable but score distributions move after a software release, inspect the pipeline. If the software is fixed but a new source population has different identifiers, examine transportability.
Do not label every shift as deterioration. Better data can move true matches upward and make the boundary safer. Drift is a prompt to re-evaluate, not proof that accuracy fell.
| Element | Question |
|---|---|
| Decision unit | Pair, cluster or case? |
| Action | Accept, review or reject? |
| Population | Which sources and dates are covered? |
| Version | Which features, code and cut-off apply? |
CHAPTER 4 OF 12 · MEASURE THE HARM
4. Refresh labels near the boundary
Random review is useful for estimating an overall rate, but threshold recalibration needs dense evidence near the current boundary. Sample cases just above and below the cut-off, plus cases from emerging populations and changed source systems. Oversampling difficult cases improves diagnosis; weighting is then needed when estimating population-wide rates.
Reviewers should see enough provenance to judge identity without being told the model score as a verdict. Use double review for ambiguous cases and adjudicate disagreements. A review label is not infallible: it is a recorded judgement supported by stated evidence.
When truth cannot be resolved, keep an indeterminate category. Forcing every case into match or nonmatch produces false certainty and teaches the calibration process from disputed labels.
CHAPTER 5 OF 12 · MEASURE THE HARM
5. Estimate errors at candidate thresholds
For each plausible threshold, estimate false-match and false-nonmatch rates with uncertainty intervals. Also report counts, because a small percentage can still mean many affected people in a large flow. Precision and recall are useful summaries, but they do not replace the underlying decision table.
The candidate-generation stage matters. A threshold audit performed only on pairs that passed blocking cannot reveal true matches that were never compared. Track a separate blocking recall study or use broadened candidate generation on a sample.
If review capacity is limited, evaluate two thresholds: one for automatic acceptance and one for automatic rejection. The interval between them becomes a review zone. The width should be justified by risk and capacity, not chosen merely to make a queue fit today's staffing.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 6 OF 12 · MEASURE THE HARM
6. Work a recalibration case
Suppose 10,000 candidate pairs were processed each month. At the old acceptance threshold, review finds 18 false links and 240 missed links. Six months later, after a new address source and a larger transient population, the same threshold produces an estimated 61 false links and 205 missed links. The total error count has not exploded, but the kind of error has changed sharply.
Raising the acceptance threshold reduces estimated false links to 24 but increases missed links to 330. Adding a review band catches 150 of the borderline pairs; after review, the estimate becomes 28 false links and 250 missed links. The correct choice depends on the consequences and the reliability of review, not simply on the smallest sum.
Changed-context transfer check: imagine the linkage authorises access to a sensitive community archive rather than deduplicating an analytical file. The cost of a false link is now much higher. The same numerical evidence supports a stricter automatic threshold and a wider human-review route. Transfer means reusing the reasoning, not copying the cut-off.
| Signal | Likely investigation |
|---|---|
| Raw-field shift | Population or source change |
| Feature shift only | Preprocessing or comparison change |
| Score shift after release | Software or model version |
| Stable scores, worse labels | Concept or ground-truth change |
CHAPTER 7 OF 12 · RECALIBRATE SAFELY
7. Choose a loss model openly
A threshold embodies a loss model whether or not anyone writes one down. Make the trade-off visible. One approach assigns relative costs to false links, missed links and review. Another sets a hard safety constraint, such as a maximum tolerable false-link rate, and chooses the best threshold within it.
Costs should be scenario-based and understandable: wrong disclosure, loss of service continuity, duplicated effort, delayed decision or irreversible record contamination. Avoid a single monetary value when it would hide rights, trust or distributional harm.
Run sensitivity analysis across a range of plausible costs. If one threshold remains preferred across the range, the decision is robust. If the preferred threshold flips easily, report the fragility and consider a review zone or a narrower use case.
CHAPTER 8 OF 12 · RECALIBRATE SAFELY
8. Recalibrate without rewriting history
Apply a new threshold prospectively unless there is a justified plan for historical review. Silently re-scoring old decisions can change identities, counts and access rights without notice. Version the rule and record an effective date.
For high-impact systems, identify past cases that would change under the new rule. Review a risk-based sample or all cases in a specified score band. Preserve the original decision, the new assessment and the reason for any correction. An audit trail should show what was known at each time.
Backfills need special care. A historical record was created under the data and rules then available. Re-linking may improve accuracy, but it can also import present-day identifiers into contexts where they never existed.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 9 OF 12 · RECALIBRATE SAFELY
9. Pilot, shadow and monitor
Before full rollout, run the candidate threshold in shadow mode beside the existing rule. Compare changed decisions, review workload and slice-level errors. Investigate large shifts before treating them as improvements.
Define monitoring triggers in advance: a sustained change in match rate, a jump in unresolved review cases, a new source system, a parser release, or a change in population coverage. Use control limits as prompts for investigation rather than automatic proof of failure.
A rollback plan should identify the last trusted model, feature pipeline and threshold. Rollback is not defeat; it is a controlled response when deployment evidence contradicts expectations.
| Option | False-link risk | Missed-link risk | Review load |
|---|---|---|---|
| Lower threshold | Higher | Lower | Variable |
| Higher threshold | Lower | Higher | Variable |
| Two-threshold review zone | Controlled | Controlled | Higher |
CHAPTER 10 OF 12 · GOVERN THE LIFECYCLE
10. Protect people and contested identities
People affected by linkage need a route to challenge a false association without being required to understand a similarity score. The notice should explain what record is disputed, what evidence can be supplied, who reviews it and how downstream copies will be corrected.
Names, kin relationships and addresses can be socially and historically complex. A difference may represent change, choice, transliteration, documentation practice or contested authority. Standardisation should support comparison without declaring one form the true identity.
Where records concern community representation or proxy authority, confirm that recalibration does not allow convenience identifiers to overrule current mandate evidence. Identity resolution and authority verification are connected but distinct decisions.
CHAPTER 11 OF 12 · GOVERN THE LIFECYCLE
11. Audit distribution and reviewer effects
Compare performance by source and context, but do not publish unstable small-cell rates as if they were precise. Use confidence intervals, pooled diagnostics and qualitative review to understand why a difference occurs.
Reviewers can drift too. Monitor disagreement, referral rates and decision patterns. Refresher cases should include near-boundary examples and cases involving changed names, shared addresses and sparse records. The objective is consistent reasoning, not mechanical uniformity.
If a group receives more review because its identifiers are less standardised, measure the delay and burden created by that policy. A technically safer review zone can still be unfair if the queue is inaccessible or chronically slow.
Practical check
Write the decision, evidence and limit in three separate sentences. If one sentence depends on an unstated assumption, name it before continuing.
CHAPTER 12 OF 12 · GOVERN THE LIFECYCLE
12. Know when recalibration is insufficient
A threshold cannot repair missing candidates, corrupted identifiers, unrepresentative training labels or a score that no longer ranks cases meaningfully. If true and false pairs heavily overlap after a source change, rebuild the comparison features or redesign collection.
Stop using automatic linkage when validation evidence is too weak for the consequence. Temporary manual review, restricted use or a narrower population can be safer than tuning a cut-off on inadequate labels.
The final record should state the current threshold, validation period, applicable population, error estimates, review policy, unresolved limits and next review trigger. That turns recalibration into a governed lifecycle rather than an emergency adjustment.
| Check | Pass condition |
|---|---|
| Label refresh | Boundary and new-source cases reviewed |
| Slice audit | No hidden concentrated harm |
| Shadow run | Changed decisions explained |
| Versioning | Effective date and rollback recorded |
Frequently asked questions
Is a falling match rate always evidence of threshold drift?
No. It may reflect real changes in mobility, coverage, missing identifiers or candidate generation. Diagnose the source before moving the cut-off.
Can we optimise one accuracy number?
A single number hides the difference between false links and missed links. Report both, with counts, uncertainty and consequences.
Should old records be re-linked after recalibration?
Only under a documented retrospective policy. Preserve the original decision and audit any changed linkage.
When should automatic linkage stop?
Pause or narrow it when labels are inadequate, candidate recall is unknown, or errors could cause high-impact irreversible harm.
Continue through verified routes
Core routes
Anthropology subject root · Gold Standard of Education · Gold Standard directory · How X Works hub
Specialist routes
- Cross-system linkage errors
- Proxy record provenance disputes
- Proxy record migration
- Correction disputes and annotations