GOLD STANDARD · PROBABILITY · HOW X WORKS
Find the decision hidden inside the model
A zero-truncated population estimate begins changing before the count model is fitted: it changes when a blocking rule decides which record pairs are even allowed to become links. Sensitivity across blocking schemes and duplicate constraints makes that hidden design choice visible.
This guide connects the Gold Standard of Probability, Gold Standard of Education, and How X Works hub.
Chapter contents and reading routes
Choose the route closest to your present task. Every chapter remains expanded, so the article also reads naturally from start to finish.
Orient the linkage design
Follow the worked comparison
Test totals and constraints
Transfer and govern
Use evidence and next routes
CHAPTER 1 OF 12 · ORIENT THE DECISION
1. Blocking defines the candidate universe
Two files with 10,000 records each contain 100 million possible pairs. A linkage system normally avoids scoring all of them by retaining candidates that agree on selected fields or appear near one another under an index. Every pair outside those blocks is treated as impossible unless another pass retrieves it.
Blocking is therefore a statistical design decision, not merely a speed trick. A narrow rule reduces computation and false candidates but can exclude true matches after spelling changes, moves or missing dates. A wide rule improves recall but creates more ambiguous competitors.
CHAPTER 2 OF 12 · ORIENT THE DECISION
2. Separate candidate generation from match classification
Candidate generation asks which pairs receive consideration. Classification asks how retained pairs are scored and accepted. Improving a classifier cannot recover a true pair that blocking removed, so the two stages need separate diagnostics.
Report candidate recall from an audit sample, candidates per record and results by missingness or name frequency. Then report score calibration and thresholds within the candidate set. One headline accuracy figure conceals where error entered.
CHAPTER 3 OF 12 · BUILD THE METHOD
3. A worked comparison of three blocking passes
Suppose two lists contain 1,000 and 800 records. Exact surname plus birth year yields 740 candidate pairs and recovers 210 audited true links. Adding a phonetic surname pass raises the pool to 1,160 and recovers 222. A postcode-neighbour pass creates 2,900 pairs and recovers 225.
The third pass finds three extra links at the cost of 1,740 additional candidates. Whether that is worthwhile depends on how those three overlaps move the unseen-zero estimate, not only on linkage recall. Preserve all three frequency tables and refit the population model.
| Blocking plan | Candidate pairs | True links recovered |
|---|---|---|
| Exact surname + birth year | 740 | 210 |
| Add phonetic surname | 1,160 | 222 |
| Add postcode neighbours | 2,900 | 225 |
CHAPTER 4 OF 12 · BUILD THE METHOD
4. Duplicate constraints encode what one unit can do
A one-to-one constraint permits each record in either file to appear in at most one accepted pair. It suits sources that should contain one record per person at the target time. A many-to-one rule may be necessary when one source contains episodes and the other contains people.
Applying one-to-one matching to episode data can discard valid repeats. Allowing many-to-many matching in a person register can inflate overlap through duplicates. State the unit, time window and within-file deduplication rule before solving the assignment.
CHAPTER 5 OF 12 · BUILD THE METHOD
5. Resolve competitors globally, not pair by pair
A record can have two plausible partners. Accepting each score independently may violate the intended constraint. A global assignment chooses a compatible set of links, but its answer can change when one blocking pass adds a new competitor.
Keep leading and runner-up scores, assignment margins and unresolved components. Sensitivity should compare whole compatible linkage structures, not flip isolated pairs while leaving impossible duplicates behind.
CHAPTER 6 OF 12 · TEST THE RESULT
6. Refit the zero-truncated model for every plausible table
Each linkage structure changes how many units appear once, twice or more. Those frequency counts are the data supplied to a zero-truncated GIGP model, so freezing one table while varying only the final overlap omits an important uncertainty path.
For each blocking and constraint scenario, rebuild the observed-count table, fit the same pre-specified family and retain diagnostics, unseen-zero estimates and intervals. A scenario that fits poorly should not be averaged in silently.
CHAPTER 7 OF 12 · TEST THE RESULT
7. Turn sensitivity into a decision table
Imagine the exact block yields a total of 1,540, the phonetic extension 1,485 and the postcode extension 1,472. The last expansion changes the answer little, whereas the first matters. Investigating spelling variants may therefore be worth more than a very broad candidate search.
Do not choose the scheme because it gives a preferred total. Choose using audit recall, precision, constraint validity and model fit, then show how the decision-relevant total responds.
| Scenario | Illustrative total | Interpretation |
|---|---|---|
| Exact, one-to-one | 1,540 | Possible missed links inflate unseen count |
| Phonetic, one-to-one | 1,485 | Material correction |
| Broad, one-to-one | 1,472 | Small further movement |
| Broad, many-to-many | 1,410 | Check duplicate legitimacy |
CHAPTER 8 OF 12 · TRANSFER AND GOVERN
8. Changed-context transfer: people to wildlife
In a camera-trap study, blocking may use location, date and coat pattern; duplicate constraints depend on whether repeated encounters are legitimate. The stable structure remains candidate generation, identity scoring, compatibility and an unseen population.
Transfer succeeds when the analyst redefines the unit and permissible repeats. Copying a person-register one-to-one constraint into repeated animal encounters would manufacture false nonmatches even if the scores were excellent.
CHAPTER 9 OF 12 · TRANSFER AND GOVERN
9. Audit pairs that blocking excludes
A clerical review drawn only from generated candidates cannot measure blocking misses. Sample from excluded pairs using wider search keys, known duplicates or a validation set with trusted identities. Oversample hard strata and use audit weights for population summaries.
Record why each true link was missed: changed surname, transposed date, missing postcode or frame error. The result should improve the next blocking pass and show whether one subgroup bears most error.
CHAPTER 10 OF 12 · TRANSFER AND GOVERN
10. Limits and misconceptions
More candidate pairs do not guarantee a better total. They can raise false links and create unstable assignment components. A globally optimal assignment is only optimal for its scores and constraints; it does not prove those ingredients are correct.
Blocking sensitivity addresses one layer. Source dependence, heterogeneous capture, migration, target definition and GIGP specification still matter. Keep linkage, capture and unseen-zero uncertainty distinguishable.
CHAPTER 11 OF 12 · CHECK AND CONTINUE
11. Questions analysts often ask
Should all schemes be averaged? Only if they represent defensible uncertainty and receive justified weights; otherwise present a structured range. Can duplicate rules be learned from data? Data reveal repeats, but permissible multiplicity depends on the real unit and process.
When is blocking wide enough? When validation supports adequate candidate recall across important strata and broader passes no longer change decisions enough to justify their error and cost. Preserve that stopping rule before inspecting the preferred estimate.
CHAPTER 12 OF 12 · CHECK AND CONTINUE
12. Evidence, reading routes and the next check
The probabilistic record-linkage primer explains the blocking trade-off and one-to-one setting. Improved Secondary Analysis of Linked Data separates cleaning, blocking, comparison and classification; multifile partitioning research treats linkage and duplicates together. Statistics Canada’s linkage-error models connect error to capture–recapture totals.
Continue with match-uncertainty propagation, dual-frame calibration, and population-total sensitivity. Recover true links from outside the current blocks, then refit rather than patch the final total.
Contents · Previous chapter · Return to the Probability root