THE GOLD STANDARD OF INTELLIGENCE · SPECIALIST STUDY
Know which world your evidence can actually describe
Evidence can be accurate about the cases it contains and still mislead about the wider population. Collection access, survival, willingness to report, editorial filters and system visibility shape who enters the record. The gold standard makes that selection process explicit, tests missing pathways and narrows claims to the coverage the evidence can support.
This guide belongs to the Intelligence subject guide, the Gold Standard of Education, and the How X Works learning library. Choose a reading route, work the examples, then use the transfer checks to test whether the method survives a changed context.
Choose a reading route
Build the foundations · chapters 1–2
Apply the core method · chapters 3–5
Test evidence and alternatives · chapters 6–7
Handle limits and disagreement · chapters 8–9
Transfer and govern · chapters 10
The whole problem at a glance
| Layer | Question | Typical failure |
|---|---|---|
| Target population | Who or what should the judgement cover? | Scope left implicit |
| Sampling frame | Who could be selected or observed? | Coverage gaps hidden |
| Observed sample | Who actually produced usable evidence? | Nonresponse treated as random |
| Analytic record | What survived processing and reporting? | Editorial and survivorship filters ignored |
Accuracy inside the observed sample does not establish representative coverage.
CHAPTER 1 OF 10 · Build the foundations
1. Separate target population, frame, sample and record
Core idea. The target population is what the decision concerns; the frame is what collection can reach; the observed sample is what responded or was detected; the record is what survived processing.
Method. Draw all four layers and annotate exclusions, duplication, timing and responsible system. State whether the unit is a person, event, transaction, location or source.
Worked example. An analyst assesses small suppliers but the frame contains only registered firms using a digital portal. Informal and offline suppliers are absent before sampling begins.
Changed-context transfer. A school survey targets all pupils but reaches only those present and comfortable with the survey language.
Limits. Some hidden populations have no complete frame. The honest result may be bounded, qualitative or multiple-frame rather than falsely representative.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 1 is complete only when the claimed result survives that review.
Misconception to avoid. The dataset is the population. It is only the observed record produced by a process.
Practical evidence check
- Take one headline claim and trace every inclusion step from target to final table.
- Return to chapter contents
CHAPTER 2 OF 10 · Build the foundations
2. Distinguish coverage from source reliability
Core idea. A reliable source can report its accessible slice accurately while leaving most of the target unseen. Coverage asks where evidence could arise; reliability asks how well a source reports within that opportunity.
Method. Create a source-by-domain coverage map beside reliability assessments. Record access, detection threshold, time window, language, geography and groups unlikely to use the channel.
Worked example. A trusted maritime source sees large commercial ports but not informal coastal landings. High reliability does not extend coverage to the hidden coastline.
Changed-context transfer. A careful tutor’s notes describe attending students, not learners who stopped attending.
Limits. Coverage and reliability estimates can both be uncertain and correlated. Preserve ranges and unknowns.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 2 is complete only when the claimed result survives that review.
Misconception to avoid. A highly reliable source makes the evidence representative. Reliability cannot repair selection.
Practical evidence check
- Write two separate confidence statements: one for observation accuracy and one for reach.
- Return to chapter contents
CHAPTER 3 OF 10 · Apply the core method
3. Model selection mechanisms and incentives
Core idea. Cases enter evidence through mechanisms: random selection, convenience, gatekeeper access, voluntary response, detection, referral, payment, fear or strategic disclosure. The mechanism determines what comparisons are defensible.
Method. For each inclusion step, ask who controls it, what incentives operate and which variables predict inclusion and the outcome. Draw a causal selection diagram before weighting.
Worked example. Victims with severe loss may be more likely to report, while those fearing retaliation are less likely. A complaint archive can overstate one feature and understate another simultaneously.
Changed-context transfer. Course feedback may overrepresent very satisfied and very dissatisfied learners.
Limits. Selection variables are often partly unobserved. Sensitivity analysis is needed when correction rests on assumptions.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 3 is complete only when the claimed result survives that review.
Misconception to avoid. Self-selected evidence is useless. It can describe cases and mechanisms when claims stay within scope.
Practical evidence check
- List three variables that influence both inclusion and the judgement, then collect an independent indicator for each.
- Return to chapter contents
CHAPTER 4 OF 10 · Apply the core method
4. Recover missing exits and survivorship
Core idea. Survivorship bias arises when only units remaining at observation time appear. Failed projects, dissolved firms, silent sources, excluded cases and people who leave can change the conclusion.
Method. Define the entry cohort, follow every exit and classify loss, censoring and outcome. Search archives, administrative traces and independent frames for units absent from the final record.
Worked example. Studying current successful suppliers makes a screening process look effective because rejected and failed suppliers disappeared from the frame.
Changed-context transfer. Evaluating a tutoring programme only among finishers ignores pupils who withdrew after difficulty.
Limits. Exit records may be incomplete or ethically sensitive. Report bounds rather than invent outcomes.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 4 is complete only when the claimed result survives that review.
Misconception to avoid. A longitudinal dataset automatically solves survivorship. It does only if entry and attrition are traced.
Practical evidence check
- Rebuild one historic cohort and compare conclusions for starters, stayers and observed finishers.
- Return to chapter contents
CHAPTER 5 OF 10 · Apply the core method
5. Treat nonresponse and silence as data about the process
Core idea. Nonresponse is not a neutral blank when willingness or ability to respond is related to risk, burden, language, trust or outcome. Silence may signal access failure, fear, overload or non-occurrence.
Method. Track contact attempts, channel health, opportunity, refusal, inability, delay and partial response. Compare known characteristics and follow a subsample through another method.
Worked example. A hotline receives fewer reports after identity checks are introduced. The fall may reflect deterrence rather than less harm.
Changed-context transfer. Low homework-question volume may reflect clarity or fear of appearing weak; anonymous checks discriminate.
Limits. Follow-up itself can be selective, and protected groups require careful privacy design.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 5 is complete only when the claimed result survives that review.
Misconception to avoid. No response means no problem. It means the reporting pathway produced no usable report.
Practical evidence check
- For one silent group, write five mechanisms and one observable that would distinguish each.
- Return to chapter contents
Bias-mechanism matrix
| Mechanism | Observable clue | Corrective move |
|---|---|---|
| Undercoverage | Known groups absent from frame | Add or bound claims |
| Self-selection | Participation follows incentives | Model selection and triangulate |
| Survivorship | Failures disappear before observation | Reconstruct exits and losses |
| Reporting filter | Channels suppress some outcomes | Audit pipeline and negative returns |
Weighting cannot repair a group with no information and no defensible selection model.
CHAPTER 6 OF 10 · Test evidence and alternatives
6. Audit reporting, censoring and editorial filters
Core idea. Evidence can be selected after observation: field notes are summarised, incidents are thresholded, databases deduplicate, managers suppress bad news and publishers prefer dramatic results.
Method. Map the pipeline from raw observation to analytic record. Compare dropped, transformed and delayed items; preserve negative reports and reasons for exclusion.
Worked example. Local reports record minor disruptions, but a headquarters brief includes only events above a severity threshold. Trend claims based on briefs confuse editorial policy with reality.
Changed-context transfer. A school dashboard hides small categories for privacy; aggregation can also hide unequal outcomes.
Limits. Raw data are not automatically unbiased and may be unsafe to retain. Governance must balance auditability, privacy and source protection.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 6 is complete only when the claimed result survives that review.
Misconception to avoid. The final report reflects everything collectors knew. Each transformation can narrow it.
Practical evidence check
- Sample omitted records and explain whether each exclusion was required, arbitrary or outcome-related.
- Return to chapter contents
CHAPTER 7 OF 10 · Test evidence and alternatives
7. Triangulate frames, not just sources
Core idea. Adding sources helps only when their access and selection mechanisms differ. Ten outlets repeating one press release do not provide frame diversity.
Method. Trace provenance to original observations, cluster shared pathways and deliberately add a frame that reaches the main missing group or event type.
Worked example. Official inspections cover registered factories; worker clinics and satellite imagery reveal different subsets. Agreement across distinct frames is stronger than repetition inside one.
Changed-context transfer. Combine attendance records, anonymous pupil surveys and family interviews to cover different school experiences.
Limits. Different frames may measure different constructs and use incompatible time windows. Harmonise cautiously.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 7 is complete only when the claimed result survives that review.
Misconception to avoid. More sources guarantee less bias. Correlated access can reproduce the same blind spot.
Practical evidence check
- Draw dependencies for every source and identify the largest uncovered region after triangulation.
- Return to chapter contents
CHAPTER 8 OF 10 · Handle limits and disagreement
8. Use weighting and sensitivity without false precision
Core idea. Weights can adjust unequal known selection probabilities or calibrated response patterns. They cannot create observations for an entirely absent group or fix an unknown selection mechanism by arithmetic alone.
Method. Document inclusion probabilities, calibration variables, extreme-weight controls and design effects. Compare unweighted, weighted and bounded scenarios, including departures from missing-at-random assumptions.
Worked example. A survey weights age and district but lacks undocumented workers entirely. The weighted estimate still cannot represent that absent group.
Changed-context transfer. A classroom poll can weight grade levels, yet language exclusion needs new collection rather than heavier weights.
Limits. Small cells and extreme weights widen uncertainty. Model-based corrections depend on untestable assumptions.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 8 is complete only when the claimed result survives that review.
Misconception to avoid. A weighted result is unbiased. It is conditional on frame, variables and selection assumptions.
Practical evidence check
- Change response odds for the missing group and show when the decision reverses.
- Return to chapter contents
Coverage claim ladder
| Claim | Evidence needed | Safe wording |
|---|---|---|
| Case description | Verified case record | In the observed cases |
| Frame estimate | Known frame and selection | Among frame members under this design |
| Population estimate | Coverage and adjustment evidence | Estimated for the target population |
| Causal claim | Identification assumptions and design | Effect under stated assumptions |
A larger dataset does not automatically move a claim up the ladder.
CHAPTER 9 OF 10 · Handle limits and disagreement
9. Communicate coverage gaps beside the judgement
Core idea. Decision-makers often read a precise percentage as universal. Coverage, time, source access and missing pathways must sit beside the conclusion, not in an appendix.
Method. Use a coverage statement: target, frame, observed share, known exclusions, likely direction of bias, uncertainty, decision implication and next collection action.
Worked example. “Among registered portal users responding this month, 62% reported delay; informal suppliers and nonresponders are not represented, so population prevalence is unresolved.”
Changed-context transfer. For a school, say which year groups and attendance states the wellbeing estimate covers.
Limits. Long caveats can obscure action. Lead with the exclusion that could change the decision and link deeper evidence.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 9 is complete only when the claimed result survives that review.
Misconception to avoid. Transparent limits make analysis useless. They make the supported action clearer.
Practical evidence check
- Give the result to a new reader and ask them to name exactly who remains outside the claim.
- Return to chapter contents
CHAPTER 10 OF 10 · Transfer and govern
10. Govern evidence coverage as a learning system
Core idea. Coverage changes when platforms, conflict, policy, language, trust or technology changes. A frame that was adequate last year can become systematically incomplete.
Method. Assign owners to frames and selection models, monitor inclusion rates and attrition, run independent coverage audits, protect vulnerable groups and trigger redesign when gaps cross decision thresholds.
Worked example. A programme adds mobile reporting; participation rises in connected areas but falls among shared-phone users. Governance treats this as a frame change, not pure growth.
Changed-context transfer. Transfer the audit to recruitment, medical screening or school admissions.
Limits. Coverage monitoring can itself expose identities or incentivise gaming. Use proportional, protected metrics.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 10 is complete only when the claimed result survives that review.
Misconception to avoid. A stable sample size means stable coverage. Composition can change while counts remain constant.
Practical evidence check
- Audit one conclusion across two periods and explain every frame, response and pipeline change.
- Return to chapter contents
Build an evidence-coverage register
Name the target population, unit, frame owner, access channel and time window.
Record known exclusions, duplicate pathways, response states and transformations.
Link each gap to the claim it limits and to a discriminating collection action.
Review after operational, policy or technology change.
Build a decision record
Choose one consequential workflow and state the decision, the people affected, the time boundary and the claim being tested. Record original evidence before interpretation so later review can distinguish source material from inference.
Apply the article’s method to one routine case and one case near a boundary. Show the calculation or rule path, identify missing information and explain why plausible alternatives were accepted or rejected.
Change one material assumption at a time. Record what remains stable, what reverses and what becomes unresolved. Missing evidence must remain visible rather than being converted into a convenient zero, default or certainty.
Compare the result with at least one simpler alternative: explicit-only reading, an unweighted description, a transparent rule or a basic case-flow count, as appropriate. If the complex method does not improve the decision or explanation, prefer the simpler method. Record both outputs so later reviewers can see what the added structure contributed and what it obscured.
Invite a reviewer who was not involved in design to challenge scope, definitions, denominators, boundary conditions and incentives. Ask what evidence is systematically absent, who bears the cost of error, and how the method could be gamed. Convert each credible challenge into a test, collection action or explicit unresolved limitation rather than a defensive footnote.
Assign an owner, correction route, retention rule and review date. Verify that correction reaches derived outputs and future decisions, then retire the method when its assumptions, authority or evidence base no longer hold.
Run a boundary-case audit
Choose one consequential workflow and state the decision, the people affected, the time boundary and the claim being tested. Record original evidence before interpretation so later review can distinguish source material from inference.
Apply the article’s method to one routine case and one case near a boundary. Show the calculation or rule path, identify missing information and explain why plausible alternatives were accepted or rejected.
Change one material assumption at a time. Record what remains stable, what reverses and what becomes unresolved. Missing evidence must remain visible rather than being converted into a convenient zero, default or certainty.
Compare the result with at least one simpler alternative: explicit-only reading, an unweighted description, a transparent rule or a basic case-flow count, as appropriate. If the complex method does not improve the decision or explanation, prefer the simpler method. Record both outputs so later reviewers can see what the added structure contributed and what it obscured.
Invite a reviewer who was not involved in design to challenge scope, definitions, denominators, boundary conditions and incentives. Ask what evidence is systematically absent, who bears the cost of error, and how the method could be gamed. Convert each credible challenge into a test, collection action or explicit unresolved limitation rather than a defensive footnote.
Assign an owner, correction route, retention rule and review date. Verify that correction reaches derived outputs and future decisions, then retire the method when its assumptions, authority or evidence base no longer hold.
Test independent reproduction
Choose one consequential workflow and state the decision, the people affected, the time boundary and the claim being tested. Record original evidence before interpretation so later review can distinguish source material from inference.
Apply the article’s method to one routine case and one case near a boundary. Show the calculation or rule path, identify missing information and explain why plausible alternatives were accepted or rejected.
Change one material assumption at a time. Record what remains stable, what reverses and what becomes unresolved. Missing evidence must remain visible rather than being converted into a convenient zero, default or certainty.
Compare the result with at least one simpler alternative: explicit-only reading, an unweighted description, a transparent rule or a basic case-flow count, as appropriate. If the complex method does not improve the decision or explanation, prefer the simpler method. Record both outputs so later reviewers can see what the added structure contributed and what it obscured.
Invite a reviewer who was not involved in design to challenge scope, definitions, denominators, boundary conditions and incentives. Ask what evidence is systematically absent, who bears the cost of error, and how the method could be gamed. Convert each credible challenge into a test, collection action or explicit unresolved limitation rather than a defensive footnote.
Assign an owner, correction route, retention rule and review date. Verify that correction reaches derived outputs and future decisions, then retire the method when its assumptions, authority or evidence base no longer hold.
Design correction and appeal
Choose one consequential workflow and state the decision, the people affected, the time boundary and the claim being tested. Record original evidence before interpretation so later review can distinguish source material from inference.
Apply the article’s method to one routine case and one case near a boundary. Show the calculation or rule path, identify missing information and explain why plausible alternatives were accepted or rejected.
Change one material assumption at a time. Record what remains stable, what reverses and what becomes unresolved. Missing evidence must remain visible rather than being converted into a convenient zero, default or certainty.
Compare the result with at least one simpler alternative: explicit-only reading, an unweighted description, a transparent rule or a basic case-flow count, as appropriate. If the complex method does not improve the decision or explanation, prefer the simpler method. Record both outputs so later reviewers can see what the added structure contributed and what it obscured.
Invite a reviewer who was not involved in design to challenge scope, definitions, denominators, boundary conditions and incentives. Ask what evidence is systematically absent, who bears the cost of error, and how the method could be gamed. Convert each credible challenge into a test, collection action or explicit unresolved limitation rather than a defensive footnote.
Assign an owner, correction route, retention rule and review date. Verify that correction reaches derived outputs and future decisions, then retire the method when its assumptions, authority or evidence base no longer hold.
Set review and retirement triggers
Choose one consequential workflow and state the decision, the people affected, the time boundary and the claim being tested. Record original evidence before interpretation so later review can distinguish source material from inference.
Apply the article’s method to one routine case and one case near a boundary. Show the calculation or rule path, identify missing information and explain why plausible alternatives were accepted or rejected.
Change one material assumption at a time. Record what remains stable, what reverses and what becomes unresolved. Missing evidence must remain visible rather than being converted into a convenient zero, default or certainty.
Compare the result with at least one simpler alternative: explicit-only reading, an unweighted description, a transparent rule or a basic case-flow count, as appropriate. If the complex method does not improve the decision or explanation, prefer the simpler method. Record both outputs so later reviewers can see what the added structure contributed and what it obscured.
Invite a reviewer who was not involved in design to challenge scope, definitions, denominators, boundary conditions and incentives. Ask what evidence is systematically absent, who bears the cost of error, and how the method could be gamed. Convert each credible challenge into a test, collection action or explicit unresolved limitation rather than a defensive footnote.
Assign an owner, correction route, retention rule and review date. Verify that correction reaches derived outputs and future decisions, then retire the method when its assumptions, authority or evidence base no longer hold.
Questions readers ask
Is a large sample representative?
- Not necessarily. Representation depends on frame and selection, not count alone.
Is source reliability the same as coverage?
- No. Reliability concerns reporting within access; coverage concerns reach.
Can weighting fix undercoverage?
- Only when missingness and inclusion can be modelled with defensible information.
What is survivorship bias?
- A distortion caused when units that failed, left or disappeared are absent from observation.
Does triangulation mean adding more reports?
- It means adding sufficiently independent access and selection pathways.
How should gaps be written?
- Name the excluded group or event, likely bias direction, decision consequence and next collection action.
Practise, verify and continue
Select one intelligence estimate and draw its target, frame, observed sample and final record. Audit selection at every arrow, reconstruct one missing group, compare three sensitivity scenarios and rewrite the judgement with a decision-facing coverage statement.
Apply the method to a school survey or hiring funnel. Identify absent people, attrition, reporting incentives and which claims must be narrowed.
Related eduKate reading
Collection requirements, source validation and analytic gaps
Competing hypotheses and diagnostic evidence
Negative reporting and absence-of-evidence safeguards
Analytic lineages, evidence provenance and judgement revision
Authoritative sources and scope
Office of the Director of National Intelligence, ICD 203 Analytic Standards
U.S. GAO, Assessing Data Reliability
U.S. Census Bureau, Quality Standard A3: Developing and Implementing a Sample Design
U.S. Census Bureau, Quality Standard D3: Producing Measures and Indicators of Nonsampling Error
This article is general analytic education and does not describe classified sources or a current operation. Examples are original composites. The cited standards support objectivity, sampling design, data reliability and nonsampling-error discipline; the coverage workflow is a practical synthesis. Protect people, sources and sensitive methods. Do not infer population prevalence, causal effects or absence solely from an observed sample. Name selection assumptions, use conservative bounds when frames are incomplete, and collect new evidence when weighting cannot repair missing coverage. Review later outcomes to improve models without punishing candid reporting of gaps.