THE GOLD STANDARD OF INTELLIGENCE · SPECIALIST STUDY
Design respondent-driven sampling degree questions as measurement instruments: define eligible ties, decompose recall, prevent double-counting, test repeatability and show how the answer changes inference
Respondent-driven sampling uses reported personal network size to help adjust for unequal inclusion, yet the degree question is often treated as a single factual prompt. It is a measurement design problem. The gold standard aligns the counted tie with recruitment eligibility, uses name-generator or category decomposition only when it improves recall, makes categories mutually interpretable, records ranges and uncertainty, tests repeatability without mistaking agreement for truth, and evaluates construct validity against recruitment behaviour and alternative elicitation methods before degree enters an inverse-weight estimator.
This guide belongs to the Intelligence subject guide, the Gold Standard of Education, and the How X Works learning library. Choose a reading route, work the examples, then use the transfer checks to test whether the method survives a changed context.
Choose a reading route
Build the foundations · chapters 1–2
Apply the core method · chapters 3–5
Test evidence and alternatives · chapters 6–7
Handle limits and disagreement · chapters 8–9
Transfer and govern · chapters 10
The whole problem at a glance
| Design choice | Benefit | Failure to watch |
|---|---|---|
| Single count | Fast and simple | Boundary ambiguity |
| Name generator | Concrete recall | Privacy and burden |
| Category decomposition | Memory cues | Overlap and double-counting |
| Range response | Honest uncertainty | Estimator integration |
The instrument changes what respondents retrieve, count and disclose.
CHAPTER 1 OF 12 · Build the foundations
1. Start from the recruitment relation
Core idea. Degree should approximate eligible population members connected through reciprocal recognition or contact that could support recruitment.
Method. Specify population, geography, time window, recognition direction and practical recruitment possibility in plain language; pilot translations.
Worked example. A participant knows 40 broad contacts but only 12 live locally, are mutually known and could receive a coupon.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Recruitment ties can be directed, private and context-dependent.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 1 is complete only when the claimed result survives that review.
Misconception to avoid. How many people do you know is self-explanatory. Each boundary word changes the set.
Practical evidence check
- Ask respondents to narrate who is included at each filter and compare with protocol intent.
- Return to chapter contents
CHAPTER 2 OF 12 · Build the foundations
2. Choose an elicitation architecture deliberately
Core idea. A total, successive filters, grouped counts and named-person recall impose different cognitive tasks and disclosure risks.
Method. Map network size and sensitivity, compare burden in cognitive interviews and choose the least intrusive method that supports the construct.
Worked example. A small network may use private initials-based recall; hundreds of weak ties may need bounded category prompts and a range.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Name generators can miss weak ties; totals invite rounding; long decompositions create fatigue.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 2 is complete only when the claimed result survives that review.
Misconception to avoid. More detailed questions always measure better. Detail can increase refusal and constructed answers.
Practical evidence check
- Randomise two safe formats in a pilot and compare completion and respondent reasoning.
- Return to chapter contents
CHAPTER 3 OF 12 · Apply the core method
3. Use name generators without collecting identities
Core idea. Name generators support recall by enumeration, but the study normally needs a count rather than identifiable contacts.
Method. Use private mental enumeration or non-identifying tokens, prohibit identity recording and delete temporary aids.
Worked example. A respondent thinks through initials by household, work and venue, then reports a deduplicated count; no identity leaves the interview.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Initials and relationship details can re-identify people in small networks.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 3 is complete only when the claimed result survives that review.
Misconception to avoid. Names are necessary for accuracy. Recall support can avoid identity collection.
Practical evidence check
- Observe whether pilots deduplicate ties while retaining no person-level list.
- Return to chapter contents
CHAPTER 4 OF 12 · Apply the core method
4. Decompose recall without assuming partitions
Core idea. Relationship, venue and activity cues aid recall, but natural social categories overlap and do not automatically sum.
Method. Tell respondents categories may overlap, count only new people after the first category or use safe deduplication, and record order effects.
Worked example. Family, coworker and friend prompts total 18, but five people have multiple roles; unique degree is 13.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Deduplication adds burden and privacy risk, while cue order can anchor later counts.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 4 is complete only when the claimed result survives that review.
Misconception to avoid. Category totals can always be added. They are prompts unless exclusivity is established.
Practical evidence check
- Reverse category order and examine overlap corrections and total changes.
- Return to chapter contents
CHAPTER 5 OF 12 · Apply the core method
5. Separate membership, contact and recognition filters
Core idea. Someone may fit the population yet not know the respondent or fall outside the contact window.
Method. Ask filters in a stable sequence, provide neutral borderline examples, retain intermediate counts and clarify contradictions.
Worked example. Twenty-five people fit eligibility, 18 were contacted recently and 14 are mutually recognised; the last set matches recruitment.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Sequential questions can anchor answers and reciprocity is rarely verifiable for every tie.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 5 is complete only when the claimed result survives that review.
Misconception to avoid. The largest count is safest. The estimator needs the tie aligned with opportunity.
Practical evidence check
- Change filter order and test whether the final count shifts.
- Return to chapter contents
Overlap example
| Category answer | Naive sum | Deduplicated interpretation |
|---|---|---|
| Family: 4 | 4 | Four eligible family ties |
| Coworkers: 6 | +6 | Two are also family |
| Friends: 8 | +8 | Three overlap earlier groups |
| Total | 18 | 13 unique eligible people |
The survey must define overlap handling before summing.
CHAPTER 6 OF 12 · Apply the core method
6. Capture uncertainty and ranges
Core idea. A forced integer hides uncertainty about a fluid network, while a range preserves recall limits.
Method. Allow a best estimate plus lower and upper bounds, record reasons and predefine interval sensitivity.
Worked example. A respondent says about 20, plausibly 15–30, because venue contacts vary weekly.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Wide ranges complicate estimators and confidence is not a calibrated probability.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 6 is complete only when the claimed result survives that review.
Misconception to avoid. Every person has one exact degree. Network boundaries and membership change.
Practical evidence check
- Recontact a subsample and see whether ranges cover later answers.
- Return to chapter contents
CHAPTER 7 OF 12 · Test evidence and alternatives
7. Test repeatability as a diagnostic
Core idea. Test–retest agreement reveals instability but identical answers can reflect remembered responses or repeated rounding.
Method. Repeat after a justified interval, vary nonessential cue order, report differences and heaping, and ask reasons for change.
Worked example. A person says 20 twice but explains that 20 means many; another says 18 then 21 with concrete recall.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Networks genuinely change and recontact can be selective.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 7 is complete only when the claimed result survives that review.
Misconception to avoid. High repeatability proves validity. Reliability is not construct validity.
Practical evidence check
- Compare repeated totals with category composition and explanations, not only correlation.
- Return to chapter contents
CHAPTER 8 OF 12 · Test evidence and alternatives
8. Build convergent and discriminant evidence
Core idea. Construct validity asks whether degree behaves as expected relative to alternative elicitation, recruitment opportunity and unrelated variables.
Method. Compare safe generator and total methods, coupon opportunity and known contexts; predeclare relationships and investigate contradictions.
Worked example. Degree tracks opportunity moderately but not an unrelated literacy score, supporting a bounded case.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Coupon limits, willingness and design make recruitment an imperfect indicator.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 8 is complete only when the claimed result survives that review.
Misconception to avoid. Correlation with recruits proves accuracy. It is one noisy external measure.
Practical evidence check
- Hold coupon supply constant and separate opportunity from successful recruitment.
- Return to chapter contents
Validity certificate
| Question | Evidence | Boundary |
|---|---|---|
| What tie is counted? | Eligibility and time window | Directed awareness |
| Can it be recalled? | Cognitive testing and burden | Privacy affects disclosure |
| Is it repeatable? | Test–retest pattern | Agreement is not accuracy |
| Does it behave as expected? | Recruitment and method comparison | No perfect gold standard |
Validity is an accumulated case, not one correlation.
CHAPTER 9 OF 12 · Test evidence and alternatives
9. Inspect interviewer, language and mode effects
Core idea. Sensitive disclosure varies by interviewer, translation and administration mode, creating systematic measurement differences.
Method. Use scripted probes, train neutral clarification, monitor distributions and treat mode shifts as instrument changes.
Worked example. One translation renders know as close friend; another includes familiar acquaintances, producing lower degree in the first.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Groups may differ substantively, so gaps do not alone prove bias.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 9 is complete only when the claimed result survives that review.
Misconception to avoid. Standard training removes interviewer effects. Monitoring remains necessary.
Practical evidence check
- Use bilingual cognitive interviews to trace each boundary term.
- Return to chapter contents
CHAPTER 10 OF 12 · Handle limits and disagreement
10. Carry uncertainty into weights
Core idea. Inverse-degree weights amplify differences among small reported degrees; treating measurement as fixed hides inferential sensitivity.
Method. Retain raw and derived degree, test alternative valid definitions and range draws, and show changes in estimates.
Worked example. Degree 2 versus plausible 3 changes inverse weight by one third; the report shows whether conclusions depend on it.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Better degree measurement cannot repair every RDS estimator assumption.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 10 is complete only when the claimed result survives that review.
Misconception to avoid. Once cleaned, uncertainty disappears. Processing cannot create certainty.
Practical evidence check
- Perturb degree within recorded ranges and compare with seed and recruitment sensitivities.
- Return to chapter contents
CHAPTER 11 OF 12 · Handle limits and disagreement
11. Govern privacy and instrument drift
Core idea. Network questions can expose stigmatised membership and relationships; repeated waves drift as platforms and recruitment practices change.
Method. Minimise detail, separate consent from recruitment, version wording, monitor refusals, allow correction and reassess validity after change.
Worked example. A platform migration changes recent contact; the study revises cues, pilots comparability and does not silently splice series.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Privacy protection limits validation detail and wording changes can break trends.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 11 is complete only when the claimed result survives that review.
Misconception to avoid. A validated question stays valid forever. Validity belongs to a use, population and period.
Practical evidence check
- Trace what the instrument reveals and what it deliberately does not retain.
- Return to chapter contents
CHAPTER 12 OF 12 · Transfer and govern
12. Publish a measurement dossier
Core idea. Readers need more than a variable label to judge how degree was produced and how results depend on it.
Method. Publish construct map, exact questions, translations, cognitive findings, distributions, repeatability, validity comparisons, processing lineage and sensitivities.
Worked example. A later analyst rebuilds each analysed degree from its raw range and sees which conclusions fail under another definition.
Changed-context transfer. Changed-context transfer: apply the same reasoning structure to a school or public-service decision. Preserve the formal relationship, then replace the population, authority, privacy duties, evidence and remedy; never copy the domain conclusion.
Limits. Confidentiality can require controlled access and synthetic examples.
Evidence and review. For this chapter, keep a dated evidence record: the input, rule or assumption, result, accountable owner and a condition that would change the conclusion. Work an ordinary case and a boundary case, preserve rejected alternatives, and require an authorised independent reader to reproduce where judgement entered. Chapter 12 is complete only when the claimed result survives that review.
Misconception to avoid. The questionnaire appendix is the whole validity case. Administration and inferential impact also matter.
Practical evidence check
- Have an independent analyst rebuild the variable and one sensitivity estimate.
- Return to chapter contents
Build the specialist decision record
For “The Gold Standard of Intelligence | RDS Degree-Question Design, Name-Generator Decomposition, Overlapping Tie Categories, Repeatability and Construct Validity”, begin with one bounded decision rather than an abstract claim. Name the affected people, data or institutions; the time window; the authority for acting; and the conclusion the method is allowed to support.
Preserve source material before interpretation. Record definitions, exclusions, transformations and the person responsible for each judgement. A reviewer should be able to separate an observed fact from an inferred result, a policy choice and a presentation shortcut.
Work one ordinary case in full. Show the input, every material rule or calculation, the output and the action that follows. Then write what the same evidence would mean under the simplest credible alternative method.
Keep an explicit unresolved state. Missing, protected, conflicting or unevaluated information must not become zero, false, consent or absence merely because a form expects one value.
Finish the record with a claim ledger. For every sentence that could drive action, identify its supporting source, inferential step, confidence or legal status, intended reader and expiry condition. This keeps descriptive evidence, formal consequence, professional judgement and institutional authority from collapsing into one impressive-looking conclusion.
Run the boundary-case audit
Select a case near the method’s domain edge: a missing input, a disputed definition, an endpoint, a sparse group, a resource limit or a legal deadline. Boundary cases reveal the contract that routine examples can conceal.
Change one assumption at a time and retain every output. Mark what stays stable, what reverses and what becomes undefined. If several assumptions move together, report that dependence instead of attributing the change to one convenient cause.
Search deliberately for a counterexample to the strongest claim. A method that works on the published illustration can still fail under a permitted input, an alternative grouping, a changed unit or an inaccessible evidence pathway.
Translate the counterexample into plain language: who would receive a different decision, what evidence would be hidden, and which safeguard or narrower claim repairs the problem.
Include an adversarial but plausible case. Let a missing group, ambiguous identifier, extreme weight, conflicting rule or urgent deadline interact with normal system incentives. The aim is not theatrical failure; it is to learn whether ordinary pressure makes the method conceal uncertainty, shift burden or reward a misleading shortcut.
Test independent reproduction
Give a reviewer the bounded inputs, public method contract and permitted access, but not the desired answer. Ask them to reproduce the result and identify every place where judgement, local knowledge or unavailable evidence was required.
Compare the reviewer’s intermediate states, not only the final number or label. Matching outputs can hide different denominators, inference paths, missing-data treatments or authorities that will diverge on the next case.
Where reproduction fails, classify the cause: ambiguous definition, undocumented transformation, incompatible implementation, inaccessible source, calculation error or legitimate alternative interpretation. Each class needs a different remedy.
Retain the disagreement and resolution. A corrected record should show what changed, why it changed and which prior outputs or decisions must be revisited.
Run the reproduction on a changed but structurally similar case after the first discrepancy is repaired. A method is not transferable merely because its authors can explain the original example. The new case tests whether the public contract carries the reasoning or whether hidden expertise is still doing essential work.
Design correction, challenge and remedy
Assign a visible route for correcting inputs, challenging assumptions and reviewing the application of the method. Technical accuracy without a usable remedy leaves the affected person carrying an error they cannot contest.
Separate urgent live-case repair from later system learning. A current decision may need pause, reassignment, fresh evidence or a human review before the organisation finishes its longer investigation.
Protect confidential and privileged material during review, but do not use confidentiality as a reason for unrecorded authority. Layer public explanation, protected oversight and individual reasons according to lawful access.
Verify that correction reaches cached results, derived records, dashboards and future decisions. Closing a complaint is not enough if the old conclusion continues to travel.
Measure remedy from the affected person’s position. A revised database row is useful, but it is incomplete when a hearing, service allocation, published claim or downstream model still relies on the old result. Record notification, restored opportunity, compensation or procedural repair where the governing framework provides it.
Set review and retirement triggers
Review after a change in definitions, population, law, source access, measurement scale, software, vocabulary or incentives. A stable formula can become invalid when the world or the meaning of its inputs changes.
Monitor leading signals as well as final outcomes: coverage gaps, timeouts, overrides, complaints, disagreement, missingness and boundary frequency. These often reveal failure before a headline metric moves.
Predefine thresholds for recalibration, suspension and retirement. Do not force a method to answer after its assumptions fail merely because replacing it is inconvenient.
At retirement, preserve the versioned contract, evidence, known limitations and migration decision. State which historic results remain interpretable and which require warning, restatement or withdrawal.
Schedule one prospective review before confidence fades or controversy forces it. Assign named owners for evidence, implementation and rights impact, and invite at least one reviewer outside the design team. The review should be able to narrow the claim, change the method or stop use without requiring proof of catastrophic failure.
Questions readers ask
What should RDS degree count?
- Eligible population members connected through a tie aligned with realistic recruitment opportunity.
Are category counts additive?
- Only when categories are disjoint or overlaps are safely deduplicated.
Must name generators collect names?
- No. Private mental enumeration or non-identifying tokens can support recall.
Does repeatability prove accuracy?
- No. It diagnoses stability and can also reflect rounding or remembered answers.
How is construct validity assessed?
- Combine cognitive evidence, alternative elicitation, recruitment behaviour and predicted relationships.
Should uncertainty affect estimates?
- Yes. Plausible ranges and definitions should be propagated through sensitivity analysis.
Practise, verify and continue
Choose one consequential case. Reproduce the complete method, work one ordinary and one boundary case, test the strongest rival explanation, and state the evidence that would reverse or suspend the result. Build an evidence pack containing original inputs, definitions, calculations or rule steps, alternatives rejected, uncertainty, reviewer comments and the final action. Give the pack to a reader who did not design the method. Ask that reader to reproduce the result, identify one missing assumption and construct a case that would make the method return unresolved. Revise the public explanation only after the evidence pack and the operational process agree.
Move the method into an education or public-service setting. Keep the reasoning structure while replacing domain authority, population, privacy, evidence and correction routes. Start with a low-stakes historical example, then compare the specialist method with a transparent baseline. Check whether the same people, records and decisions remain inside scope; whether the scale or vocabulary still means the same thing; and whether a person affected by error can understand and challenge it. Pilot under human review, record reversals and burden, and stop the transfer if the formal resemblance hides a different ethical or legal problem.
Related eduKate reading
RDS degree heaping and weight sensitivity
RDS network-size estimation and hidden-population inference
RDS recruitment effectiveness and nonresponse
Authoritative sources and scope
Gile and Handcock, Respondent-Driven Sampling: An Assessment of Current Methodology
McCreesh et al., Evaluation of Respondent-Driven Sampling Estimates
Gile, Johnston and Salganik, Diagnostics for Respondent-Driven Sampling
Measuring Network Sizes in Respondent-Driven Sampling
Latent Variable Model for RDS Network Size
This article is survey-measurement and RDS education, not a promise that one degree question removes selection bias. Examples are original teaching cases. Protect identities and sensitive membership, obtain ethics review, preserve wording and translations, and present degree sensitivity alongside seed, recruitment, homophily and estimator assumptions.