THE GOLD STANDARD OF INTELLIGENCE · SPECIALIST STUDY
Treat an interviewer change as a measured fieldwork intervention, not an administrative footnote
In respondent-driven sampling, an interviewer can influence consent, network-size questions, coupon instructions, language choice, probing and recruitment continuity. A replacement may arrive exactly when sites, waves and recruitment chains are changing, so a before-and-after difference cannot be assigned to the person by inspection. A defensible changeover preserves scripts and versions, overlaps or bridges where possible, records paradata continuity, tests whether items behave comparably and shows whether weighted conclusions survive plausible interviewer and language effects.
This guide belongs to the Intelligence subject root, the Gold Standard of Education, and the How X Works hub. It begins with a concrete decision, then tests whether the method still holds when the context changes.
Choose a reading route
Define roles and estimands · chapters 1, 2, 3
Standardise and bridge the handover · chapters 4, 5
Preserve paradata continuity · chapters 6, 7, 8
Test measurement and estimates · chapters 9, 10, 11
Govern, report and transfer · chapters 12, 13
The whole problem at a glance
| Risk | Observable signal | Useful response |
|---|---|---|
| Coaching drift | Probe and correction differences | Recorded calibration and retraining |
| Language mismatch | Switching, pauses and item nonresponse | Verified language roster and bridge |
| Recruitment disruption | Coupon return or chain break | Transition map by recruiter chain |
| Paradata discontinuity | Changed codes or timing rules | Versioned crosswalk and dual run |
Do not compress these pathways into one interviewer indicator.
CHAPTER 1 OF 13 · Define roles and estimands
1. Define interviewer responsibilities before measuring change
RDS interviewers may screen eligibility, obtain consent, administer items, explain coupons, record network degree and manage incentives. Different duties create different bias pathways.
Evidence record: Keep role matrix, site, shift, languages, modules, coupon duties, authority limits and exact dates for each interviewer. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Do not attribute a discontinuity until the pathway between the changed duty and the observed outcome is stated. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Move coupon management to a separate clerk; update the causal map rather than retaining one generic interviewer effect. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 1 check · what should survive an audit
- State the decision unit for “Define interviewer responsibilities before measuring change”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 2 OF 13 · Define roles and estimands
2. Fix the target estimand and decision window
The concern may be prevalence, network degree, recruitment productivity, consent or trend. Each responds differently to interviewer change and calendar time.
Evidence record: Record population, outcome, estimator, wave, sites, recruitment chains, analysis date and decision threshold before inspecting effects. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Reject post-hoc switching among outcomes unless the change and multiplicity are disclosed. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A policy question shifts from prevalence to subgroup disparity; reassess power and invariance evidence. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 2 check · what should survive an audit
- State the decision unit for “Fix the target estimand and decision window”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 3 OF 13 · Define roles and estimands
3. Map confounding by time, site and recruitment chain
A new interviewer often starts after a leave, site expansion or slowing wave. Their cases can occupy different network depths and population strata.
Evidence record: Retain interview date, site, seed, recruiter, chain depth, coupon history, interviewer, language, shift and fieldwork intervention. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Never read a raw before-and-after difference as an interviewer effect when assignment was entangled with these factors. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: The replacement covers weekends at one site; add schedule and site support to the comparison. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 3 check · what should survive an audit
- State the decision unit for “Map confounding by time, site and recruitment chain”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 4 OF 13 · Standardise and bridge the handover
4. Standardise scripts, probes and coupon coaching
WHO field manuals emphasise consistent administration because wording and delivery variation can undermine validity. RDS also requires consistent explanation of recruitment eligibility and coupon use.
Evidence record: Version scripts, permitted probes, refusal conversion, network-size prompts, coupon instructions, demonstrations and deviations. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Certify the replacement on observed practice, not attendance alone, and retrain when coded deviations cross a threshold. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A translated probe has no exact equivalent; document the approved conceptual wording and examples. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 4 check · what should survive an audit
- State the decision unit for “Standardise scripts, probes and coupon coaching”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 5 OF 13 · Standardise and bridge the handover
5. Run a bounded overlap or bridge when feasible
An overlap window can expose administration differences under more comparable field conditions. It need not be large, but its selection and purpose must be honest.
Evidence record: Keep case allocation rule, common dates and sites, language, network position, recordings or observations, reviewer coding and exclusions. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Use the bridge to test protocol and plausible effect size, not to claim random assignment when allocation was operational. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: No direct overlap is possible; use archived recordings, paired role-play and restricted temporal comparisons with wider uncertainty. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 5 check · what should survive an audit
- State the decision unit for “Run a bounded overlap or bridge when feasible”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
Worked overlap window
| Cohort | Old interviewer | New interviewer | Purpose |
|---|---|---|---|
| Bilingual bridge cases | 12 | 12 | Compare language and administration |
| Same recruitment chains | 8 | 9 | Inspect chain continuity |
| Independent quality review | 6 recordings | 6 recordings | Code protocol adherence |
| Total completed | 26 | 27 | Not a causal estimate alone |
Case mix and network position still require adjustment and sensitivity analysis.
CHAPTER 6 OF 13 · Preserve paradata continuity
6. Control language matching as a measured process
Language comfort can affect disclosure, comprehension and network-size reporting. Matching by a broad label is insufficient when dialect, literacy and switching matter.
Evidence record: Record offered languages, selected version, interviewer proficiency verification, switches, interpreter use, comprehension repairs and item nonresponse. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Assign only verified language capability and trigger review when switching or repair patterns change after handover. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A participant begins in one language and answers sensitive items in another; preserve item-level exposure. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 6 check · what should survive an audit
- State the decision unit for “Control language matching as a measured process”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 7 OF 13 · Preserve paradata continuity
7. Version paradata definitions across the change
Timing, contact attempts, refusal codes, observations and correction flags can change because a new interviewer records them differently or because software changed simultaneously.
Evidence record: Keep data dictionary version, collection application, device, clock rule, code crosswalk, missing-value semantics and dual-coded sample. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Do not join old and new paradata until every field is identical, crosswalked or marked non-comparable. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A software release rounds duration differently on the first day of handover; separate instrumentation from behaviour. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 7 check · what should survive an audit
- State the decision unit for “Version paradata definitions across the change”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 8 OF 13 · Preserve paradata continuity
8. Use paradata as evidence, not as a verdict
Long interviews can mean difficulty, rapport or conscientious probing; short interviews can mean fluency or shortcutting. Interpret patterns with item, site and observation evidence.
Evidence record: Compare duration distributions, edits, breakoffs, missingness, contact attempts, recruitment explanations and quality-review notes by relevant strata. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Escalate a pattern only when a plausible pathway and corroborating evidence exist; avoid ranking staff from one proxy. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Remote interviewing changes pause and connection patterns; update interpretation before applying old thresholds. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 8 check · what should survive an audit
- State the decision unit for “Use paradata as evidence, not as a verdict”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 9 OF 13 · Test measurement and estimates
9. Test item-level comparability and invariance
If response probabilities change after accounting for the underlying construct and relevant case mix, items may function differently. Sample size and RDS dependence limit formal tests, so combine statistics with cognitive and field evidence.
Evidence record: Retain item wording, response categories, interviewer and language exposure, bridge results, model assumptions, fit, flagged items and multiplicity control. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Revise or qualify trend claims when non-invariance is material and cannot be separated from the changeover. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A sensitive item shifts but neutral items do not; investigate privacy and probing rather than forcing one global adjustment. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 9 check · what should survive an audit
- State the decision unit for “Test item-level comparability and invariance”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
Estimator sensitivity set
| Analysis | Changeover treatment | Decision use |
|---|---|---|
| Primary | Observed interviewer and language covariates | Declared estimate |
| Restricted overlap | Common sites, dates and chain depths | Comparability check |
| Worst plausible shift | Item responses moved within justified bounds | Decision robustness |
| Leave-one-interviewer-out | Remove each interviewer in turn | Influence check |
Report where conclusions change, not only the preferred estimate.
CHAPTER 10 OF 13 · Test measurement and estimates
10. Carry changeover uncertainty into the RDS estimator
Interviewer effects can alter reported network degree, coupon return and outcomes used in weighted estimates. Treating the final sample as if fieldwork were unchanged understates uncertainty.
Evidence record: Recalculate estimates under observed covariates, restricted overlap, plausible item shifts, degree perturbations and interviewer exclusions with confidence intervals. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Report the conclusion as robust only if it survives justified alternatives relevant to the decision threshold. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A subgroup is almost entirely interviewed after the handover; disclose weak separation instead of presenting precise adjustment. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 10 check · what should survive an audit
- State the decision unit for “Carry changeover uncertainty into the RDS estimator”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 11 OF 13 · Test measurement and estimates
11. Protect chain continuity during operational transfer
Outstanding coupons, appointments, incentive records and participant questions need a controlled handover. Lost context can suppress recruitment or create inconsistent promises.
Evidence record: Reconcile coupon states, recruiter links, appointments, contact preferences, unresolved issues, incentive approvals and access revocation. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Close the handover only when outstanding cases have an accountable owner and participants receive consistent information. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A former interviewer retains device access; treat revocation and data custody as part of field validity and security. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 11 check · what should survive an audit
- State the decision unit for “Protect chain continuity during operational transfer”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 12 OF 13 · Govern, report and transfer
12. Monitor coaching drift without punitive shortcuts
Coaching can drift gradually as staff adapt to difficult cases. Supportive calibration, observation and feedback protect consistency better than hidden surveillance or crude performance rankings.
Evidence record: Schedule calibration samples, independent coding, disagreement review, retraining, participant complaints and corrective verification with privacy safeguards. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Use quality evidence to correct the process; separate research validity review from employment action and follow applicable law. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A high-performing interviewer has unusual recruitment rates; inspect protocol and case mix before praise or blame. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 12 check · what should survive an audit
- State the decision unit for “Monitor coaching drift without punitive shortcuts”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
CHAPTER 13 OF 13 · Govern, report and transfer
13. Publish a transparent changeover account
Readers need dates, role and language coverage, overlap evidence, paradata comparability, sensitivity analyses and unresolved limits. Personal identities are usually unnecessary.
Evidence record: Version the fieldwork appendix, interviewer codes, data dictionary, analysis scripts, disclosure controls and correction path. Preserve versions, timestamps, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Do not claim unchanged trend when the evidence only shows that conclusions were insensitive within tested bounds. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Transfer the account to a longitudinal household survey while replacing RDS recruitment-chain fields. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 13 check · what should survive an audit
- State the decision unit for “Publish a transparent changeover account”.
- Bind every claim to an observed version, time boundary and comparison set.
- Connect the evidence to an owner, bounded action and later verification.
Contents · Previous chapter · Return to the Intelligence guide
Verification lab: work the decision twice
Design a 10-day interviewer overlap for two sites and two languages. Specify allocation, observation, participant safeguards and what the bridge cannot prove.
Create a paradata crosswalk when duration changes from seconds to rounded minutes during handover; show which comparisons remain valid.
Perturb network-degree responses and one sensitive item within justified bounds, then decide whether a policy conclusion remains stable.
Limits and misconceptions
A before-and-after difference is not a causal interviewer effect
Time, site, network chain and population composition can move together.
Language labels do not prove equivalence
Version, proficiency, switching and comprehension evidence matter.
Paradata are ambiguous proxies
Use them with protocol, item and observation evidence.
Adjustment cannot repair complete confounding
If interviewer and subgroup are inseparable, report the limitation.
Questions readers ask
Why does interviewer change matter in RDS?
- It can affect measurement, network-degree reporting, coupon coaching and recruitment continuity.
Is overlap mandatory?
- Not always, but a bounded bridge or documented alternative strengthens comparability evidence.
What is coaching drift?
- A gradual change in how scripts, probes or recruitment instructions are delivered.
What are paradata?
- Data about collection processes, such as attempts, timing, edits, outcomes and administrative events.
How is measurement invariance checked?
- By combining item models, bridge comparisons, cognitive evidence and sensitivity analyses.
Can an interviewer indicator solve the problem?
- Only when there is enough overlap and the model reflects site, time, language and chain structure.
Practise, verify and continue
Write an interviewer-changeover protocol for an RDS study with two languages, three sites and active recruitment chains. Include role transfer, overlap, paradata crosswalk, invariance checks and estimator sensitivity.
Move the protocol to a school climate survey. Replace recruitment coupons and network degree with classroom allocation while preserving standardisation, language matching, paradata continuity and sensitivity analysis.
Practical evidence check
Before implementation, write a one-page evidence map with the decision, unit of analysis, version or time boundary, inputs, outcomes, uncertainty, authority and stop rule. Attach one worked case that passes and one that fails. A reviewer should be able to reproduce the reasoning without relying on the memory of the person who built the system.
Run one changed-context rehearsal before approval. Alter a mechanism rather than a cosmetic detail, then identify which calculation, safeguard or governance gate must change. Record any conclusion that does not transfer. This short rehearsal is often the fastest way to expose a hidden assumption before it becomes an operational dependency.
Related eduKate reading
RDS survey-language version changes and trend comparability
RDS adaptive fieldwork governance
RDS multi-site harmonisation
RDS degree-question design and construct validity
Authoritative sources and scope
WHO World Health Survey Plus interviewer manual
WHO Global Adult Tobacco Survey field interviewer manual
CDC National Survey of Family Growth paradata user guide
CDC NSFG interviewer observations and paradata files
WHO and CDC sources support consistent interviewing and documented paradata. They do not prescribe this RDS changeover design or identify a universal interviewer-effect model. The overlap counts are fictional, and actual recording, employment, privacy, consent and language practices must follow the study protocol and local law.