THE GOLD STANDARD OF INTELLIGENCE · SPECIALIST STUDY
Protect respondent-driven sampling from workload pressure without turning staff metrics into a substitute for participant welfare or data validity
An RDS interviewer’s workload is more than completed interviews. It includes screening, consent, language matching, network-degree questions, coupon explanations, disputes, privacy concerns and unresolved follow-up. Unequal allocation can concentrate difficult cases or late shifts; fatigue and conversion targets can change pacing, probing and refusal handling. A defensible workload system measures exposure and complexity, preserves participant choice, uses paradata as a prompt for review rather than punishment and shows whether conclusions survive plausible workload-related measurement and recruitment effects.
This guide belongs to the Intelligence subject root, the Gold Standard of Education, and the How X Works hub. It begins with a concrete decision, then tests whether the method still holds when the context changes.
Choose a reading route
Define work and valid outcomes · chapters 1, 2, 3
Allocate cases fairly · chapters 4, 5
Detect pressure and fatigue · chapters 6, 7, 8
Protect estimates and participants · chapters 9, 10, 11
Govern, learn and transfer · chapters 12, 13
The whole problem at a glance
| Work component | Useful measure | Misleading shortcut |
|---|---|---|
| Interview administration | Minutes by module and language | Completed cases alone |
| Recruitment support | Open coupons and participant questions | Coupon returns as staff score |
| Difficult contact | Attempts, travel and accessibility | Nonresponse as failure |
| Quality repair | Edits, debriefs and recontact | Treating all long cases as inefficiency |
No single productivity count represents a fair workload.
CHAPTER 1 OF 13 · Define work and valid outcomes
1. Define workload around actual duties
Interviewers may screen, consent, administer, explain recruitment, handle incentives and record incidents. Counts that omit these tasks reward shortcuts and conceal burden.
Evidence record: Keep duty taxonomy, case state, site, travel, language, module, coupon support, incident handling, administrative time and unresolved work. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Use only measures linked to a declared staffing, safety or quality decision and disclose excluded labour. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Separate payment support into another team; update workload rather than preserving obsolete assumptions. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 1 check · what should survive an audit
- State the decision unit for “Define workload around actual duties”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 2 OF 13 · Define work and valid outcomes
2. Distinguish productivity, quality and participant rights
Fast completion, high response and many coupons are not interchangeable with valid, voluntary data. Pressure can improve one metric while harming another.
Evidence record: Define success separately for protocol adherence, participant understanding, refusal respect, data completeness, recruitment continuity and staff safety. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Never compensate one mandatory participant safeguard with productivity elsewhere. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A participant needs a long accessibility accommodation; classify the time as valid work, not inefficiency. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 2 check · what should survive an audit
- State the decision unit for “Distinguish productivity, quality and participant rights”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 3 OF 13 · Define work and valid outcomes
3. Fix the estimand before analysing workload effects
The question may concern prevalence, network degree, recruitment probability, item missingness or subgroup representation. Each has different pathways from workload.
Evidence record: State population, outcome, RDS estimator, time window, sites, chains, workload exposure and decision threshold before modelling. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Reject post-hoc outcome switching unless multiplicity and rationale are disclosed. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A new concern is subgroup exclusion rather than overall prevalence; rebuild the comparison. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 3 check · what should survive an audit
- State the decision unit for “Fix the estimand before analysing workload effects”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 4 OF 13 · Allocate cases fairly
4. Allocate by capacity and case complexity
Equal case counts can be unequal work when languages, accessibility, geography, sensitive modules, chain state and follow-up differ. Allocation rules should be explicit and revisable.
Evidence record: Record assignment time, capacity, skills, complexity markers, outstanding duties, shift, site, travel and reason for manual override. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Prevent new assignment when protected rest, verified language capacity or declared load ceilings would be breached. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A surge arrives in one language; add capacity or change scheduling rather than misassigning cases. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 4 check · what should survive an audit
- State the decision unit for “Allocate by capacity and case complexity”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 5 OF 13 · Allocate cases fairly
5. Audit allocation for systematic imbalance
Difficult cases can accumulate with experienced staff, new staff or inconvenient shifts. That may be justified operationally but can confound interviewer comparisons.
Evidence record: Compare complexity, nonresponse history, site, network depth, subgroup, language, schedule and reassignment by interviewer over time. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Escalate unexplained imbalance and preserve legitimate operational reasons instead of forcing numerical equality. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A specialist deliberately handles sensitive cases; assess workload and estimator overlap separately. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 5 check · what should survive an audit
- State the decision unit for “Audit allocation for systematic imbalance”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
Worked allocation week
| Interviewer | Assigned cases | High-complexity cases | Late shifts |
|---|---|---|---|
| A | 18 | 3 | 1 |
| B | 15 | 9 | 4 |
| C | 21 | 2 | 0 |
| D | 12 | 8 | 5 |
Complexity categories and shift effects require local validation.
CHAPTER 6 OF 13 · Detect pressure and fatigue
6. Measure fatigue through multiple signals
Overtime, consecutive shifts, self-report, errors, pauses and protocol deviations can indicate fatigue, but each is noisy and culturally sensitive.
Evidence record: Retain scheduled and actual hours, breaks, consecutive days, travel, case complexity, voluntary check-ins, observed deviations and privacy protections. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Trigger supportive review from converging evidence; do not infer fitness or employment consequences from one proxy. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Remote work reduces travel but increases screen and emotional load; update the model. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 6 check · what should survive an audit
- State the decision unit for “Measure fatigue through multiple signals”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 7 OF 13 · Detect pressure and fatigue
7. Control refusal-conversion pressure
Repeated respectful contact can reduce avoidable nonresponse, but coercive persistence, incentive misrepresentation or refusal override violates participant choice and can distort selection.
Evidence record: Version contact limits, refusal codes, cooling-off, approved scripts, conversion authority, participant complaints and final outcome. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Stop contact at explicit refusal or safety conditions defined by protocol; do not reward staff for reversing protected refusals. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A soft appointment decline differs from a clear refusal; train and audit the distinction. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 7 check · what should survive an audit
- State the decision unit for “Control refusal-conversion pressure”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 8 OF 13 · Detect pressure and fatigue
8. Use paradata as a review prompt
Durations, attempts, edits and missingness help locate unusual work, but case mix and valid accommodation can produce the same pattern as poor practice.
Evidence record: Compare distributions within site, language, module, complexity and network depth; preserve small denominators and censored cases. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Require contextual review and corroborating evidence before changing staff, data or protocol status. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A software update changes keystroke timing; separate instrumentation from behaviour. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 8 check · what should survive an audit
- State the decision unit for “Use paradata as a review prompt”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 9 OF 13 · Protect estimates and participants
9. Set quality thresholds with known error costs
A low threshold floods supervisors and stigmatizes staff; a high threshold misses harm. Define what false alarm and missed detection mean for participants, staff and data.
Evidence record: Backtest thresholds on reviewed cases, retain sensitivity, specificity, delays, subgroup effects, appeals and holdout performance. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Use thresholds to route review, not to auto-delete interviews or determine discipline. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Case mix shifts after a new seed; revalidate thresholds before comparison. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 9 check · what should survive an audit
- State the decision unit for “Set quality thresholds with known error costs”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
Quality-threshold response
| Signal | First response | Unsafe response |
|---|---|---|
| Short duration cluster | Review module and case mix | Assume fabrication |
| Rising refusals | Inspect scripts, allocation and timing | Increase pressure |
| Degree-value heaping | Retrain and sensitivity test | Edit values silently |
| Excess overtime | Reduce load and review safety | Praise output |
They do not decide guilt or validity by themselves.
CHAPTER 10 OF 13 · Protect estimates and participants
10. Carry workload exposure into estimator sensitivity
Workload may affect degree reporting, item response, recruitment explanation and coupon returns. Adjustments require overlap; sensitivity bounds may be more honest when exposure and subgroup are confounded.
Evidence record: Re-estimate under workload covariates, restricted overlap, plausible item shifts, degree perturbations and leave-one-interviewer-out analyses. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Call conclusions robust only across justified alternatives relevant to the decision threshold. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: One site is covered entirely during high workload; report weak identification rather than precise correction. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 10 check · what should survive an audit
- State the decision unit for “Carry workload exposure into estimator sensitivity”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 11 OF 13 · Protect estimates and participants
11. Protect staff and participants during intervention
A workload response can include redistribution, rest, retraining, extra language support, supervision or temporary pause. Preserve confidentiality and avoid retaliatory metrics.
Evidence record: Record trigger, consultation, interim safeguards, reassignment, participant continuity, access changes, verification and support route. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Choose the least intrusive response that protects consent, safety and validity while meeting employment and study rules. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A staff member raises a concern anonymously; protect the report and investigate the system, not the identity. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 11 check · what should survive an audit
- State the decision unit for “Protect staff and participants during intervention”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 12 OF 13 · Govern, learn and transfer
12. Test unintended effects of workload limits
Caps can protect quality but also delay hard-to-reach groups, concentrate queues or encourage off-system work. Monitor who waits and which duties move elsewhere.
Evidence record: Track queue age, subgroup coverage, overtime displacement, unrecorded work, referral, missed appointments and participant burden after intervention. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Revise a limit when it shifts rather than reduces harm, while preserving non-negotiable rest and consent safeguards. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: A strict daily cap creates weekend backlog; change staffing and scheduling rather than relaxing all limits. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 12 check · what should survive an audit
- State the decision unit for “Test unintended effects of workload limits”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
CHAPTER 13 OF 13 · Govern, learn and transfer
13. Publish a bounded workload and quality account
Readers need allocation rules, thresholds, paradata limitations, interventions and estimator sensitivity without identifying workers or participants unnecessarily.
Evidence record: Version definitions, analysis scripts, privacy controls, appeals, unresolved confounding and corrections. Preserve timestamps, versions, denominators, excluded cases, failures and rejected alternatives so another reviewer can reconstruct the decision rather than inherit a conclusion.
Decision rule: Report what the evidence can and cannot attribute; avoid league tables or causal claims from operational assignment. Name the authorised owner, trigger, stop condition, remedy and verification window. A control is incomplete until a changed-context case tests the mechanism it is meant to protect.
Evidence discipline: record the observation, version or time boundary that supports this chapter’s conclusion. Preserve the rejected alternative as well as the selected action. That makes a later review able to distinguish a changed world from an undocumented change of mind.
Changed-context transfer: Transfer the method to a longitudinal household survey without recruitment coupons. The method transfers only if the decision question and measurement meaning survive the change. If either changes, rebuild the comparison rather than copying the earlier threshold or weight.
Chapter 13 check · what should survive an audit
- State the decision unit for “Publish a bounded workload and quality account”.
- Bind the claim to versions, denominators, missing cases and time boundaries.
- Connect evidence to an owner, bounded action and later verification.
Contents · Previous chapter · Return to the Intelligence guide
Verification lab: work the decision twice
Use the fictional roster to build a complexity-adjusted allocation review without converting it into an employee ranking.
Design a refusal-conversion protocol that distinguishes appointment delay, hesitation, explicit refusal and safety stop.
Perturb network degree and one sensitive outcome under high workload, then identify whether the RDS conclusion survives.
Limits and misconceptions
Case counts do not equal workload
Language, complexity, travel and unresolved duties matter.
Paradata are not a verdict
They need case-mix and protocol context.
Refusal conversion has ethical limits
Participant choice is non-compensatory.
Adjustment needs overlap
Complete confounding requires transparent sensitivity, not confident correction.
Questions readers ask
What should an RDS workload measure include?
- Interview duties, case complexity, language, recruitment support, incidents, administration and unresolved work.
Can interview duration identify poor quality?
- No. It is a review signal whose meaning depends on module, case and collection context.
What is case-allocation imbalance?
- Systematic concentration of harder, sensitive or inconvenient cases with particular staff or shifts.
How should refusal conversion work?
- Through bounded, respectful contact rules that stop at explicit refusal or safety conditions.
Why test estimator sensitivity?
- Because workload can affect degree reports, outcomes and recruitment patterns used by RDS estimators.
Should thresholds trigger discipline?
- They should route quality review and support; employment action needs separate lawful evidence and process.
Practise, verify and continue
Design a workload system for four RDS interviewers across two sites and languages. Include complexity, allocation, fatigue signals, refusal safeguards, threshold review and estimator sensitivity.
Move the system to a school survey. Replace coupons and network degree with class allocation while preserving participant choice, workload complexity, paradata limits and sensitivity analysis.
Practical evidence check
Before implementation, write a one-page evidence map with the decision, unit of analysis, version or time boundary, inputs, outcomes, uncertainty, authority and stop rule. Attach one worked case that passes and one that fails. A reviewer should be able to reproduce the reasoning without relying on the memory of the person who built the system.
Run one changed-context rehearsal before approval. Alter a mechanism rather than a cosmetic detail, then identify which calculation, safeguard or governance gate must change. Record any conclusion that does not transfer. This short rehearsal is often the fastest way to expose a hidden assumption before it becomes an operational dependency.
Related eduKate reading
RDS interviewer changeover and measurement invariance
RDS survey-language changes and trend comparability
RDS adaptive fieldwork governance
RDS multi-site harmonisation
Authoritative sources and scope
WHO World Health Survey Plus interviewer manual
CDC National Health Interview Survey field representative manual
CDC National Survey of Family Growth paradata user guide
WHO Global Adult Tobacco Survey field interviewer manual
WHO and CDC sources support consistent interviewing, quality assurance and contextual use of paradata. They do not prescribe this RDS workload framework or employment decisions. The roster is fictional; consent, staff welfare, recording, accessibility, privacy and employment practices must follow the study protocol and applicable law.