The Gold Standard · How X Works
The same question can become a different social event
A question asked face to face, by telephone or on a private form does not arrive in the same setting. Interviewer identity, translated wording and response display can change what feels safe, polite or possible to say. This guide shows how to preserve meaning while measuring those differences honestly.
Choose a reading route
- Start from first principles — define the unit and the threat.
- Follow the worked example — make the hidden arithmetic visible.
- Test transfer — change the context and check the rule.
- Audit the evidence — inspect records, assumptions and failure points.
- Use the practical checklist — apply the method to new work.
Chapter contents
Foundations: Chapters 1–3
Method: Chapters 4–6
Transfer and limits: Chapters 7–9
Connections and use: Chapters 10–12
1. The interviewer is part of the instrument
In interviewer-administered research, tone, pace, accent, age, gender, status and familiarity can affect disclosure and interpretation. This does not mean the interviewer is a contaminant to be removed. It means the interaction is part of how evidence was produced.
An interviewer effect is a systematic difference associated with who asks or how they ask, beyond the intended question. It can enter through probing, paraphrase, skipped options, non-verbal cues, rapport or respondent expectations.
First principle
Standardise the task enough to compare interviews, while documenting the social conditions needed to interpret them.
2. Translation must carry the construct
Literal word replacement does not guarantee equivalent meaning. A term such as household, work, care, partner or income may divide experience differently across languages. Grammar can also force gender, certainty or time distinctions that the source question leaves open.
Translation drift occurs when meaning changes during drafting, interviewer paraphrase, repeated revision or local adaptation. Use a translation team, concept definitions, review, adjudication and pretesting in each production language. Back-translation can flag differences, but it cannot replace direct evaluation with speakers.
| Layer | Check | Risk |
|---|---|---|
| Concept | What is being measured? | Same word, different category |
| Question | What task is requested? | Added judgement or time rule |
| Response | Are options usable? | Missing local answer |
| Field delivery | What is actually said? | Unrecorded paraphrase |
← Previous · Contents · Next →
3. Modes create different response conditions
Self-administered forms offer privacy and visible response options. Telephone interviews remove visual context. Face-to-face interviews allow clarification but can heighten social desirability. Web instruments can branch consistently yet exclude people with access or literacy barriers.
Mode comparability asks whether instruments collect meaningfully equivalent information, not whether screens and spoken scripts look identical. Sometimes different presentation is needed to preserve the same task. Any adaptation should be specified, tested and versioned.
- Keep the construct and reference period stable.
- Record the mode used for every case.
- Test sensitive questions for privacy effects.
- Check visual versus spoken response options.
- Do not pool modes automatically when assignment was selective.
← Previous · Contents · Next →
4. Worked example: three ways to ask
A fictional pilot randomly assigns 120 adults to answer the same financial-decision question by private web form, telephone or face-to-face interview, 40 per mode. The proportion answering “I decide alone” is 55% on web, 70% by telephone and 62.5% face to face.
| Mode | Yes responses | Rate |
|---|---|---|
| Private web | 22 of 40 | 55.0% |
| Telephone | 28 of 40 | 70.0% |
| Face to face | 25 of 40 | 62.5% |
The spread may reflect privacy, option presentation, rapport or chance. Because mode was randomised, it can be investigated as a mode effect in this pilot. If respondents had chosen their mode, user differences such as literacy or access would be mixed into the comparison.
A follow-up cognitive interview can reveal whether “decide alone” was interpreted as authority, payment or final approval.
← Previous · Contents · Next →
5. Train for faithful flexibility
Rigid reading protects consistency but can fail when a participant does not understand. Free paraphrase supports conversation but can change the construct. Training should define permitted clarifications, neutral probes and escalation rules.
Give interviewers a concept card, not only a script. Role-play misunderstandings, sensitive refusals and interpreter involvement. Record deviations in a field note or paradata rather than pretending every interview followed the same path.
| Situation | Allowed response | Record |
|---|---|---|
| Question unclear | Repeat or approved clarification | Clarification code |
| Answer incomplete | Neutral probe | Probe used |
| Translation unavailable | Pause or approved interpreter route | Language and interpreter |
| Participant declines | Accept refusal | Item nonresponse |
← Previous · Contents · Next →
6. Separate interviewer, language and mode
These factors often travel together: one interviewer works in one language; telephone is used only for distant participants; web is offered to younger respondents. Then observed differences cannot be assigned cleanly to one cause.
Where feasible, cross interviewers with languages or modes, randomise assignment, rotate workloads and monitor case mix. At minimum, retain interviewer ID, language, mode, date, duration and deviation fields. Multilevel models can describe clustering, but good design remains the first defence.
Practical check
Compare conclusions after adjusting for observed case mix and after restricting to the most comparable mode-language groups. If results shift, report the dependence.
← Previous · Contents · Next →
7. Transfer test: a migration study
Move the method to interviews about remittances across two countries. In one language, the closest common word for “gift” includes obligations; in another, gift implies no repayment. A telephone team paraphrases freely, while the in-person team shows a card with categories.
To restore comparability, define the construct as transfers and obligations, develop examples in both languages, align response categories and pilot each mode. Transfer succeeds when the research compares the intended relationship rather than labels that only appear equivalent.
Narrative accounts can preserve local meanings even when quantitative categories require harmonisation.
← Previous · Contents · Next →
8. Misconceptions and limits
- “A trained interviewer has no effect.” Training reduces avoidable variation; social interaction remains.
- “Back-translation proves equivalence.” It detects some wording differences but not lived meaning.
- “The same words make modes comparable.” Presentation and privacy still differ.
- “Adjusting statistically fixes selection.” Unmeasured reasons for choosing a mode can remain.
- “Local adaptation ruins standardisation.” Documented adaptation may be necessary to preserve the construct.
Interviewers can face emotional burden and safety risk. Ethical design includes workload limits, debriefing, privacy and a way to stop an unsafe interview. Participants should know when an interpreter or recording is involved.
← Previous · Contents · Next →
9. Evidence and ethics audit
Audit scripts, translations, audio where consent permits, paradata and interviewer debriefs. Look for unusually short interviews, repeated category patterns, high refusal on sensitive items, one interviewer using many “other” responses, or language versions with different missingness.
| Audit question | Evidence | Warning sign |
|---|---|---|
| Was each version tested? | Cognitive and field-test notes | Translation released untested |
| Was delivery consistent? | Recordings or observations | Frequent private paraphrase |
| Can effects be separated? | Assignment and IDs | One mode equals one interviewer |
| Was consent respected? | Consent and interpreter log | Unannounced third party |
← Previous · Contents · Next →
10. Questions readers often ask
Should interviewer wording ever change?
Only within defined clarification rules, unless the study is explicitly conversational. Record material deviations.
Is one collection mode best?
No. Suitability depends on privacy, access, complexity, population and the claim. Mixed modes require deliberate comparability work.
Can translated answers be pooled?
Yes when the construct and response task have been tested as comparable; otherwise show language-specific results or uncertainty.
← Previous · Contents · Next →
11. Reading routes and ecosystem connections
Begin with the Gold Standard of Anthropology. Use Survey Methodology for instrument testing, Linguistic Anthropology for meaning in social context, and Ethnography for field relationships.
The companion guides on recall periods and event calendars and household definitions and moving denominators cover time and classification. Continue through learning that transfers into life, The Gold Standard and the How X Works hub.
← Previous · Contents · Next →
12. Sources and field checklist
Authoritative guidance includes the US Census Bureau’s standards for developing and pretesting collection instruments, its guideline for translation, its research on mode and interviewer influences on comparability, and its multimode questionnaire guidance.
- Define the construct before translating.
- Pretest every production language.
- Specify neutral probes and clarifications.
- Retain interviewer, language and mode IDs.
- Randomise or balance mode where feasible.
- Audit patterns without blaming staff.
- Protect privacy, consent and interviewer safety.