The Gold Standard of Anthropology | Collaborative Coding, Inter-Coder Disagreement and Interpretive Diversity

1. Start with the analytic purpose

Back to contents

A team studying market traders might code the same sentence as “risk management,” “care for kin” and “distrust of officials.” That difference is not automatically an error. It may show that the passage carries several relations at once.

Before comparing coders, state the analytic approach. A structured content analysis may need a stable codebook and reproducible category assignment. Reflexive thematic analysis treats coding as situated interpretation and does not assume that one context-free code is waiting to be discovered. The quality check must match that purpose.

2. Define the unit before measuring agreement

Back to contents

Coders can agree on meaning yet disagree on where a unit begins. One may code a sentence; another may include the previous question and the participant’s pause. Percentage agreement then mixes unitisation disagreement with category disagreement.

Specify whether the unit is a turn, sentence, paragraph, event or analyst-selected segment. If units remain flexible, use comparison and memoing rather than a coefficient that assumes identical units. Store enough surrounding context to prevent short excerpts from changing meaning.

3. Build a codebook that records boundaries

Back to contents

A useful codebook states the code name, definition, inclusion rule, exclusion rule, typical example, difficult example and relation to neighbouring codes. Version it. A code called “support” is too loose if it combines money, advice, emotional reassurance and institutional access.

Definitions do not remove interpretation. They make the current interpretation discussable. When a disagreement exposes a missing dimension, revise the codebook and record why rather than quietly overwriting earlier work.

4. Choose independent and collaborative phases deliberately

Back to contents

Independent coding can reveal differences that disappear when one confident analyst speaks first. Collaborative coding can then test definitions, surface positionality and develop a shared analytical language.

A practical sequence is orientation, independent coding of a varied sample, comparison, codebook revision, another independent sample and a decision about later coding. Do not require every transcript to be double-coded if the project’s purpose and audit trail do not justify the cost.

5. Classify the disagreement before resolving it

Back to contents

Some disagreements are clerical: one analyst missed a passage. Others concern scope, overlapping codes, different context, translation or theory. A numerical agreement score cannot tell those causes apart.

Use a disagreement log with the excerpt, both decisions, each rationale, disagreement type and final action. Legitimate actions include clarifying a definition, applying both codes, creating a new code, preserving alternative interpretations or escalating an ethical concern.

6. Worked example: disagreement improves the claim

Back to contents

Two analysts independently code 40 fixed excerpts with one primary code. They agree on 30. Raw agreement is 30/40 = 75%. Of the ten disagreements, six concern “family obligation” versus “economic exchange,” three concern segment boundaries and one is a missed code.

The team does not simply target a higher number. It adds an “obligation–exchange overlap” rule, fixes the unit boundary and re-codes a new 20-excerpt sample. They agree on 18, or 90%, but still preserve two theoretically meaningful dual readings. The audit reports both rounds and the conceptual change.

7. Use coefficients only when their assumptions fit

Back to contents

Cohen’s kappa, Krippendorff’s alpha and related measures adjust agreement in different ways and suit different data structures. A coefficient requires defined units, categories and independent judgments. It does not prove that the codebook is meaningful or that missing perspectives were represented.

Report the coefficient, sample, coder count, code prevalence, calculation rule and response to disagreement. Very rare codes can produce unstable or counter-intuitive results. Never cite a threshold without explaining the consequence of misclassification.

8. Changed-context transfer check

Back to contents

Move the worked example to participatory analysis with community advisers who have lived experience but no formal research training. The need for clear units, examples and decision logs transfers. A reliability test may not be the primary goal.

Use accessible excerpts, explain confidentiality, compensate participation appropriately and prevent academic members from treating technical vocabulary as superior insight. A divergent community reading may change the analytic frame rather than count as a coding failure.

9. Limits and common misconceptions

Back to contents

High agreement can result from broad, obvious codes and still produce a shallow analysis. Low agreement can reflect unclear training, but it can also reveal ambiguity worth studying. Consensus can silence junior, multilingual or community analysts when hierarchy is ignored.

Do not claim that multiple coders eliminate subjectivity. Do not convert reflexive analysis into a reliability exercise after the fact. Do not let software defaults decide unit boundaries, missing values or overlapping-code rules.

10. Audit interpretation, not only counts

Back to contents

An audit should connect raw material, codebook versions, coder training, independent samples, disagreement logs, memos, theme development and final claims. Check whether quoted examples include both confirming and boundary cases.

Review whose suggestions changed the framework and whose were repeatedly overruled. If one analyst adjudicated every dispute, report that authority. If interpretations remain plural, show readers how the alternatives affect the conclusion.

11. Frequently asked questions

Back to contents

Must every qualitative project calculate intercoder reliability?

No. It can suit coding-reliability and content-analysis approaches, while reflexive approaches often use collaboration for dialogue and reflexivity rather than reproducibility.

Should disagreements be forced into one code?

No. Apply multiple codes or retain parallel readings when the research question and method support them.

How much data should be double-coded?

Use a purposive sample large and varied enough to test difficult boundaries, then justify the decision rather than applying a universal percentage.

12. The gold standard makes difference usable

Back to contents

Collaborative coding earns its value when disagreement improves definitions, exposes assumptions and sharpens the claim. The goal is accountable interpretation, not a decorative agreement statistic.

Before calling a coding process rigorous, ask what kind of difference the team expected, how it was documented, who could challenge the framework and what the final analysis would lose if every disagreement disappeared.