HOW SINGAPORE EDUCATION WORKS · University qualitative research, coder dialogue and transparent decisions
Find your next learning step
Code independently first, compare the same units, log the disagreement and its reason, adjudicate without erasing either view, then update the codebook and recode affected material when needed.
Full chapter index · Practice and checking · Sources and next reading · How Singapore Education Works
Coding disagreement is not automatically error. It can reveal an unclear definition, an overlapping code, a different unit of analysis or a genuinely ambiguous passage. An adjudication log makes that reasoning visible instead of leaving only a polished final code.
This is a general research-learning workflow, not a substitute for supervisor, ethics, data-management, disciplinary or institutional requirements. Protect participant confidentiality. Store quotations, coder identities and controlled data only in approved locations, and use agreement statistics only when they fit the study’s methodology.
The choices at a glance
| What you notice | Likely issue | Useful next move |
|---|---|---|
| Coders compare only final totals | They cannot locate which passages or decisions differ | Compare the same coded units with stable segment IDs |
| The adjudicator chooses a code without a reason | The audit trail shows authority but not analysis | Record the interpretation, codebook rule and decision rationale |
| A code definition changes but earlier data stay untouched | The dataset contains two coding standards | Identify the affected range and recode or document the boundary |
Full chapter index
See the structure
Work and diagnose
Repair and practise
Recall, transfer and questions
Two coders can read the same passage differently because the codebook is incomplete, the segment boundary differs or the passage supports more than one meaning. The first task is to classify the disagreement before deciding its resolution.
The CDC Field Epidemiology Manual describes qualitative coding as iterative and recommends written definitions and boundary rules to reduce code drift. An adjudication log extends that discipline by showing how the team responded when definitions met real data.
Consensus is not proof of objective truth. It is a documented team decision inside a stated analytic approach.
| Field | Example | Purpose |
|---|---|---|
| Segment ID | INT07-L142–147 | Points to the same approved data unit |
| Coder A | ACCESS | Preserves first independent judgement |
| Coder B | COST | Preserves second independent judgement |
| Disagreement type | Overlapping definitions | Diagnoses the coding system |
| Evidence discussed | Fee and travel-time sentence | Makes the reasoning inspectable |
| Decision | Apply both codes | Records the adjudicated result |
| Codebook change | Add co-coding example to v1.3 | Carries learning forward |
| Recode range | Previously coded interviews 01–06 | Prevents mixed standards |
Keep a stable version number or date for the codebook used during each coding round. Without that reference, later readers cannot tell which definition guided a decision.
Do not place direct participant identifiers in the log. Use approved segment references and controlled access.
Assumptions: two coders independently code the same 20 de-identified response segments using codebook v1.2. They agree on 15 segments and disagree on 5. No agreement threshold was preregistered, so the team treats 15/20, or 75%, as a descriptive check rather than a pass mark.
Adjudication shows that two segments fit Coder A’s code, two fit Coder B’s code and one legitimately needs both codes. The fifth discussion also reveals that the definition of “access barrier” omits travel time, so codebook v1.3 adds an example and requires review of all 20 segments.
Checked accounting under these assumptions: 15 agreements plus 5 disagreements equals 20 compared segments. The five final outcomes are 2 A, 2 B and 1 dual-coded. Because the definition changed, reviewing only the five disagreements would be incomplete; earlier agreements may also contain travel-time material.
The log records the original coding, final coding, definition change, responsible reviewers and recode completion date.
If coders selected different text spans, a code comparison can look like substantive disagreement when the first problem is segmentation. Align the unit or record boundary disagreement separately.
If a senior researcher announces the answer before junior coders explain their readings, consensus may reflect hierarchy. Let each coder state evidence and codebook reasoning before adjudication.
A high percentage can hide systematic disagreement on the most important theme. Examine which codes and passages differ, not only the overall number.
- Preserve both independent codes and the codebook version.
- Classify the disagreement: segment, definition, overlap, omission or interpretation.
- Discuss the same evidence without exposing unnecessary participant information.
- Record the decision, rationale and any unresolved minority view permitted by the method.
- Propagate codebook changes through the defined recode range, memos and outputs.
CDC guidance treats coder comparison as a way to improve dialogue, reflection and codebook quality, not merely to manufacture a score. The repair therefore includes both the row-level decision and the system-level change.
If the disagreement affects a published claim, sampling decision or ethics issue, follow the project’s escalation and reporting route.
Guided
Students compare two coded examples, identify the disagreement type and select the missing log field.
Shared
Pairs code the same short, de-identified dataset independently, then adjudicate while a third person checks that reasons and codebook changes are recorded.
Independent under governance
The researcher maintains the log, versions the codebook, completes required recoding and reconciles the final coded dataset with analytic memos and reported methods.
A month later, sample three final codes and trace each back to the original segment, independent judgements, adjudication and codebook version.
Qualitative traditions differ in whether they seek coder agreement, reflexive interpretation or another form of analytic rigour. A workflow must fit its methodology. The transferable principle is transparency about who interpreted what, under which definitions and how changes affected the analysis.
Changed-context transfer: apply the log to image annotation or multilingual coding, adding fields for translation choice, annotator expertise or label ontology only when those are relevant.
Must every project calculate Cohen’s kappa?
No. Choose an approach that fits the research design and approved protocol. A statistic does not replace discussion of the disagreements.
Should coders discuss before coding?
They should be trained on the codebook, but independent coding must remain genuinely independent when the design uses it.
Can the codebook change after coding starts?
Often yes, depending on the method. Version the change and decide which earlier material must be reviewed.
Should the log be public?
Only if governance and confidentiality permit. A public summary can omit sensitive text while describing the process.
Sources and next reading
- CDC — Collecting and Analyzing Qualitative Data
- CDC — Practical Strategies to Enhance Qualitative Analytic Rigor
- CDC Q-Notes — Analysis and Audit Trails
Continue inside eduKateSG
- University open-ended response codebooks
- University preregistration deviation logs
- Laboratory notebooks and research records