eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Training Label Quality — Annotation Disagreement, Ambiguity and Repair

—

by

eduKate Asia
Three students study around open books and worksheets in a shared classroom.

Training-label quality is the degree to which the answers attached to examples represent a clear, relevant and defensible target for a machine-learning task. A classifier can optimise its loss perfectly against labels that were rushed, ambiguous, inconsistent or simply wrong, and still fail the user’s real need. To improve an AI dataset, define what each label is supposed to mean, train annotators on explicit evidence rules, compare independent judgments, investigate disagreements and document which examples cannot be resolved. A fluent model prediction is not evidence that its training labels were reliable.

The practical method is to audit the label before blaming the model. This chapter starts with an eighty-record teaching set marked independently by two fictional reviewers, calculates raw agreement and a chance-adjusted statistic, diagnoses fifteen disagreements, repairs resolvable cases and protects a separate evaluation set. You will see why 81.25% annotator agreement is neither 81.25% ground-truth accuracy nor permission to force an answer onto uncertain evidence. The same reasoning helps people check AI-generated study summaries, document classifiers and the human labeling work that sits behind automated decisions.

This is a specialised chapter in How AI Works, the general explanation of artificial intelligence from data and models to tested outputs. It differs from agent reviewer calibration, which concerns reviewers of live agent outcomes: here we investigate the labels used to build and evaluate a predictive model. Follow the How Intelligence Works hub when the question becomes how people perceive ambiguity and combine evidence; follow the Super Intelligence engineering library for operational agents. No hypothetical artificial superintelligence is presumed to exist.

Choose a reading route

  • Building your first classifier? Read the target, annotation contract and numerical laboratory before thinking about model architecture.
  • Managing a dataset or annotation team? Focus on the disagreements, adjudication protocol and versioned labels.
  • Teaching Mathematics, English or Science? Use the evidence examples and transfer exercise to discuss what a marked answer actually proves.
  • Auditing high-consequence AI? Prioritise evaluation separation, population coverage, uncertainty and corrections.

Contents

1. A label is a rule applied to evidence

A picture, sentence, audio clip or spreadsheet row is not born with a machine-learning label attached. A person or measurement process has to map an observation to a category, quantity or judgment. That mapping answers a question. Does this message request a timetable change? Does the photograph contain a bicycle? Is the provided explanation supported by the named source? Each of those is a different label task, and an annotation rule that works for one can become misleading in another.

Suppose a student asks, “Could I move my lesson if I finish the project early?” One annotator marks “reschedule request” because the sentence mentions moving a lesson. Another marks “conditional question” because the student has not yet instructed anyone to change the appointment. Neither is necessarily careless. The task definition failed to specify whether a hypothetical question counts as a present request. A model trained on that ambiguity may produce confident but unstable distinctions. More annotator instructions cannot repair a label target that remains conceptually confused.

A label also depends on how evidence was collected. A message excerpt without the previous sentence may be ambiguous, while the complete exchange makes the intent plain. A photograph cropped tightly around an object may exclude the relevant context. A transcription may omit a negation. Data quality therefore includes record selection, source provenance, missingness, language context and access permissions. Treat a disputed label as a clue about the representation and task contract, not only the individual who clicked a button.

Google’s Machine Learning Crash Course guidance on data characteristics describes label errors, unreliable features and sampling concerns as components of data quality. Its categorical-data guidance discusses human raters and inter-rater agreement. These sources support the mechanism: a dataset is fit for purpose only when its labels and examples are assessed against the task, rather than assumed clean because they have been encoded numerically.

2. Define the target before collecting examples

Define the unit of analysis. Will one label describe an entire conversation, a single message, a sentence within it or an action the user explicitly approved? The choice changes what counts as positive. A model judging complete helpdesk cases may need context that would be inappropriate or unavailable for a model judging one message in isolation. If developers mix units during data collection, the model learns the annotation process’s inconsistency as though it were a property of the world.

Define the positive and negative categories through observable conditions, not synonyms. A positive “booking-change instruction” might require the user to identify an existing appointment, ask for an actual change now and satisfy the designated permission pathway. An inquiry about policies is not a change instruction. A quotation of someone else’s instruction is not permission. A sentence that requests a change for another person without verified authority is neither a safe action nor necessarily a negative intent; label the intent and authority separately if they are different tasks.

Explain how uncertainty is represented. Some tasks require one category; others legitimately support multiple categories or probabilistic judgments. A forced yes/no choice on genuinely incomplete context creates artificial noise. An “insufficient context” or “defer” category can be appropriate when it has an explicit definition and handling path. The point is not to make every hard case disappear into uncertainty. It is to prevent the training process from inventing a ground truth that the source material does not establish.

State the population and use. A classifier trained on short English requests from one study centre may not generalise to spoken requests, translated messages or other institutional processes. The positive class may be rare in live traffic even if it was deliberately balanced during collection. Record what is known about source distribution and what is missing. A label contract that includes only easy, tidy cases is a poor basis for claiming robust performance on diverse real requests.

3. Write an annotation contract

An annotation contract should contain a reader-friendly statement of the task, an operational decision rule, inclusion and exclusion examples, rules for quoted text and partial context, treatment of negation and conditional language, permitted source fields, an uncertainty route, privacy instructions and an appeal procedure. Provide border cases with explanations. The useful feature is not length; it is that two qualified readers can apply the same rule to the same evidence and explain why they differed.

Version the contract. If the instruction changes from “any interest in rescheduling” to “a present authorised request for an appointment change”, the old and new labels should not be merged as though they mean the same thing. Record which guide version each label used, the source example version and whether a change requires reannotation. If only new examples use a stricter definition, apparent model drift may actually be a change in what the team means by a positive example.

Use training examples that challenge the boundary. “Please move Tuesday’s lesson to Thursday” is clear intent, assuming the person may request a change. “Can lessons ever be moved?” is a policy inquiry. “My friend asked you to move her class” is a report about someone else’s instruction. “Do not move my lesson” contains the keyword move but expresses the opposite action. If an annotation process relies on keyword spotting, these examples expose it immediately.

Give annotators an evidence field that captures a short reason or span when feasible, without requiring them to copy confidential material into an unsecured spreadsheet. The reason can be a controlled code: explicit imperative, modal question, quotation, contradiction, missing context or unresolved. Those codes help adjudicators identify whether the disagreement stems from task wording, individual error or source insufficiency. They also make audit sampling more informative than a single average agreement figure.

4. Independent raters and qualified disagreement

Independent marking means that the first rater has not seen the second rater’s choice before recording an initial judgment. If two reviewers consult each other for every case, their high agreement could reflect social influence rather than a clear label rule. A pair of independent labels offers an opportunity to detect boundary ambiguity. Keep the raw judgments, guide version and controlled adjudication separate, so the final label does not erase evidence that the task was originally difficult.

Raters need relevant competence, appropriate language and access to enough context. Expertise alone does not remove ambiguity; two qualified people may still adopt different interpretations of a vague rule. Conversely, forcing unanimity may hide reasonable differences about inherently subjective tasks. Decide whether the dataset needs a single agreed operational definition, a distribution of judgments or a specialist determination. The right choice depends on what the resulting model will be asked to do.

Some metrics adjust agreement for the frequency with which raters choose categories. Cohen’s kappa is one such statistic for two raters on categorical labels, but it depends on assumptions about label categories and expected agreement, and it can behave unexpectedly with skewed class prevalence. Agreement statistics do not validate the underlying source, reveal bias against a group automatically or prove that a model will generalise. Use them as diagnostics alongside observed error types and case-level review.

Do not confuse annotation quality with reviewer productivity. A labeler who processes eight hundred easy examples in an hour may be useful for screening, but the benchmark should not be “maximum labels completed” when nuanced disagreements determine safety. Sample hard cases intentionally; provide breaks, training and appeal routes; and avoid incentivising people to click the majority category simply to obtain a green agreement score. Good annotation is a measurement process requiring attention to the measurand.

5. Worked eighty-record classification laboratory

Imagine the fictional Willow Study Desk is preparing a training dataset for a model that classifies whether a message contains a present request to change an existing appointment. It draws eighty examples from a synthetic collection created for teaching. The messages are invented, so they contain no actual student data. Two trained raters, A and B, receive the same version of the labeling guide and work independently. They may choose positive or negative on the initial pass; the later adjudication stage will allow an explicit unresolved state.

The two-by-two agreement table has four cells. Both raters marked positive on thirty records. Both marked negative on thirty-five. A marked positive while B marked negative on nine. A marked negative while B marked positive on six. Check the total: 30 + 35 + 9 + 6 = 80. It is easy to misread the number fifteen in the off-diagonal cells as fifteen factual mistakes. What the table actually shows is fifteen initial differences in judgment. Without reference evidence and a defined target, we cannot say which rater, if either, was correct on those examples.

  • A positive, B positive: 30 messages; initial agreement.
  • A negative, B negative: 35 messages; initial agreement.
  • A positive, B negative: 9 messages; initial disagreement.
  • A negative, B positive: 6 messages; initial disagreement.
  • Total: 80 distinct messages, 65 agreements and 15 disagreements.

Before calculating statistics, read several borderline examples. One says “Please reschedule my class to 4pm.” Both mark positive. Another says “Are Thursday lessons usually available?” Both mark negative, because availability is not a present change instruction under the guide. A third says “I might move my lesson if I finish early.” A marks positive based on the word move, B marks negative because the statement is conditional. A fourth contains a truncated reply, “Yes, that works,” but the preceding request is missing. Different raters can legitimately reach different interpretations when they are not told whether history should be available.

The reviewers group the fifteen disagreements by apparent source after looking at a permitted explanation of each case. Nine arise from the unclear boundary between hypothetical interest and a present instruction; four are caused by missed context or insufficient snippets; two appear to be defective source records, including a missing negation. This grouping is a reasoned diagnosis of the fictional cases, not a frequency estimate for real educational messages. The categories guide different remedies: clarify the rule, re-fetch the right context or fix the underlying record. A generic instruction to “try harder” would not repair the source process.

A senior reviewer does not simply replace every minority choice with the majority decision. There is no majority in a two-rater disagreement. After consulting the revised contract and available source evidence, the adjudication panel can support seven of the fifteen as positive and five as negative. Three remain genuinely unresolved because the required context is unavailable. The final defensible distribution is thirty shared positives plus seven resolved positives equals thirty-seven; thirty-five shared negatives plus five resolved negatives equals forty; three are unresolved. Thus 37 + 40 + 3 = 80. The unresolved records receive an explicit treatment rather than a fabricated binary answer.

The repaired dataset has more useful information than a single column of class labels. The team now has initial raters, adjudicated result, guide version, evidence category, uncertainty state and source defect flag. It can decide to exclude the three unresolvable records from binary training while retaining them, under appropriate access controls, as examples of an uncertainty boundary. Exclusion should be reported. If the cases occur disproportionately in one language or user group, simply dropping them might reduce coverage and reinforce unequal performance.

6. Recalculate agreement and Cohen’s kappa

Observed agreement is the forty? No—the relevant diagonal cells are thirty plus thirty-five, so 65 of 80. Divide 65 by 80 to obtain 0.8125, or 81.25%. The remaining fifteen of eighty represent 18.75% disagreement. Rater A’s positive total is 30 + 9 = 39 and negative total is 35 + 6 = 41. Rater B’s positive total is 30 + 6 = 36 and negative total is 35 + 9 = 44. Each rater’s marginal totals add to eighty. If your arithmetic gives a different result, return to the four cells before evaluating any model.

For Cohen’s kappa, estimate chance agreement from those observed marginal label proportions. Expected positive-positive agreement is (39 ÷ 80) × (36 ÷ 80) = 1,404 ÷ 6,400. Expected negative-negative agreement is (41 ÷ 80) × (44 ÷ 80) = 1,804 ÷ 6,400. Sum them: 3,208 ÷ 6,400 = 0.50125, or 50.125%. This expected-agreement calculation assumes a particular independence model based on the observed category frequencies. It is not evidence that the raters literally guessed at random.

Kappa equals (observed agreement minus expected agreement) divided by (one minus expected agreement). Substitution gives (0.8125 − 0.50125) ÷ (1 − 0.50125) = 0.31125 ÷ 0.49875 ≈ 0.6241. We can report observed agreement 81.25% and Cohen’s kappa approximately 0.624 for this invented two-rater, two-category packet. Avoid labeling that number “62.4% accurate” or claiming an empirical threshold for adequate safety. Kappa is a chance-adjusted rater agreement statistic under its definitions, not a test against an independently established truth.

Why report both? Imagine almost all records were negative and both raters usually clicked negative. Raw agreement might be high even if the rare positive cases were the ones missed. Kappa’s marginal adjustment can surface a different perspective, but it too has prevalence-related limitations. A practical report includes the confusion table, positive and negative agreement, class distribution, uncertainty cases, boundary examples and any known source-quality defects. A single number cannot explain the shape of disagreement.

There is another denominator problem. After adjudication, seventy-seven of eighty examples have a resolvable binary answer, while three do not. The resolvable share is 77 ÷ 80 = 96.25%. That number says nothing about whether those seventy-seven labels are unbiased, generalise to future messages or predict real user authorisation. The team should describe exactly what “resolved” means—agreement with the documented contract under accessible evidence—not treat it as a universal validation certificate.

7. Resolve differences without inventing certainty

Repair starts by identifying who can change each part of the process. An annotation lead may revise the guide and retrain reviewers. A source-data owner may repair a truncated message record. A privacy officer may restrict access to sensitive fields needed for an adjudication, which can leave some cases unresolved. A model engineer can retrain a classifier on the corrected, versioned data. Those are separate responsibilities; a model’s own generated rationale should not become the authority that decides which original human label was correct.

A sound adjudication meeting presents each disputed case with its original evidence, both initial labels, guide version and the precise question that must be decided. Raters should first explain the evidence rather than defend a click. The panel records the final operational determination or an unresolved status, together with a reason code. If the guide itself is changed, the panel asks which earlier records need retrospective review. Otherwise the training set contains labels assigned under two incompatible definitions.

The four missed-context examples demonstrate why collecting more labels may be less useful than collecting better source material. If a record says “Yes, please,” without the preceding message, no amount of confidence by an annotator can prove what was being approved. The remedy may be a lawful retrieval of prior context with access controls or a decision that the case is unlabelable. Avoid turning unavailable information into a guessed label because the training pipeline demands a value.

The two defective-source cases need a different repair. A missing “not” reverses an instruction’s meaning. If the original message is available through an authoritative versioned channel, correct the derived transcript and preserve the relationship to the prior defective record. If the original is gone, document the limitation and exclude the example from hard-label training. Source changes should flow through future data loads; otherwise the next indexing job may reintroduce the defect.

Check the repair outside the original fifteen examples. Construct a separate, fresh set of boundary cases not used to write the revised guide. Ask independent reviewers to label them without seeing the adjudicated answers. Compare error types and observed agreement, not only an average that can be inflated by adding many easy negatives. A corrected guide that performs well on the exact examples it was written to solve may still fail on new phrasing, other languages and atypical users.

For high-consequence deployments, involve people who understand the affected context. A schooling message that mentions a student’s disability or a payment dispute should not be casually categorised from incomplete snippets by an unqualified annotator. Consider whether the task should be automated at all, whether an assistive rather than decisive output is appropriate, and what affected people can challenge. AI training data is a sociotechnical design choice, not merely a spreadsheet-cleaning exercise.

8. Keep training labels separate from evaluation

Never claim improved model performance merely because the same revised labels were used both to train the model and to score it. Training examples shape parameters; validation examples guide choices such as architecture and threshold; a final held-out test is intended to give a less biased estimate of generalisation after design decisions have been made. Exact partitioning depends on the project, but no item should leak into a purported independent test through duplicates, near duplicates, source overlap or an alternative export.

Imagine eighty workshop messages are only a small annotation audit, not the complete production dataset. A later model can train on a separate, suitably representative corpus. The team should retain a distinct set of untouched messages for final evaluation and provide annotation instructions that are consistent with the training target. If multiple messages originate from the same conversation or participant, splitting records at random may allow information leakage between training and test. Choose grouping and temporal rules aligned with the real question of generalising to new cases.

Evaluation labels require independent scrutiny too. A test set with unresolved human disagreement should not be treated as perfect ground truth merely because a file name contains gold. For some tasks, a qualified consensus decision is defensible. For others, reporting the distribution of reasonable judgments is more truthful. The final metric should state how disputed examples were handled and whether performance differs on the ambiguous subset. Otherwise a system that echoes an arbitrary adjudication convention appears more objectively correct than it really is.

Version the provenance of both training and test labels. Record which source samples, guide versions, raters, adjudication rules, exclusions and corrections produced the data. A later evaluation that changes its label policy has changed the target, so results cannot be compared naively to a previous release. This is one reason Datasheets for Datasets, published in Communications of the ACM in 2021, proposes structured documentation of datasets’ motivation, composition, collection and uses. Documentation is not proof of quality, but it helps reviewers ask the right questions.

9. Diagnose label failures at the source

Failure A: the model appears to miss conditional language. Inspect whether the annotation rule treated conditional questions inconsistently. Compare false positives on a controlled set of “might,” “if” and “could” examples. Repair by specifying the intended speech act and retraining on correctly interpreted examples rather than adding an arbitrary keyword exception. Retest with new phrasings. The lesson is that an apparent model error may be a target-definition error inherited from human labels.

Failure B: agreement rises suddenly after a meeting. Check whether raters began sharing answers or were informally told the expected majority choice. Correlated judgments can inflate apparent agreement without increasing validity. Restore independent initial marking for a new sample and separate subsequent deliberation from original ratings. Keep the log of when and why guidance changed. A metric without process provenance is easy to misread.

Failure C: a classifier performs well in development but poorly on live messages from families using a different language mix. Diagnose source coverage, translation loss, changed task distributions and label guide clarity in the affected contexts. Extend the sampling and qualified review rather than declaring the users anomalous. If new linguistic evidence reveals a category ambiguity, revise the guide with a clear change log and evaluate whether the model ought to abstain on that group of cases.

Failure D: reviewers resolve every disputed record after an administrator demands a one-hundred-percent complete dataset. Examine whether missing context was invented or uncertainty suppressed. Reintroduce an explicit insufficient-evidence status with an actionable path. A dataset with three unresolved examples and honest documentation can be superior to a seemingly immaculate dataset whose edge cases were forced into unsupported labels.

Failure E: accuracy scores rise when a test set is relabeled using model suggestions. This risks contamination by the candidate model being evaluated. Re-establish independent ground-truth procedures, log each source and reviewer, audit changed examples and maintain a protected final evaluation split. AI assistance in annotation is not inherently prohibited, but it must be described as an influence on the measurement process rather than hidden inside “human gold labels.”

10. Changed-context transfer problem

Your next project classifies whether an audio transcript contains an explicit request to cancel a library booking. Two raters independently assess sixty synthetic clips. They agree positive on twenty, agree negative on twenty-five, and disagree on fifteen: rater A positive/B negative on ten, A negative/B positive on five. Calculate raw agreement. Then explain why transcriber omissions of the word “not” require a different fix from a policy disagreement about whether “Could I cancel?” is a present instruction.

Worked solution: diagonal agreement is 20 + 25 = 45. Therefore observed agreement is 45 ÷ 60 = 0.75, or 75%, and disagreements are 15 ÷ 60 = 25%. The two causes must be separated. A missing negation is a source-fidelity problem, so the original audio must be checked through an authorised, privacy-respecting path and the transcript corrected or the example excluded. Different judgments about the speech act need a clarified labeling contract, examples and independent retest. Neither can be repaired by telling a classifier to be more confident.

Now change one further condition: thirty of the sixty clips are from an unfamiliar variety of spoken English that neither reviewer understands well. Their agreement may remain 75% if they both apply the same wrong heuristic, but the result no longer provides adequate evidence of label validity for that group. Recruit appropriate linguistic expertise, examine consent and representativeness, sample subgroup error patterns and consider an uncertainty route. The transfer principle is that two people agreeing is useful evidence of a shared rule, not a guarantee that the rule captures the world.

A useful independent practice question is to redesign the annotation sheet with five columns: stable clip reference, initial label, confidence or uncertainty reason, evidence span and guide version. Then propose a sixth column for adjudication that does not overwrite the initial decision. Test the sheet on a negated request and a hypothetical question. The goal is to preserve evidence and repair the point of failure; it is not to produce a prettier data file.

11. Limits, rights and responsible use

No labeling workflow can turn subjective social categories into objective physical facts merely by standardising them. Some categories are policy choices, not discoveries. Others are unobservable from the available data or change with context and time. Inter-rater agreement statistics have assumptions and sampling limits; consensus can hide power imbalance; and labels assigned with one purpose may be harmful when repurposed. Explain the target’s social and technical boundaries before making deployment claims.

Protect the people who supply examples and those who label them. Minimise personal details, define access restrictions and retention, avoid exposing private messages to unnecessary workers, and obtain the permissions the use case requires. Raters should receive clear instructions, realistic time, training and an escalation route rather than punitive pressure to match a majority. A dataset’s label-quality report is also a record of decisions about people; it deserves accountable oversight.

AI systems can assist with sorting, suggestions and discovery of questionable labels, but they should not be described as revealing a hidden absolute truth. Independent evaluation must retain power to find failures in the assistant’s suggestions. Use human adjudication where expert judgment is required, preserve uncertainty and recheck after data or distribution changes. These controls apply to today’s machine learning systems; they do not demonstrate artificial general intelligence or artificial superintelligence.

Frequently asked questions

Does 81.25% agreement mean 81.25% of the labels are correct?

No. Agreement compares two raters with each other, not either rater against an independent truth. They may agree on a shared mistake, especially when the source or label definition is poor. Assess validity through task rules, qualified adjudication and independent evidence.

Why calculate Cohen’s kappa if raw agreement is simpler?

Kappa adjusts observed agreement for agreement expected from the raters’ marginal category frequencies under its assumptions. It gives a different diagnostic view, but it is sensitive to class balance and is not a universal measure of dataset fitness.

Can unresolved records stay in a dataset?

Yes, if their purpose and handling are explicit. They may be excluded from hard-label training, used to test abstention or analysed as evidence about insufficient context. Do not silently assign guessed ground truth to make a table complete.

Does the term gold label guarantee correctness?

No. It often denotes a reference label used for training or evaluation. Review how it was obtained, whether qualified independent judgment was involved, and whether the target is sufficiently well defined.

Will thousands more labels fix disagreement?

Not necessarily. More examples can amplify a vague rule or a systematic source error. Repair the definition, collection and adjudication process before scaling volume. Sample difficult and underrepresented cases, not only easy negatives.

Can students use this method without private datasets?

Yes. Invent a small set of fictional questions, ask two people to classify them independently, compute their agreement, discuss the boundary cases and write a better rule. Keep examples non-sensitive and make the correction test different from the examples used to design the rule.

Primary references and reading

Learning and career pathways

For hands-on ML learning, return to How AI Works to place labels inside data collection, training, evaluation and inference. For the difference between a model score and an accepted result, read AI evaluations and agent test oracles. For the larger human skill of thinking about evidence and uncertainty, use How Intelligence Works, the How X Works library and the Gold Standard of Education. The transferable lesson is to define what you are measuring before treating a number as a fact.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading