A dataset does not become labelled merely because someone typed a word beside each example. It becomes labelled when a community has decided what the labels are allowed to mean.
Annotation schemes are the rules that turn complex observations into structured labels. They define the unit being annotated, the allowed categories, the boundary between categories, how multiple labels can coexist, how uncertain cases are handled and how disagreements are resolved.
This article sits beneath How Classification Works, How Metadata Works and How Ground Truth Works. Classification owns the broad logic of categories. Metadata owns structured description. Ground Truth explains why labels used for evaluation still need provenance and uncertainty. Annotation schemes own the operational bridge from those abstractions into repeatable labelling work.
1. First Decide the Unit of Annotation
Are we labelling a whole document, one sentence, one word, one image, one bounding box, one event, one person, one relationship or one time segment?
The unit changes the task. A news article can be labelled by topic at document level while individual claims inside it are labelled for evidence status. An image can be labelled as “contains a car” or annotated with the exact pixels belonging to the car.
If the unit is vague, annotators can agree on the category and still disagree on what object the category applies to.
2. Categories Need Operational Definitions
“Positive,” “unsafe,” “sarcastic,” “urban,” “misinformation,” “mastered” and “damaged” sound like labels until two competent annotators apply them differently.
An annotation guide therefore needs inclusion rules, exclusion rules, examples, counterexamples and edge cases.
The stronger question is not “what does this word usually mean?” but what observable conditions cause this dataset to assign this label?
3. Mutually Exclusive and Multi-Label Schemes Solve Different Problems
Some tasks require exactly one category. Others allow several.
A photograph can contain a bus, pedestrian, road sign and building simultaneously. A document may belong to education and technology. Forcing one label can destroy real structure.
Conversely, allowing unlimited labels can make the system too loose to compare reliably. The scheme should follow the ontology of the task rather than convenience.
4. Hierarchies Can Preserve Coarse and Fine Categories
A dog is an animal. A sedan is a vehicle. Algebra is mathematics.
Hierarchical annotation can preserve these nested relations so a dataset supports both broad and fine-grained analysis.
The risk is inconsistent depth: one annotator chooses “animal,” another chooses “golden retriever.” The guide should define the required resolution.
5. Boundary Rules Matter
In text annotation, where does an entity begin and end? Does “National University of Singapore” include “the”? In image segmentation, does a shadow belong to the object? In speech annotation, do pauses belong to the preceding utterance or following one?
Boundary rules are often more difficult than category names because they determine the geometry of the labelled object.
6. Ambiguity Should Be Represented, Not Hidden
Some cases are genuinely ambiguous.
A strong scheme can allow “uncertain,” confidence scores, multiple acceptable labels or adjudication flags. Forcing every case into one crisp category creates false certainty and can teach downstream models that the world has cleaner boundaries than it really does.
7. Annotator Training Is Part of the Measurement System
Annotators need more than a one-page definition.
Training examples, calibration rounds, feedback and difficult-case discussion help different people internalise the same operational boundary.
This makes annotation analogous to measurement: the human annotator is part of the instrument chain.
8. Agreement Measures the Scheme as Much as the People
If competent annotators disagree often, the problem may be poor training. It may also be that the category is intrinsically fuzzy or the guide is underspecified.
Inter-annotator agreement should therefore be diagnostic rather than punitive. Low agreement is evidence about the task design.
9. Adjudication Creates the Final Label — but Also a Record of Disagreement
When annotators disagree, a senior reviewer or consensus process can assign the final label.
The disagreement itself should not vanish. It is evidence that the case sits near a boundary, that the rule may need revision, or that future users should treat the label with caution.
10. Label Leakage Can Corrupt Evaluation
If annotators know the model’s prediction while creating the “ground truth,” their judgment can be influenced by the system being evaluated.
If training and test labels are created from the same downstream artefact, the evaluation can become circular.
Annotation pipelines should therefore preserve independence where the evaluation requires it.
11. Annotation Schemes Drift Over Time
Categories can change as institutions, language and technology change.
A content policy can add a new prohibited category. A medical definition can be revised. A product taxonomy can split one category into several.
Version the scheme. Otherwise labels produced under different definitions can be compared as though they mean the same thing.
12. Worked Example: Named-Entity Annotation
In the sentence “Apple opened a store in Singapore,” the annotator may label “Apple” as an organisation and “Singapore” as a location.
The scheme must specify whether brands count as organisations, whether geopolitical entities and cities use separate labels, and how nested names are handled.
Those choices later affect entity-resolution and knowledge-graph systems.
13. Worked Example: Educational Mastery
A teacher labels student responses as “mastered,” “partial” or “not mastered.”
The labels are only useful if the rubric says what evidence counts: correct answer alone, correct method, explanation, transfer to a new form, or delayed retrieval.
Different definitions create different datasets even when the student responses are identical.
14. Worked Example: Image Safety Annotation
An image can be labelled by content type, age context, severity, uncertainty and whether the depiction is documentary or instructional.
A scheme that collapses all these distinctions into one binary label may be easier to operate and less useful for nuanced policy.
15. A Strong Scheme Has a Feedback Loop
Difficult cases should return to the scheme designers. Repeated disagreement should trigger better definitions. New real-world cases should test whether the categories still cover the domain.
The annotation guide is therefore a living operational document rather than a finished dictionary.
16. An Annotation-Scheme Checklist
- Define the annotation unit.
- Define every label operationally.
- Provide positive, negative and boundary examples.
- Specify single-label, multi-label or hierarchical rules.
- Define span, temporal or spatial boundaries.
- Represent uncertainty explicitly.
- Train and calibrate annotators.
- Measure disagreement and investigate its source.
- Adjudicate difficult cases with traceable reasons.
- Version the scheme when definitions change.
- Preserve label provenance so downstream users know how the data was produced.
17. Read the Mechanism Forward, Backward and Sideways
Forward: raw observation → annotation unit → scheme rules → human or machine label → adjudication → structured dataset. Backward: start from a suspicious downstream model error and ask whether the label itself came from an unstable boundary. Sideways: compare annotator, domain expert, data engineer and model builder. Each sees a different failure mode in the same label.
18. The Civilisation Lesson
Modern civilisation increasingly converts messy reality into machine-readable categories. Those labels shape search, medicine, research, education, regulation and AI.
The responsible system remembers that the label is a representation created under rules. Better labels come from better boundaries, better disagreement handling and better provenance — not from pretending complexity disappeared when the category name was assigned.
An annotation scheme is a contract for turning reality into labels without allowing the labels to impersonate reality itself.
Continue through How Classification Works, How Ground Truth Works and the master How X Works hub. Next: entity resolution — how records using different names are determined to refer to the same underlying thing.