The Gold Standard of Computational Linguistics

Computational Linguistics builds explicit models of language and examines what those models can explain or do. A useful starting question is smaller than “Does the computer understand?” Ask whether a system can distinguish who performed an action, identify the intended referent or preserve a qualification when translating.

The gold standard is a traceable connection between a language question, a representation, a procedure and evidence. The output needs a defined interpretation. A label, parse, probability and generated paragraph are different kinds of answer; none should silently stand in for all the others.

Imagine a fictional school-library assistant that sorts messages requesting a reservation. Its task is to identify a new request, not to infer a pupil’s ability, personality or reliability. The same sentence might be useful evidence for the first task and entirely inadequate for the others.

This guide uses original toy messages and stipulated counts. They are teaching exercises, not results from an implemented system. The aim is to make assumptions and failure modes visible before a larger model makes them harder to inspect.

Continue into: Linguistics root.

Contents · Next chapter

Linguistics asks how language is organised and used. A computational investigation makes some of those ideas precise enough to implement or test. Natural language processing emphasises tasks performed with language data. The activities overlap; a rigid boundary would conceal useful connections.

For the library assistant, classifying a message is an application task. Comparing whether word order or grammatical roles better predicts the right label is an investigation. Asking whether a system’s behaviour demonstrates a human-like mechanism requires additional evidence.

A successful application can rely on a shortcut. If most reservation messages in a training collection begin with “Please”, a classifier may learn that cue without representing the request. A message that politely declines a reservation can reveal the gap.

Keep a claim ledger: what was measured, which examples were available and what remains untested. An impressive answer is worth inspecting, but it does not remove the need to say exactly what the answer establishes.

Contents · Previous chapter · Next chapter

Write the task contract first. Input: a message from a consenting participant in a fictional classroom exercise. Output: NEW REQUEST, NOT A NEW REQUEST, or NEEDS REVIEW. Decide whether conversation history is available and what to do when it is missing.

“Please reserve the atlas” is a new request. “You reserved the atlas yesterday” reports an earlier event. “Could you reserve it?” may be a request whose book cannot be resolved without a previous message. One label cannot represent both intent and the missing object unless that is explicitly the design.

Separate classification from execution. Recognising a reservation request does not authorise an automated system to place one. A classroom project can display a suggested label without connecting to a real borrowing account.

The contract should also identify exclusions. Do not score spelling as intent, treat every question as a request or force a book name from insufficient evidence. A defined review state makes uncertainty observable rather than hiding it inside an arbitrary guess.

Contents · Previous chapter · Next chapter

A corpus is a collection of language material assembled for a purpose. For this exercise, create fictional messages with variations in wording, politeness, negation, tense and reference. Keep a record of which phenomenon each message is intended to test.

Suppose the collection contains twenty near-copies of “Please reserve the atlas” and one indirect request. High performance could mainly reflect the repeated template. Count distinct situations as well as message totals before declaring that the collection is varied.

Do not scrape classmates’ messages or upload private borrowing records for convenience. Synthetic classroom examples avoid those data in the first place. In a real project, permission, collection limits, retention and access require a suitable governance process.

State the scope honestly: English text, a few library intents, constructed examples and no claim of representative deployment performance. A small transparent collection is more useful for learning than a large unexplained pile whose provenance nobody can reconstruct.

Continue into: Corpus linguistics and concordance.

Contents · Previous chapter · Next chapter

Tokenisation defines the units processed by a model. Depending on the method, units can be whole words, subword pieces or other segments. A token is not automatically a morpheme or a meaningful word. The Words and Tokens chapter linked below explains these distinctions.

For an explicit teaching convention, split “Please reserve the atlas.” at spaces and separate the full stop. That produces five units: Please, reserve, the, atlas, and punctuation. Lowercasing would change the representation again; record it rather than treating it as invisible cleaning.

Compare “reserve”, “reserved” and “reservation”. A system that matches only the first string will miss related forms. Yet collapsing them all to one unit could erase the difference between an action request and a report about an existing reservation.

Try two tokenisation conventions on the same five sentences. Write down what changed, what information remained recoverable and which downstream decision could be affected. That is a clearer experiment than assuming one token count is an intrinsic property of the sentence.

Foundation reading: Jurafsky and Martin: Words and Tokens.

Continue into: Morphological awareness · Tokenisation vocabulary.

Contents · Previous chapter · Next chapter

Annotation assigns labels according to a scheme. For this exercise, NEW REQUEST means the speaker is asking the recipient to make a fresh reservation. It does not include a report of a completed action or a quotation of somebody else’s request.

Two readers independently label “The teacher said, ‘Please reserve the atlas.’” One may focus on the quoted words; the other may focus on what the present speaker is doing. Their disagreement reveals an underspecified rule, not necessarily an inattentive reader.

Revise the rule to identify the level being labelled: the present message’s communicative act. If reporting a quotation is NOT A NEW REQUEST under that contract, record the decision and preserve a note about the embedded request.

A gold reference label is an annotation decision, not an infallible revelation of intent. Keep unclear cases, adjudication notes and scheme versions. A model should not receive credit for matching an answer key whose interpretation changes silently halfway through an evaluation.

Contents · Previous chapter · Next chapter

A parse represents grammatical structure. Constituency groups expressions into phrases; dependency analysis relates words through grammatical links. These are representations with conventions, not a guarantee that every ambiguity has disappeared.

Compare “Mira reminded Arun” with “Arun reminded Mira”. Both contain the same names and verb, but the subject and object exchange places. A bag-of-words representation cannot distinguish the pair by word counts alone.

Now compare “Mira was reminded by Arun”. Its surface subject is Mira, while Arun is the person doing the reminding. A system that equates subject with agent will fail this contrast even if it labels the words correctly.

Universal Dependencies provides documented cross-language annotation conventions for morphology and syntax. Use its guidelines when claiming a UD analysis. The informal role descriptions in this guide are teaching explanations, not a complete UD treebank or a replacement for those conventions.

Annotation reference: Universal Dependencies: introduction.

Continue into: Structural attachment · CYK parsing.

Contents · Previous chapter · Next chapter

Syntax describes grammatical relationships. Semantic roles describe participants’ relationships to an event. Reference concerns which entity an expression points to. These layers cooperate, but a correct answer at one layer does not automatically supply the others.

In “Arun gave Mira the atlas”, Arun is the giver, Mira the recipient and the atlas the transferred item. In a passive version, “Mira was given the atlas by Arun”, the grammatical arrangement changes while the giving roles can remain the same.

“Mira thanked Arun because she was relieved” supplies a likely interpretation of “she”, but pronoun resolution depends on the available context and linguistic cues. A useful assistant should record the entity it selected and preserve uncertainty where the text does not settle it.

For a reservation workflow, inspect three separate outputs: who asks, what item is requested and whether the request is current. This decomposition helps locate the fault when a fluent response reserves the wrong book for the wrong person.

Continue into: Pronoun ambiguity · Semantic roles.

Contents · Previous chapter · Next chapter

Use an intentionally small grammar: S becomes NP VP; NP becomes Mira or Arun; VP becomes V NP; V becomes reminded. Under these rules, “Mira reminded Arun” receives a subject noun phrase and a verb phrase containing the verb plus object noun phrase.

The grammar also permits “Arun reminded Mira”. It does not permit “Mira was reminded by Arun”, because the required auxiliary, passive pattern and preposition have not been included. Rejection here means outside this grammar, not necessarily outside English.

Extend the grammar only after naming the construction you want to add. Test that the extension accepts the intended example, and check whether it also accepts an unwanted pattern. A rule has consequences beyond the sentence that motivated it.

This is the central value of a toy formalism: you can inspect its coverage. The connected CYK parsing article takes the algorithmic route further. Real grammars and parsers require much broader structures and ambiguity handling than this deliberately restricted demonstration.

Foundation reading: Context-Free Grammars and Constituency Parsing.

Contents · Previous chapter · Next chapter

A bigram model approximates the next unit using the preceding unit. A simple maximum-likelihood estimate divides a pair count by the count of the conditioning unit in the relevant positions. This describes a model of the collection, not a universal law of language.

In a fictional collection, “reserve” has ten recorded following-word positions: “the” follows six times, “an” three times and “my” once. The unsmoothed estimates are 0.6, 0.3 and 0.1. Their sum is one because the stated outcomes exhaust those ten positions.

“Reserve our” has zero observed occurrences. The unsmoothed model assigns zero to that continuation; it has not proved that the phrase is impossible. Smoothing is one way of handling unseen events, but its method and assumptions must be specified.

Ask what the probability is conditioned on before interpreting a number. A high probability for “the” after “reserve” is not the probability that a message is a request, nor the probability that a reservation can be fulfilled.

Foundation reading: N-gram Language Models.

Continue into: Corpus frequency and collocation.

Contents · Previous chapter · Next chapter

Embeddings represent items as vectors learned or constructed for a purpose. Similarity in that space can help identify related expressions, but it is not a truth test. Distributional neighbours can share a topic while differing in polarity, direction or intended action.

Consider two original toy count vectors for three chosen contexts: atlas = (4,2,0) and dictionary = (2,1,0). Their cosine similarity is 1, since one vector is a positive multiple of the other. The construction shows matching proportions, not identical objects.

A catalogue still needs distinct records for the atlas and dictionary. A search system might use the similarity to propose related items while retaining the identifiers and exact title that control a reservation.

Inspect pairs such as “available” and “unavailable” rather than testing only easy synonyms. If a representation makes them neighbours, the downstream task must still preserve the difference. Similarity can support retrieval without authorising substitution.

Calculation check: the dot product is 4 × 2 + 2 × 1 = 10. The vector lengths are √20 and √5. Therefore cosine = 10/(√20 × √5) = 1. These hand-built counts are not measured embeddings from a trained system.

Foundation reading: Embeddings.

Continue into: Distributional semantics · Word embeddings.

Contents · Previous chapter · Next chapter

A trained model adjusts parameters to improve an objective on its training material. That objective determines what counts as progress during learning. It is not automatically the same as a teacher’s goal, a user’s intention or a scientific explanation of language acquisition.

For the library exercise, predicting the next word and identifying a reservation intent are different objectives. Success on one can help another task, but the link requires testing. Do not infer intent classification quality from the fluency of a generated reply.

Likewise, a model’s account of why it produced an answer is another output to examine, not an experimental record of its internal cause. Abnar and Zuidema’s attention-flow study also illustrates why raw attention weights need care as explanatory evidence.

Keep the explanation proportionate to the evidence. Say “this system labelled these messages under these conditions” when that is what was measured. Claims about what children know, how humans understand or which mechanism caused an output need their own investigation.

Research example: Abnar and Zuidema: Quantifying Attention Flow in Transformers.

Continue into: Transformer architecture vocabulary.

Contents · Previous chapter · Next chapter

Create training, development and test partitions before tuning. Training adjusts the model, development supports design choices and the final held-out test estimates performance under the defined evaluation. Repeatedly choosing a method on test answers weakens that final role.

Suppose “Please reserve the atlas” is in training and “Please reserve the dictionary” is in test. The words differ, but the template may make the second item a weak test of a new construction. Group related templates when testing transfer beyond memorised phrasing.

If messages come from conversations, splitting individual turns can expose the same conversation in several partitions. If the goal is generalisation to new users, a user-level split may be needed. The correct split follows the claim being tested.

For this classroom exercise, record a template family identifier beside each invented example. Hold out some families entirely and label that as a transfer test. Do not present a tiny constructed score as an estimate of real-world performance.

Contents · Previous chapter · Next chapter

Take a fictional held-out collection of 100 messages: 20 are new requests and 80 are not. The assistant predicts 25 requests. Of those, 15 are correct and 10 are false positives; five actual requests are missed. The remaining 70 messages are true negatives.

Precision is 15/25 = 0.60. Recall is 15/20 = 0.75. F1 is 2 × 0.60 × 0.75 / (0.60 + 0.75), approximately 0.667. Accuracy is (15 + 70)/100 = 0.85.

A system that always predicts NOT A NEW REQUEST obtains 0.80 accuracy while finding none of the 20 requests. The comparison shows why accuracy alone would be an incomplete report for this task. It does not establish that F1 alone is sufficient.

Keep the counts beside the percentages. A wrong reservation suggestion and a missed request can impose different costs. Compare systems on the same held-out cases, inspect their errors and state how any review option is included in the measurement.

Metric reference: Logistic Regression and Text Classification.

Contents · Previous chapter · Next chapter

Construct pairs that change one relevant feature. “Please reserve the atlas” versus “Please do not reserve the atlas” tests negation. “I want to reserve it” versus “I wanted to reserve it, but changed my mind” tests whether later context changes the interpretation.

Add meaning-preserving variations such as “Could you reserve the atlas for me?” A system should not require one memorised imperative form to recognise a request under the defined task. Then add a report of that request inside quotation marks.

The CheckList research provides a task-independent framework for behavioural testing. Here, the original library pairs serve a narrower teaching purpose: write the expected relationship first, then check whether the output follows it.

Do not assume every word swap preserves meaning. Replacing a name may alter a context-dependent pronoun interpretation. Review the test examples themselves, document the intended reading and allow an ambiguous item to remain ambiguous.

Research framework: Ribeiro and colleagues: Beyond Accuracy — Behavioral Testing of NLP Models with CheckList.

Contents · Previous chapter · Next chapter

An assistant might turn a request into another language or summarise a conversation. A plausible sentence is only one part of success. Check the requested action, book identity, speaker, negation, time and degree of certainty against the source.

Use a simple original contrast: “Mira may reserve the atlas” is not equivalent to “Mira reserved the atlas”. One permits or leaves open a future action; the other reports a past event. A polished translation that crosses that distinction has changed the message.

Automatic translation metrics provide useful comparison signals, not a complete account of fidelity in every case. The Machine Translation chapter linked below discusses evaluation approaches. For the classroom exercise, a short source-to-output checklist makes individual changes inspectable.

Where the source leaves “it” unresolved, do not reward a confident invented title merely because the output reads smoothly. Preserve the uncertainty or ask for the missing context. Evaluation should recognise that a faithful incomplete answer can be better than a fabricated complete one.

Foundation reading: Machine Translation.

Continue into: Why word-for-word translation fails.

Contents · Previous chapter · Next chapter

Do not assume that spaces, word order or morphology behave the same way in every language. Decide how each language’s examples will be segmented and annotated, and involve people competent to judge the intended constructions and meanings.

For a manageable project, compare the same communicative scenarios across two languages rather than replacing each English word mechanically. A request may be natural in one grammatical form and unnatural in a copied form. Preserve the scenario while allowing the language to differ.

Report each language’s outcomes separately before pooling them. A large collection in one language can dominate an aggregate score and conceal much weaker evidence in another. The same caution applies to varieties, genres and accessibility needs.

Cross-language annotation conventions can support comparison, as Universal Dependencies demonstrates for morphology and syntax. They still require language-specific analysis. A common label set does not make every surface pattern identical or eliminate the need to examine disagreements.

Continue into: Language root.

Contents · Previous chapter · Next chapter

For the fictional assistant, keep all predictions as suggestions. A person confirms the book and intended action. If the task later becomes a real service, its allowed actions, review process and failure response need explicit design before deployment.

A confidence score is not automatically a calibrated probability of correctness. Test the meaning of the score on appropriate held-out cases. Do not equate a high number with authority, and do not set a threshold only because it produces a reassuring-looking interface.

Write a small failure protocol: show the original message, display the proposed interpretation and ask a focused question when the item or intent is uncertain. “Which atlas do you mean?” exposes the missing information more clearly than a generic error message.

An abstention or review state is useful only if the handoff is usable. Count how many messages require review, inspect who receives them and check whether the proposed workflow creates more work than it saves. Evaluate the whole task, not only the model’s labelled subset.

Continue into: Localisation quality assurance.

Contents · Previous chapter · Next chapter

Begin with twelve fictional messages across requests, reports and cancellations. Write the task rules, annotate independently and resolve disagreements. Keep a table containing the original text, expected label, linguistic feature and template family.

Build a transparent baseline on paper: label a message NEW REQUEST if it contains “reserve”. Test the baseline on “Do not reserve the atlas” and “The atlas was reserved yesterday”. Explain the failures before proposing a more sophisticated method.

Change one part of the baseline, then use fresh examples to check the effect. A rule that handles “not” may still fail “I no longer want it reserved”. Progress means a better-defined coverage claim, not a promise to solve every English sentence.

Save the rules, versions, example provenance, split plan, counts and unresolved cases. This makes the investigation reproducible without pretending it is a production system. Read the corpus, parsing, vocabulary and translation branches below to deepen the part where your explanation is weakest.

Continue into: Corpus keyness.

Contents · Previous chapter · Next chapter

The strongest transfer is the habit of separating what the words say, how you represented them and what the evidence supports. That habit improves a language-model experiment, but it also helps a student check a paraphrase or notice an ambiguous pronoun.

When reading an automated answer, ask which source, which task and which assumptions are doing the work. When writing instructions, ask whether a reader can recover the intended actor, action, object and exception without guessing.

The gold standard of Computational Linguistics is therefore not simply a fluent system. It is a model whose purpose, representations, tests and limitations can be inspected, and whose claims remain proportionate when the examples change.

Use the connected directory as a map rather than a compulsory reading list. Start with the bottleneck: word units, grammatical structure, meaning, corpus evidence or evaluation. Return to the original language question after each specialist branch and ask what you can now test more precisely.

Continue into: Education parent · How X Works.

Contents · Previous chapter · Connected article directory