Corpus search precision Corpus search precision is the proportion of retrieved items that genuinely meet a declared relevance criterion. In a language corpus, the relevant item might be an occurrence of a word with a particular meaning, a grammatical construction or a passage that answers a reading question. Precision is not the number of search hits, and it does not by itself tell us how many relevant items were missed.
Did you know that finding the word bank eight times can still leave you with only five examples about finance? A search for the written form bank may retrieve ‘the bank approved the loan’ and ‘the river bank collapsed’. Both match the spelling. Only the first matches a question about financial institutions. A long results list is useful only when readers understand what actually qualifies.
This guide explains true and false positives, missed relevant results, precision, recall and F1. The worked example uses twelve fictional sentences and a clearly defined information need; it shows why precision is different from spelling-match accuracy, how a search can have high precision but poor coverage, and why relevance judgments need documentation.
Evidence note: All search records, sentences, counts and exercises below are original constructed illustrations. They are not claims about the measured performance of any commercial search engine or the language of real pupils. The standard definitions of precision and recall are linked to published information-retrieval methodology.
EDUKATE · VOCABULARY EVIDENCE · SEARCH WITHOUT GUESSWORK
Choose the route closest to your question
- Understand spelling matches versus relevant meaning
- Calculate precision, recall and F1
- Design a reliable search evaluation
- Use the result in reading, teaching and reporting
All chapters · Twelve-sentence table · Practical cases · Worked exercises · FAQ
Contents of the complete guide
- 1. Begin with the actual question, not the search box
- 2. Define relevance before counting results
- 3. Distinguish exact-string retrieval from sense retrieval
- 4. Count true positives as useful retrieved items
- 5. Recognise a false positive without blaming the reader
- 6. Find false negatives beyond the result page
- 7. Understand true negatives and why they can dominate
- 8. Read the complete twelve-item register
- 9. Calculate precision from the retrieved set
- 10. Calculate recall to expose missing evidence
- 11. Understand the precision–recall tradeoff
- 12. Compute F1 without hiding the components
- 13. Do not mistake accuracy for search usefulness
- 14. Choose the right evaluation unit
- 15. Check tokenisation and query representation
- 16. Distinguish precision from annotation correctness
- 17. Evaluate a system on examples it has not been tuned to
- 18. Recognise the limits of evaluating only returned hits
- 19. Use precision at a chosen ranking cut-off
- 20. Apply the method to vocabulary reading
- 21. Consider higher-stakes searches with extra care
- 22. Document the source and query before reporting a percentage
- 23. Audit the whole claim before making a recommendation
CHAPTER 1 OF 23 · LANGUAGE AND RETRIEVAL
1. Begin with the actual question, not the search box
A query is a tool for locating candidates. It is not identical to the user’s information need. The words typed into a search system can be narrower or broader than the concept someone hopes to investigate.
A grounded example
A learner wants examples referring to financial institutions and types bank into a corpus search. The tool can retrieve the riverbank sense because the spelling matches, and miss credit union because that phrase lacks the string bank.
What readers can mistake
The mistaken conclusion says that everything returned is evidence of financial institutions merely because the text matches the query. Matching the query syntax is not the same as satisfying the reader’s real task.
The more useful method
Write two separate lines: information need ‘references to financial institutions’ and operational query ‘whole-word bank, case-insensitive’. Then judge each returned item against the first line.
Quick test: Does every exact hit for bank meet the finance information need?
Explained answer: No. A whole-word match can refer to a riverbank or another meaning, and the query can miss other financial-institution expressions.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 2 OF 23 · LANGUAGE AND RETRIEVAL
2. Define relevance before counting results
Relevance is the rule that determines whether a candidate answers the stated question. It might concern word sense, subject, genre, grammatical role, population or source period. Without a criterion, precision has no stable denominator.
A grounded example
Our toy project labels a sentence relevant if it refers to a financial institution, including a bank or credit union. A mention of ‘the river bank’ is nonrelevant; a sentence about a credit union is relevant even without the word bank.
What readers can mistake
The shortcut labels the first few attractive results relevant and quietly changes the criterion for later cases. Two reviewers could then compute different precision values from the same source list.
The more useful method
Specify the target, unit and inclusion tests. Keep a rationale for borderline cases and, if needed, a second reviewer to check decisions before reporting a score.
Quick test: Would a credit union sentence be relevant to this project’s information need?
Explained answer: Yes. The declared need is financial institutions, not only exact spelling bank.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 3 OF 23 · LANGUAGE AND RETRIEVAL
3. Distinguish exact-string retrieval from sense retrieval
A strict string query can be excellent at finding a written form yet imperfect for the intended sense. Precision must be evaluated against the chosen task. An exact-form evaluation and a financial-sense evaluation could give different results from the same matches.
A grounded example
A search for bank returns both ‘The bank charged a fee’ and ‘They sat on the grassy bank’. Both are correct matches for the string but only one expresses a finance-related institution.
What readers can mistake
The error calls the riverbank result a search-engine malfunction even though the query merely asked for the exact token. The problem may be an underspecified query rather than bad indexing.
The more useful method
State which level is being evaluated. If the task is exact form, both strings are valid hits; if the task is financial meaning, the riverbank hit is an irrelevant candidate.
Quick test: Can one hit be correct for a spelling query but irrelevant for a meaning query?
Explained answer: Yes. Relevance changes when the information need changes.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 4 OF 23 · LANGUAGE AND RETRIEVAL
4. Count true positives as useful retrieved items
A true positive is an item that the query retrieved and that independent task-specific review finds relevant. For a word-sense search, the unit can be a target occurrence or a sentence, provided the policy is consistent.
A grounded example
In the constructed table, ‘The bank approved the loan’ is retrieved by the string bank and relevant to financial institutions. That is one true positive under our criterion.
What readers can mistake
The common shortcut treats a correct grammar tag or a word’s popularity as proof of relevance without actually checking whether the sentence addresses the declared concept.
The more useful method
Inspect the complete context, not only the highlighted word. Record the retrieved status and relevance label in different fields.
Quick test: If the query retrieves ‘A bank transferred the funds’, which category applies?
Explained answer: True positive: it has the exact token and describes a financial institution.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 5 OF 23 · LANGUAGE AND RETRIEVAL
5. Recognise a false positive without blaming the reader
A false positive is a retrieved item that does not meet the specified relevance definition. False positives often result from homographs, broad variants, ambiguous names or weak search patterns. They can be legitimate matches for another question.
A grounded example
‘The river bank collapsed’ contains bank and is retrieved. For the financial-institution task, it is a false positive even though the sentence is linguistically correct and informative about geography.
What readers can mistake
The mistake declares the sentence itself wrong or deletes it from the underlying corpus. The text is not a linguistic error; it simply does not serve this search objective.
The more useful method
Keep the sentence and its source, but mark it nonrelevant for this task. A different geography question could legitimately treat the same occurrence as relevant.
Quick test: Why is a riverbank hit a false positive here?
Explained answer: Because it was retrieved but is not about the financial-institution concept that the search is intended to find.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 6 OF 23 · LANGUAGE AND RETRIEVAL
6. Find false negatives beyond the result page
A false negative is a relevant item in the eligible collection that the retrieval method did not return. Finding these requires some way to inspect material outside the retrieved set, usually a reviewed reference collection or another validated discovery process.
A grounded example
‘The credit union approved the loan’ is relevant to the financial-institution question but lacks the exact written word bank, so the strict bank query does not retrieve it. This is a false negative for the broad concept task.
What readers can mistake
A careless report says recall must be 100 percent because every retrieved hit was reviewed. Precision can be measured from retrieved items, but missed relevant items lie outside that set.
The more useful method
Audit an appropriately sampled or complete reference set to identify missed relevant material. State clearly when full recall is unknown because exhaustive relevance labels were not obtained.
Quick test: Can checking only returned search hits reveal every false negative?
Explained answer: No. The missed relevant items are, by definition, outside the returned set.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 7 OF 23 · LANGUAGE AND RETRIEVAL
7. Understand true negatives and why they can dominate
A true negative is an eligible item that is not relevant and is not retrieved. A large corpus can contain vast numbers of such items. They matter for some classification analyses but do not enter the ordinary precision denominator.
A grounded example
‘The school library closes at five’ contains no bank and does not concern a financial institution. In our toy corpus it is a true negative.
What readers can mistake
The shortcut calculates ordinary classification accuracy and treats a huge true-negative count as proof that the search works for finding the few interesting relevant items.
The more useful method
Use precision and recall for retrieval effectiveness. Record true negatives when they clarify a full contingency table, but do not let them hide irrelevant hits or missed evidence.
Quick test: Does finding many nonrelevant documents and correctly skipping them guarantee useful search results?
Explained answer: No. A system could ignore every document and obtain many true negatives while retrieving nothing valuable.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 8 OF 23 · LANGUAGE AND RETRIEVAL
8. Read the complete twelve-item register
The worked table below contains twelve constructed sentences: eight contain the token bank and are returned by a strict whole-word query, and four do not. Seven sentences meet the independently declared financial-institution relevance criterion.
A grounded example
Among eight retrieved items, five are relevant finance examples and three are riverbank examples. Two more finance references occur in the unretrieved group, alongside two other nonrelevant sentences.
What readers can mistake
The mistaken summary says there are eight finance examples because bank was searched eight times. That treats homography as semantic identity and overlooks the credit-union and financial-institution examples.
The more useful method
Write a two-by-two contingency table after labelling every record. Only then calculate precision, recall and any combined score.
Quick test: How many items are retrieved by the exact bank query?
Explained answer: Eight, of which five meet the finance relevance rule and three do not.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 9 OF 23 · LANGUAGE AND RETRIEVAL
9. Calculate precision from the retrieved set
Precision is the number of retrieved relevant items divided by all retrieved items. In the standard notation, P = TP/(TP+FP). The denominator asks how much of the search output is useful under the declared criterion.
A grounded example
Our toy set has five true positives and three false positives, so precision is 5/(5+3) = 5/8 = 0.625, or 62.5 percent. This number describes the returned set under the selected relevance rule.
What readers can mistake
The shortcut divides by the seven total relevant items and calls the result precision. That actually computes recall, not the fraction of retrieved items that help the task.
The more useful method
When explaining a precision percentage, write ‘of the items returned’ and show raw numerator and denominator. Readers should be able to reproduce it without trusting the label.
Quick test: What is precision when five of eight retrieved results are relevant?
Explained answer: 5/8, or 62.5 percent, for this constructed search.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 10 OF 23 · LANGUAGE AND RETRIEVAL
10. Calculate recall to expose missing evidence
Recall measures the fraction of all relevant eligible items that the search retrieved. With a reviewed reference set, R = TP/(TP+FN). Recall asks a different question from precision.
A grounded example
The table has five relevant results retrieved and two relevant references missed. Recall is 5/(5+2) = 5/7, about 71.43 percent.
What readers can mistake
A weak interpretation says 71.43 percent of the returned hits are correct. That confuses recall’s denominator, all relevant eligible items, with the retrieved set.
The more useful method
Label recall as ‘of all relevant items in the inspected collection’. If the full collection is not labelled, report recall as unknown rather than guessing.
Quick test: What does the numerator five mean in both precision and recall?
Explained answer: The five true positives are the same; their denominators answer different questions.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 11 OF 23 · LANGUAGE AND RETRIEVAL
11. Understand the precision–recall tradeoff
A narrower query may reduce false positives but miss more relevant uses. A broader query may retrieve additional relevant records while also admitting more irrelevant ones. The tradeoff depends on the actual query and corpus, not on a universal claim that one must always rise when the other falls.
A grounded example
Adding credit union OR financial institution to our fictional query could retrieve the two missed relevant sentences, improving recall. In a real corpus, similar expansions might also retrieve unrelated mentions or false matches.
What readers can mistake
The mistake assumes that every expansion always harms precision or that every restriction improves it. The direction is an empirical result that must be counted under the revised rule.
The more useful method
Re-run a documented query and review all newly retrieved items. Report the new TP, FP and FN instead of predicting a result from intuition alone.
Quick test: Does broadening a query guarantee lower precision?
Explained answer: No. It can increase, decrease or maintain precision depending on the relevance of newly retrieved items.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 12 OF 23 · LANGUAGE AND RETRIEVAL
12. Compute F1 without hiding the components
The balanced F1 measure is the harmonic mean of precision and recall. It gives a single summary but should not replace the two components, particularly when the cost of missed evidence differs from the cost of false positives.
A grounded example
For TP=5, FP=3 and FN=2, F1 = 2TP/(2TP+FP+FN) = 10/(10+3+2) = 10/15, or about 66.67 percent. It agrees with 2PR/(P+R) using P=5/8 and R=5/7.
What readers can mistake
The shortcut takes the ordinary arithmetic average of 62.5 and 71.43 and calls it F1. The harmonic mean is defined differently and responds more strongly when one component is much lower.
The more useful method
Publish P and R alongside F1 and describe the task context. A combined score does not tell readers whether misses or false matches are more damaging for a particular use.
Quick test: What is the toy F1 score?
Explained answer: Approximately 66.67 percent, using five true positives, three false positives and two false negatives.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 13 OF 23 · LANGUAGE AND RETRIEVAL
13. Do not mistake accuracy for search usefulness
Classification accuracy measures correct decisions over all classified items, including true negatives. In large retrieval collections irrelevant documents can vastly outnumber useful ones, so accuracy can be high even when a search returns no helpful result.
A grounded example
Our tiny table has five true positives and two true negatives among twelve records, giving (5+2)/12 = 7/12 ≈ 58.33 percent classification accuracy. In a huge corpus with many irrelevant items, an all-negative strategy may appear accurate while failing the user’s task.
What readers can mistake
The misleading report chooses overall accuracy because it yields a flattering percentage, then omits the fact that the retrieval system missed many or all important passages.
The more useful method
Use metrics that match the task and class distribution. For searching language examples, precision and recall more directly reveal output usefulness and coverage.
Quick test: Why can a search tool score well on accuracy while finding nothing?
Explained answer: It may correctly ignore many irrelevant items yet retrieve no relevant evidence.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 14 OF 23 · LANGUAGE AND RETRIEVAL
14. Choose the right evaluation unit
A retrieval evaluation might score whole documents, passages, sentences or individual word occurrences. A search hit containing ten occurrences could be one document result but ten token events. The relevance criteria and denominators must match the chosen unit.
A grounded example
Our miniature table explicitly treats each of twelve sentences as one item. It would require a different table to evaluate a search engine returning whole articles with many sentences.
What readers can mistake
The mistake compares document-based precision in one tool with target-token precision in another as though they were the same measure.
The more useful method
Define the item unit at the start, then use stable IDs throughout the search output and reference set.
Quick test: Can twelve sentence rows establish precision per whole document?
Explained answer: Not automatically. That would require a defined mapping from sentence matches to document results and appropriate document-level labels.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 15 OF 23 · LANGUAGE AND RETRIEVAL
15. Check tokenisation and query representation
A word query can be case-sensitive or case-insensitive, exact-form or lemma-based, contiguous phrase or grammar-aware pattern. These settings change which candidates are retrieved, not necessarily which concepts the reader seeks.
A grounded example
The exact spelling bank does not retrieve banking or banker, while a broad morphological search might. But grouping derivational relatives is not automatically correct for a sense-specific search.
What readers can mistake
The error combines scores from different query settings without noting that the retrieved sets and relevance decisions may have changed.
The more useful method
Keep the query string, text normalisation, token boundaries and expansion rules with the score. Validate ambiguous extra matches using full context.
Quick test: Would a lemma search for BANK necessarily retrieve every sentence about financial institutions?
Explained answer: No. It may still miss credit union and other paraphrases while retrieving non-financial senses.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 16 OF 23 · LANGUAGE AND RETRIEVAL
16. Distinguish precision from annotation correctness
Precision relies on relevance labels. If the labels themselves are poorly defined or inconsistently applied, the result may be arithmetically correct for a flawed reference. Inter-annotator agreement can diagnose consistency but is not proof of truth.
A grounded example
Two reviewers could disagree about a sentence mentioning a central bank or an organisation named Bank that is not actually a financial institution. The task needs explicit criteria before adjudication.
What readers can mistake
The shortcut calculates 100 percent precision because a single reviewer declared every result relevant without independently checking the task definition.
The more useful method
Write a reference guideline, test a sample of decisions, preserve disagreements and record adjudication. Report limitations of the labelled evaluation set.
Quick test: Can a precise decimal percentage make unreliable relevance labels reliable?
Explained answer: No. Mathematical precision does not validate the underlying judgments.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 17 OF 23 · LANGUAGE AND RETRIEVAL
17. Evaluate a system on examples it has not been tuned to
If a search rule is repeatedly changed to match a tiny labelled dataset perfectly, performance on that same set can be optimistic. A fresh collection is needed to check whether the improved rule transfers beyond development examples.
A grounded example
A class might add credit union and financial institution after seeing the two missing records. That improves the constructed set, but a new corpus could contain unrelated usages and different expressions.
What readers can mistake
The mistaken report calls the revised score an unbiased prediction of future searches when the same examples guided every adjustment.
The more useful method
Separate development and independent evaluation where feasible. Show both the improvement on the toy set and the uncertainty about new material.
Quick test: Does a revised query’s 100 percent toy recall prove perfect real-world coverage?
Explained answer: No. It covers the labelled constructed set under its rule, not every possible document or expression.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 18 OF 23 · LANGUAGE AND RETRIEVAL
18. Recognise the limits of evaluating only returned hits
A reviewer can often estimate precision from a suitable sample of search results, but recall requires knowledge of relevant items outside the returned set. In massive corpora, exhaustive annotation may be impractical.
A grounded example
If the returned set contains twenty sentences and fifteen appear relevant, a local precision estimate is possible. But without surveying other eligible text, the number of relevant misses is not known.
What readers can mistake
The error supplies a recall percentage because a tool displays only the twenty retrieved examples. Missing relevant sources have not been observed.
The more useful method
Use a designed reference sample, pooling methods or appropriately qualified audits for recall when full labelling is impractical. Do not invent a false-negative total.
Quick test: Can precision be evaluated without knowing every relevant item in the corpus?
Explained answer: Often a retrieved-result precision sample can be evaluated, but complete recall generally requires a broader reference or justified estimation.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 19 OF 23 · LANGUAGE AND RETRIEVAL
19. Use precision at a chosen ranking cut-off
Ranked search systems put some candidates first. Precision at k measures the fraction of the top k returned items that are relevant. It is useful when a reader will inspect only the first few results, but it is not the same as precision across the entire retrieved set.
A grounded example
A school search interface may show ten highlighted lines but a teacher can only inspect the first five. If four of those five meet the task, P@5 is 80 percent, independently of how many relevant lines occur farther down.
What readers can mistake
The shortcut reports top-five precision as whole-system precision without naming k or checking whether the sample was sorted by relevance or merely by source order.
The more useful method
Specify the ranking rule and cut-off. Compare P@k only between similarly ranked lists and avoid claiming that top-k precision measures total coverage.
Quick test: What does P@5=80 percent say?
Explained answer: Four of the first five evaluated retrieved items are relevant under the stated task, not that 80 percent of all relevant evidence was found.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 20 OF 23 · LANGUAGE AND RETRIEVAL
20. Apply the method to vocabulary reading
For a learner, a corpus query is often a way to inspect examples of a new word. High precision reduces the number of irrelevant sentences they must sort through; enough recall helps them see other useful meanings or constructions.
A grounded example
A student studying the financial meaning of bank benefits from a set of clear finance examples. A geography student might deliberately search the riverside sense instead. The same word form is useful in both but should not be mixed indiscriminately.
What readers can mistake
An unhelpful vocabulary worksheet gives twenty unfiltered hits and expects students to infer one sense without explaining that some hits have a different meaning.
The more useful method
Tell students the target sense, select and verify contexts, then introduce contrasts deliberately. End with a new sentence where the learner justifies the correct interpretation.
Quick test: Is the ideal vocabulary example list always the one with the most hits?
Explained answer: No. Relevant, varied, clear examples are more useful than an inflated count.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 21 OF 23 · LANGUAGE AND RETRIEVAL
21. Consider higher-stakes searches with extra care
Searching for medical, legal or safety guidance can be harmed by both irrelevant hits and missed important information. Precision and recall remain useful concepts, but a simple keyword check cannot replace qualified professional interpretation or authoritative sources.
A grounded example
A hospital instruction concerning one drug name could share a word with an unrelated condition. A strict search may miss synonyms, while a broad search may flood reviewers with false matches.
What readers can mistake
The mistake applies the 62.5 percent toy score to medical document retrieval or offers unsafe conclusions from a small classroom example.
The more useful method
Use domain experts, validated references, source controls and task-specific risk analysis in consequential settings. This article’s constructed data are for learning only.
Quick test: Should a classroom precision exercise be used as a clinical retrieval benchmark?
Explained answer: No. Real health decisions require validated domain-specific methods and qualified oversight.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 22 OF 23 · LANGUAGE AND RETRIEVAL
22. Document the source and query before reporting a percentage
A checkable report names the corpus, target information need, exact query, item unit, reference labels, counts, precision and recall. A score without those details cannot reliably be reproduced.
A grounded example
A clear toy report could say ‘Against twelve constructed sentence references, the whole-word case-insensitive bank query returned eight, five of which concerned financial institutions; two further relevant sentences were missed.’
What readers can mistake
The misleading version says simply ‘the bank search is 62.5 percent accurate’ without naming precision, the intended sense or the hand-built dataset.
The more useful method
Put the contingency counts beside the rate and include a short note explaining how false negatives were found. Save the evaluation version.
Quick test: What must accompany the figure 62.5 percent here?
Explained answer: That it is precision: five relevant among eight retrieved sentences in this constructed financial-institution search.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
CHAPTER 23 OF 23 · LANGUAGE AND RETRIEVAL
23. Audit the whole claim before making a recommendation
Before publishing a search-quality claim ask: what information need, query, eligible corpus, evaluation unit, reference definition, true positives, false positives, false negatives, ranking cut-off and source independence? A reader should be able to reconstruct the result.
A grounded example
Our toy conclusion is deliberately narrow: precision is 5/8, recall 5/7, and F1 10/15 for a fixed twelve-item set. These are calculations, not a measured score of Google, an AI system, a school or real readers.
What readers can mistake
The error turns a teaching illustration into a boast about a live search engine or an education programme. No real-world evaluation was conducted.
The more useful method
Keep the calculation, reference and limits visible. Use the method to improve whether search results actually answer a reader’s question.
Quick test: What separates a useful precision calculation from a misleading search headline?
Explained answer: An explicit relevance definition, inspectable counts and a conclusion no broader than the evaluated items.
In another subject, begin by separating the question the reader wants answered from the words a search engine matches. The distinction determines what counts as relevant. Keep both the returned set and the reference set clearly defined, and report unreviewed material as unreviewed rather than assigning invented labels. A useful number is one that another reader can reconstruct and interpret correctly.
Worked dataset: twelve sentences and four result categories
We define the information need as any sentence referring to a financial institution, including a bank or credit union. The retrieval query is the case-insensitive whole-word written form bank. Each of the following twelve original invented sentences is a separate evaluation item; all twelve were reviewed under the declared information need. An exact hit is not automatically relevant to the financial concept.
| ID | Constructed sentence | Relevant to finance? | Exact bank retrieved? | Classification |
|---|---|---|---|---|
| R01 | The bank approved the loan. | Yes | Yes | TP |
| R02 | She deposited money at the bank. | Yes | Yes | TP |
| R03 | The river bank collapsed. | No | Yes | FP |
| R04 | The bank charged a fee. | Yes | Yes | TP |
| R05 | They sat on the grassy bank. | No | Yes | FP |
| R06 | The bank branch opened at nine. | Yes | Yes | TP |
| R07 | The hikers walked along the bank. | No | Yes | FP |
| R08 | A bank transferred the funds. | Yes | Yes | TP |
| R09 | The credit union approved the loan. | Yes | No | FN |
| R10 | The financial institution raised its interest rates. | Yes | No | FN |
| R11 | A bridge crossed the stream. | No | No | TN |
| R12 | The school library closes at five. | No | No | TN |
Read R01–R08 as the eight exact-string retrieval results. Five refer to financial institutions, making them true positives. Three refer to land beside water, making them false positives for this specific task. The spelling bank is present in all eight, so the string search itself is behaving as declared.
R09 and R10 are relevant financial-institution references not returned by the exact bank query: two false negatives. R11 and R12 are neither retrieved nor relevant: two true negatives. The full contingency table therefore has TP=5, FP=3, FN=2, TN=2, total twelve items.
Precision = TP/(TP+FP) = 5/(5+3) = 5/8 = 62.5 percent. The denominator is the eight retrieved sentences. Recall = TP/(TP+FN) = 5/(5+2) = 5/7 ≈ 71.43 percent. The recall denominator includes all seven relevant sentences in the reference set.
F1 = 2TP/(2TP+FP+FN) = 10/(10+3+2) = 10/15 ≈ 66.67 percent. Classification accuracy across the twelve rows would be (TP+TN)/12 = (5+2)/12 ≈ 58.33 percent, but this is not the same as search precision.
If the constructed query were expanded to bank OR credit union OR financial institution, and no additional sentences matched, it would return ten items: seven relevant and three nonrelevant, giving precision 7/10=70% and recall 7/7=100%. That improvement is a fact only of this deliberately finite toy register, not a prediction about a real corpus.
All rows were assigned relevance labels by the teaching author. The table has not undergone independent annotation or been tested on an operational search engine. It illustrates arithmetic and the logical distinction between lexical string matches and useful information retrieval.
Twelve decisions: apply search quality to real reading tasks
All cases are constructed to teach the difference between query matching, relevance, missing evidence and appropriate instruction. They do not measure a specific live system.
1. Riverbank homograph
Situation. A bank query retrieves ‘river bank’.
Risk. Spelling matches but sense is irrelevant to finance.
Response. Mark FP relative to the declared need.
Next question. Would geography treat it differently?
2. Credit union synonym
Situation. A credit-union sentence is not retrieved.
Risk. A relevant concept is expressed without the query word.
Response. Mark FN and consider justified query expansion.
Next question. What other phrases can describe the institution?
3. Exact-word task
Situation. A user wants every occurrence of the letters bank.
Risk. All eight hits are correctly retrieved tokens.
Response. Redefine relevance for the exact-form task before calculating precision.
Next question. Do riverbank instances now qualify?
4. One source with repetitions
Situation. Ten matches come from one copied paragraph.
Risk. Hit count may exaggerate independent examples.
Response. Track unique source spans and copies.
Next question. Is the unit token, sentence or document?
5. Case-sensitive search
Situation. The query misses sentence-initial Bank.
Risk. Capitalisation changes retrieval.
Response. Check case settings and newly matched senses.
Next question. Could proper names introduce false positives?
6. Broad wildcard
Situation. A search matches banker and banking.
Risk. Expanded forms may serve different lexical tasks.
Response. Define lemma and derivative boundaries explicitly.
Next question. Which added matches satisfy the target?
7. High precision low recall
Situation. A strict pattern returns two perfect examples but misses twenty.
Risk. Output usefulness is high but coverage poor.
Response. Report both P and R and the task’s priorities.
Next question. Are the missed forms important?
8. High recall noisy set
Situation. A broad search returns all relevant lines and many irrelevant ones.
Risk. Review workload is large despite coverage.
Response. Use filters and ranking without hiding missed evidence.
Next question. Does the task prioritise recall?
9. Inconsistent judgments
Situation. Two reviewers disagree on an organisation type.
Risk. The relevance rule is underdefined.
Response. Clarify and document reference labels.
Next question. Which independent facts settle the status?
10. Short context window
Situation. The target word appears without enough surrounding context.
Risk. Sense cannot be resolved confidently.
Response. Inspect fuller permitted context or mark uncertain.
Next question. Would the next sentence clarify the meaning?
11. Ranked corpus results
Situation. An interface places useful hits mostly after page two.
Risk. Whole-set precision hides user experience at the top.
Response. Measure precision at a declared rank cut-off.
Next question. How many top-five results are useful?
12. Student vocabulary
Situation. A pupil reads twenty unfiltered bank examples.
Risk. Different meanings may confuse the lesson target.
Response. Choose verified examples and teach contrasts deliberately.
Next question. Can the learner select the right sense independently?
Eight practice questions and explained solutions
Work out the answer independently, then compare your reasoning. For every percentage, show which items were counted and why they were relevant to the stated need.
Exercise 1: Count true positives
Question. How many returned items meet the finance information need in the table?
Model answer. Five true positives: R01, R02, R04, R06 and R08.
Common error. All eight exact-string bank hits are not finance references.
Exercise 2: Count false positives
Question. Which returned items do not meet the need?
Model answer. R03, R05 and R07; they concern land beside water.
Common error. They are not wrong sentences, only irrelevant to this task.
Exercise 3: Count missed relevant items
Question. Which relevant sentences are absent from exact bank results?
Model answer. R09, the credit-union sentence, and R10, the financial-institution sentence: two false negatives.
Common error. Checking retrieved hits alone cannot find the missed items.
Exercise 4: Calculate precision
Question. Find precision for five relevant among eight returned.
Model answer. P=5/(5+3)=5/8=62.5%.
Common error. Dividing by seven relevant items computes recall instead.
Exercise 5: Calculate recall
Question. Find recall when five of seven relevant items were returned.
Model answer. R=5/(5+2)=5/7≈71.43%.
Common error. The denominator is all relevant items, not all retrieved items.
Exercise 6: Calculate F1
Question. Use TP=5, FP=3, FN=2 to calculate balanced F1.
Model answer. F1=2×5/(10+3+2)=10/15≈66.67%.
Common error. An ordinary arithmetic mean of the two percentages is not the harmonic mean.
Exercise 7: Change the task
Question. If the goal is exact bank spelling rather than finance meaning, how are riverbank hits judged?
Model answer. They count as valid exact-form matches under a whole-word spelling task. The relevance definition has changed.
Common error. A previous FP label cannot be reused without checking the task.
Exercise 8: Read a broader query
Question. The toy query adds credit union and financial institution, with no new irrelevant matches. What happens?
Model answer. Seven of ten retrieved are relevant, so P=70%; all seven relevant are retrieved, so R=100%, for this toy dataset only.
Common error. Do not generalise the improved numbers to real web or corpus search.
Frequently asked questions about corpus search precision
What is corpus search precision?
The fraction of returned corpus-search items that genuinely meet a declared relevance criterion.
Is precision the number of search hits?
No. It is a proportion based on whether returned items are relevant.
What is the formula?
Precision = true positives divided by true positives plus false positives.
What is a false positive?
A returned item that fails the declared relevance rule.
Can a correct spelling match still be a false positive?
Yes, when the actual information need concerns a meaning or context that the matching sentence does not satisfy.
What is recall?
The share of all relevant eligible items that the search returned.
What is a false negative?
Relevant material that was eligible but missed by the retrieval method.
Why is recall hard to estimate?
Missed relevant items lie outside the retrieved set, so a broader labelled reference or justified sampling method is needed.
What is F1?
The balanced harmonic mean of precision and recall.
Can a query have 100% precision but poor recall?
Yes. It can return only a few unambiguously useful examples while missing many others.
Does widening a query always lower precision?
No. The effect depends on what newly retrieved items are relevant.
Is precision the same as accuracy?
No. Classification accuracy includes true negatives; ordinary retrieval precision does not.
Why does the target meaning matter?
The same string can have several senses and a question about one sense may exclude other exact matches.
Can a reviewer measure precision on a small sample?
A carefully selected and labelled sample can estimate retrieved-result precision, with uncertainty, but does not automatically establish full recall.
What is precision at k?
The share of relevant items among the first k ranked results under a specified ranking.
How should a tutor apply this concept?
Choose verified sentence examples for a targeted meaning, then challenge the learner with an appropriate contrasting sense.
Does the toy score evaluate a real AI search engine?
No. The twelve sentences and counts are constructed solely for teaching.
Glossary for better vocabulary research
- Information need: The user’s actual question or intended concept.
- Query: The operational search expression used by a system.
- Retrieval item: The document, sentence or occurrence scored as returned or not returned.
- Relevance: Whether an item meets the defined information need.
- True positive: Relevant item retrieved.
- False positive: Irrelevant item retrieved.
- False negative: Relevant item not retrieved.
- True negative: Irrelevant item not retrieved.
- Precision: Fraction of retrieved items that are relevant.
- Recall: Fraction of all relevant items that were retrieved.
- F1: Harmonic mean of precision and recall.
- Homograph: A shared spelling that can support distinct meanings.
- Sense annotation: Classification of which meaning an occurrence expresses.
- Precision at k: Precision among the first k ranked retrieval results.
- Gold-standard reference: A reviewed task-specific set of accepted relevance labels.
- Query expansion: Adding justified alternative terms or patterns to retrieve more candidates.
A teaching and checking sequence
Begin with the twelve-sentence table and ask students what the search engine did correctly. It retrieved every example containing the literal bank token under the constructed rule. Next ask what the reader actually needed: examples of financial institutions. The second question shows why retrieving a text pattern and answering a human question are separate steps.
Then have learners classify each row before showing the TP, FP, FN and TN labels. A peer should challenge the first set of decisions by checking the full context and declared meaning target. Calculate precision and recall from the checked table, with the denominators written in words. Emphasise that a good search can return irrelevant matches for a broader information need without being technically broken.
Next change the information need to geography and let the student relabel relevant items. Keep the query unchanged. The calculation may change because relevance is task-relative. This is a more transferable insight than memorising one percentage.
Finally, use the method for another educational subject, such as finding examples of charge in electricity rather than payment, or estimating uses of solution in chemistry rather than a mathematical answer. A learner should finish with a clear question, a justified search pattern, checked examples and a statement of what may still be missed.
Published methodology and related research
- Manning, Raghavan and Schütze (2008), Introduction to Information Retrieval — Evaluation of Unranked Retrieval Sets
- Manning, Raghavan and Schütze — Evaluation of Ranked Retrieval Results
- Stanford — Information Retrieval book and chapters
- eduKateSG — Corpus Query and Search Rules
- eduKateSG — Annotation Guidelines
The retrieval definitions come from published methodology. The twelve-item register and its results were invented to show the calculations; no real-world search engine performance is claimed.