A relevance judgement is a decision about how well a document, passage, result or other item satisfies a specified information need. A page can contain the words Secondary 1 Mathematics and still fail to answer a student’s question about solving a particular equation. Another page may use different terminology but explain exactly the missing step. Relevance is not simply the presence of matching keywords; it concerns the relationship between a reader’s question and what the retrieved information can actually provide.
This makes vocabulary a practical evaluation technology. Students need words for distinguishing direct answers, partial answers, tangentially related material and off-topic results. Parents and tutors can use those distinctions to select exercises that serve the current learning goal, while editors can discover why a search engine returns convincing-looking but unhelpful pages. A relevance judgement does not automatically certify a source’s factual reliability, and a factually accurate article is not necessarily the right answer to a particular query.
This guide explains binary and graded relevance assessments, user intent, information needs, source scope, evidence, evaluation pools and disagreement between assessors. It includes original educational search cases, a worked precision–recall example, a small practical relevance rubric and exercises for testing a vocabulary library. The primary research references are NIST’s TREC evaluation guidance and Stanford’s Introduction to Information Retrieval. All fictional scores and sample collections are teaching illustrations, not measurements of eduKateSG or other live search engines.
Editorial scope: All learners, libraries, example queries, relevance grades and numeric performance scores are fictional instructional illustrations. No claim is made about current live search-engine performance or official NIST evaluations.
Contents: does this result actually answer the question?
Open all 56 chapters, examples and evaluation exercises
- Start with a real question, not just a keyword
- The research meaning of relevance
- Information need and query are not the same
- A document–query pair needs a defined context
- Topical relatedness is weaker than answering the question
- Binary relevance judgements
- Graded relevance judgements
- A clear local four-level rubric
- A title can be misleading
- An exact keyword match can still be off-topic
- A synonym can identify a useful page
- Relevant is not the same as true
- Credibility is a separate check
- Currentness depends on the topic
- A resource can be relevant but inaccessible
- Reading level can matter to educational relevance
- Resource format can be part of the need
- A single relevant paragraph may be enough
- A relevant result can be only partially useful
- A prerequisite guide is not always the final answer
- Reader intent affects relevance
- Query ambiguity makes relevance uncertain
- Judges can disagree
- Adjudication needs evidence and rules
- Judgements need stable document identifiers
- Qrels connect topics with document labels
- A test collection has three main parts
- Pooling reduces assessment work
- Unjudged is not automatically nonrelevant
- Binary and graded measures answer different questions
- Precision can be calculated for a toy library
- Recall needs a total relevant set
- Precision at a cut-off helps examine top results
- Rank position matters
- A search result snippet can distort judgement
- The relevance rubric should precede scoring
- Search relevance and task success are not identical
- The learner may need a different type of resource
- Worked English case: inference versus prediction
- Worked Mathematics case: solving an equation
- Worked Science case: evaporation and boiling
- Worked current-policy case
- Worked project-resource case
- A four-field judgement record
- Compare two assessors fairly
- A negative test detects false relevance
- Do not overfit the judgement scheme to one site
- Relevance feedback can improve the next search
- A controlled vocabulary helps evaluators communicate
- A crosswalk can help find relevant material
- A provenance record helps check an evaluation
- What a small study library could measure
- A staged review prevents false certainty
- Ten questions and short answers
- Frequently asked questions
- Research sources and next task
1. Start with a real question, not just a keyword
A parent searches ‘Primary 5 equivalent fractions explanation’. The immediate information need is a suitable explanation of why two fractions can represent the same quantity, not every article that contains the words Primary 5.
A page about general school schedules might mention fractions in passing. The words match, but the page does not substantially address the mathematical question.
A relevance judgement should record the intended question and whether the document actually helps answer it.
2. The research meaning of relevance
NIST’s TREC guide explains that relevance judgements, often called qrels, record which documents assessors judge relevant to particular search topics.
The Stanford information retrieval textbook treats relevance as a central part of evaluating retrieval systems. It distinguishes the user’s underlying information need from the literal query string used to express it.
A result is not context-free relevant merely because it contains a subject word. Relevance depends on the intended task and the collection being assessed.
3. Information need and query are not the same
A learner types ‘why multiply zero equation wrong’. The intended question may concern why multiplying both sides of an equation by zero can produce a non-equivalent equation.
An evaluator who judges results only against the rough wording may dismiss a precise explanation of equivalence or accept an unrelated article about multiplication by zero.
Write the information need in ordinary language before scoring results. This protects the evaluation from treating the user’s imperfect phrasing as the only valid vocabulary.
4. A document–query pair needs a defined context
A resource can be relevant to a broad research query and irrelevant to a beginner’s exercise. The same article might give a complete technical proof but no accessible explanation of the first concept the student needs.
Assess relevance to the target stage, task, format and desired depth, where those conditions form part of the information need.
Do not assume a resource’s inherent quality fixes its relevance for every learner. A strong university article may be unsuitable for an elementary question without being incorrect.
5. Topical relatedness is weaker than answering the question
A page about plants, water and sunlight is topically related to evaporation. It may still not explain the specific distinction between surface evaporation and boiling.
A relevance judgement should ask whether the needed explanation is present and usable rather than reward every shared subject word.
Category membership helps candidates appear in search; task-specific information determines whether they satisfy the need.
6. Binary relevance judgements
A binary scheme classifies an item as relevant or not relevant to the defined topic. Some traditional TREC collections use this approach, and the meaning of relevant depends on the task guidelines.
A simple classroom rubric can set clear criteria: Relevant when the document directly answers the target question or supplies a necessary part of the answer; otherwise Not Relevant. The criterion must be stated before the test.
Binary classification is efficient but may hide differences between a brief useful mention and a comprehensive explanation.
7. Graded relevance judgements
Graded schemes assign several levels of relevance, such as not relevant, partially relevant, relevant and highly relevant. The particular labels and scoring definitions vary by evaluation programme.
The TREC 2024 overview notes that modern evaluations frequently use graded judgements even though early collections often used binary labels.
A local education-library test can adopt a four-level scale, but should clearly mark it as its own rubric rather than misrepresent it as NIST’s universal scale.
8. A clear local four-level rubric
Level 0: Off-topic or does not help answer the intended need. Level 1: Related but offers only a minor useful detail. Level 2: Substantially relevant but incomplete for the intended task. Level 3: Directly and sufficiently answers the defined need, assuming its information is correct.
These are proposed local educational definitions. Another project may legitimately set different grade boundaries or incorporate resource accessibility.
The important test is consistency: can two assessors use the definitions to explain why one page gets a higher grade than another?
9. A title can be misleading
A search result advertises ‘Everything about Fractions’ but consists mainly of a short list of fraction names with no explanation of equivalent values.
The title may attract the parent, but the body fails the specific information need. A relevance judge should examine the actual content.
Do not allow SEO wording, confidence of a headline or page length to substitute for evidence of task satisfaction.
10. An exact keyword match can still be off-topic
A learner searches ‘line graph slope’. A result about a geographical slope containing one sentence about line graphs may share keywords without teaching the intended Mathematics concept.
An evaluation based solely on term overlap can give that page an inflated score.
Inspect the central claim, worked examples and information the reader could use. Lexical overlap is a discovery cue, not a relevance proof.
11. A synonym can identify a useful page
A student searches ‘how to compare fractions equal size’. Another document uses ‘equivalent fractions’ and explains the idea correctly.
The document can be relevant even if its title does not repeat the student’s words. Vocabulary knowledge or a controlled synonym mapping helps recognise the connection.
See Query Expansion for methods that may retrieve these differently worded results.
12. Relevant is not the same as true
A page might directly address the right question but contain an incorrect calculation. It is topically relevant while factually unreliable.
Another source may be authoritative about a neighbouring topic but not answer the user’s actual question. Accuracy and relevance are separate dimensions.
An educational result review should examine both, especially before recommending a method or reporting official information.
13. Credibility is a separate check
A source’s author, evidence, date and authority may matter to whether it should be trusted. Yet high authority alone does not guarantee relevance to a specific narrow question.
For example, an official timetable may be authoritative about schedule dates but irrelevant to a pupil’s request for a worked algebra problem.
Use one question for relevance—does this meet the need?—and another for credibility—is the claim supported and current?
14. Currentness depends on the topic
A historical article about a mathematical theorem may remain relevant for foundational learning without requiring a publication date this week. A question about current examination rules requires up-to-date official information.
Old does not automatically mean wrong, and new does not automatically mean authoritative. The time requirement depends on the question.
A relevance judgement should state whether currentness is a necessary condition and compare a retrieved resource accordingly.
15. A resource can be relevant but inaccessible
An academic paper may answer a question but require payment or specialist background the student cannot access. Its topical relevance remains separate from practical usability.
A learning service may offer an accessible explanation of the same concept, even if the specialist source is more comprehensive.
Record accessibility or fit as an additional field if it matters to the user. Do not silently confuse relevance with ease of access.
16. Reading level can matter to educational relevance
A Primary learner asks what evaporation is and encounters a highly technical thermodynamics derivation. The article may discuss the correct phenomenon but not support the learner’s intended first explanation.
An age-appropriate demonstration or clearly written introduction may receive a higher practical relevance grade for that need.
Level is a task condition, not a judgement that advanced material is inferior. The same specialist page may be highly relevant to a university student.
17. Resource format can be part of the need
A tutor searching for printable practice questions may not be satisfied by a commentary article that explains why practice matters but offers no exercises.
The page could be conceptually related, yet it fails the requested resource type. A format filter or clear content description can improve retrieval.
Keep subject relevance and format suitability visible as separate labels when diagnosing a search problem.
18. A single relevant paragraph may be enough
For a focused factual question, a reliable passage within a longer document can answer it directly. Relevance need not require every paragraph to discuss the query.
TREC’s English relevance judgement notes describe a particular binary task definition in which a document can be relevant if some of its content would be useful to the hypothetical report.
Different search tasks set different criteria. A classroom judge should follow the stated local guideline rather than generalise one historical evaluation policy to all search use.
19. A relevant result can be only partially useful
A page describes the meaning of a fraction but does not show how to compare equivalent fractions. For a student who asks specifically for comparison methods, the page offers prerequisite background.
Under a graded rubric, it might earn a partial-relevance label. Under a strict binary rubric requiring direct task completion, its classification depends on the published rules.
The assessor should explain which part helps and which part is missing.
20. A prerequisite guide is not always the final answer
A learner struggling with equations may need earlier teaching on signed numbers. A prerequisite article could be highly useful for a diagnosed blocker even though the original query says ‘solve equations’.
The information need can evolve after the learner discovers the actual difficulty. That does not mean the original evaluation rubric should silently change midway through a controlled test.
Record whether a new learning diagnosis created a new question, and evaluate the next search against that revised need.
21. Reader intent affects relevance
One reader wants a definition, another wants exercises and a third wants proof. A page with a rigorous derivation may score differently for the three legitimate intents.
A good index can help separate conceptual introductions, worked examples, practice, research references and official requirements.
Do not declare one document universally most relevant without identifying the user’s intended outcome.
22. Query ambiguity makes relevance uncertain
A user searches ‘scale’. They might mean a weighing instrument, map ratio, musical scale or a factor in a chart.
An assessor cannot reliably grade the results until the intended sense is established. Different interpretations create different correct results.
Clarify the subject when necessary and keep a record of the chosen information need before comparing retrieval methods.
23. Judges can disagree
Two careful assessors may assign different relevance grades to a borderline page because one considers its examples sufficiently useful and the other expects a complete worked answer.
The Stanford chapter on retrieval evaluation notes that relevance judgements contain subjective and task-dependent elements and are not perfect measurements.
Disagreement should prompt review of the rubric, the user’s need and the actual text rather than an unsupported allegation that one judge is careless.
24. Adjudication needs evidence and rules
If two assessors disagree, ask each to point to the passage or resource feature that supports their grade. Compare those observations with the published rubric.
A third reviewer or agreed rule can resolve consequential cases, while unresolved differences may be recorded where the task permits it.
The goal is a reviewable decision, not unanimous agreement obtained by pressuring the least confident assessor.
25. Judgements need stable document identifiers
A search evaluation may include two versions of the same article with different explanations. A judgement of the earlier version should not automatically be assigned to the later content.
Use stable document identifiers and version or snapshot references when comparing results over time. A title alone may not distinguish revisions.
The Concept Identifier guide concerns stable subject identity, while evaluation also needs reliable identifiers for specific documents and versions.
26. Qrels connect topics with document labels
In TREC terminology, qrels are relevance judgement records for a collection’s topics and documents. They can be used to evaluate retrieval runs.
The format and allowed grade values depend on the track. NIST’s How To TREC explains the terms assessor, document, qrels, pooling and query.
Do not invent an official qrel score or claim that an unjudged real web page has an authoritative NIST relevance label.
27. A test collection has three main parts
A retrieval evaluation needs a fixed document collection, a set of information needs or topics, and judgements describing which documents serve those needs. Search runs can then be compared against that reference.
Without a defined collection, recall may be impossible to calculate. Without a clear question, a relevance grade lacks an interpretable target.
A local ten-document exercise can demonstrate the principle, but large-scale evaluation requires much more careful design.
28. Pooling reduces assessment work
For large collections, judging every document for every topic is usually impractical. TREC often uses pooling: top results from several retrieval systems are combined into a set for assessors to judge.
This creates a practical sample but can miss relevant documents that none of the contributing systems retrieved. Evaluation can be affected by those incomplete judgements.
The TREC 2024 overview describes the approach and its limitations. A small classroom model should explain what was actually judged.
29. Unjudged is not automatically nonrelevant
A document outside the evaluation pool may simply not have been assessed. Treating it as nonrelevant for a particular calculation is an evaluation convention, not a factual discovery about its contents.
A responsible report distinguishes an observed irrelevant page from an unjudged page whose relevance remains unknown.
That distinction matters when a retrieval method finds genuinely new material not included in the original pool.
30. Binary and graded measures answer different questions
A binary metric focuses on whether an item passes a defined relevance threshold. Graded measures can take account of higher and lower levels of usefulness.
Measures such as discounted cumulative gain are designed for ranked graded retrieval settings, while precision and recall are common for binary judgements. The detailed mathematical definitions depend on the evaluation framework.
Choose a metric matching the user’s goal and the available judgement scale rather than selecting one because it produces a flattering score.
31. Precision can be calculated for a toy library
Imagine a fictional fixed collection containing four relevant pages for a specific reading question. Search A retrieves four pages, of which two are relevant.
The precision is 2 divided by 4, or 50 percent. This tells us half the returned set is useful under the declared binary relevance criteria.
It does not tell us what fraction of all relevant pages were found, nor whether the useful pages contain factually accurate explanations.
32. Recall needs a total relevant set
In the same fictional collection, four pages are relevant and Search A returns two. Its recall is 2 divided by 4, or 50 percent.
Now Search B retrieves three of the four relevant pages among seven results: recall is 3/4, or 75 percent, while precision is 3/7, approximately 43 percent.
Those values illustrate how one query may find more useful material while also returning more irrelevant pages. They are not measured search-engine results.
33. Precision at a cut-off helps examine top results
A parent may only read the first five search results. Precision at five asks what fraction of those five are relevant under the defined judgement rule.
If four of five meet the need, precision at five is 80 percent for that toy query. If two do, it is 40 percent.
Do not treat a single query’s top-five precision as an overall site-quality score. Many representative information needs are necessary for meaningful evaluation.
34. Rank position matters
A highly relevant introductory guide appearing first may serve a beginner better than the same guide buried under many tangential pages.
Graded ranking evaluation can give greater weight to highly relevant items placed earlier, but metric choices and scale definitions must be stated.
A useful site test should inspect what real target readers encounter, not merely the total number of indexed articles.
35. A search result snippet can distort judgement
A snippet may show a few matching words while omitting the paragraph that states a crucial limitation. Judging only the snippet can misclassify the full document.
Read enough of the actual resource to understand whether the information need is met. Some tasks allow passage-level judgements; others require whole-document labels.
Be explicit about the unit being judged and what content the assessor actually inspected.
36. The relevance rubric should precede scoring
If an editor decides what relevant means only after seeing the results, they may unconsciously favour a preferred search method or article.
Write the task, intended audience, required information and grade boundaries first. Use the same criteria across candidate queries or retrieval systems.
Then record exceptions or newly discovered information needs separately instead of silently altering the original evaluation.
37. Search relevance and task success are not identical
A page may be relevant and accessible, but the learner may still misunderstand the explanation or need guided practice. The retrieval system has found a useful candidate, not completed the educational process.
Assess comprehension or performance through a suitable task if the goal is learning. Do not infer mastery from a click or long reading time.
This is a crucial boundary between information retrieval and actual teaching.
38. The learner may need a different type of resource
A pupil who already understands a definition but cannot apply it may benefit from a worked example or carefully designed practice, rather than another introductory glossary.
A tutor can update the information need to ‘independent practice with feedback’ after diagnosing the gap.
That is not a contradiction of the earlier relevance judgement: the goal has changed, and a new retrieval decision is needed.
39. Worked English case: inference versus prediction
A fictional learner asks how to identify evidence supporting a reading inference. Search Result A offers a chapter summary, Result B explains how to connect a textual clue to a conclusion, and Result C forecasts what may happen next in a story.
Under the defined question, B is directly relevant, A may be partially useful and C may concern a different reading activity unless it explicitly teaches the required evidence link.
Ask the pupil to explain why each grade was assigned. A title containing inference does not by itself settle the evaluation.
40. Worked Mathematics case: solving an equation
A learner’s current task is to solve a linear equation with negative signs. A page about graphing quadratic functions is mathematically related but may not address the needed method.
Another guide that walks through equivalent transformations and sign checking could be highly relevant, even if its heading says ‘Preserving Equality’ rather than repeating the learner’s query.
Relevance judgements can therefore reward accurate conceptual matching rather than crude keyword similarity.
41. Worked Science case: evaporation and boiling
A child asks why clothes dry below the boiling temperature of water. A result on evaporation at room temperature directly addresses the problem; a guide to the boiling point under standard atmospheric pressure may provide partial background.
A page listing types of rain clouds might share water-cycle vocabulary without answering the specific mechanism question.
Use the defined question to explain the grades rather than assume all Science pages mentioning water belong in the same result set.
42. Worked current-policy case
A parent seeks a current official educational requirement. A historical article may explain earlier policy but cannot be treated as a complete current answer simply because its title matches.
The relevance evaluation should include the authority, year, applicable stage and whether the document’s claim has been superseded.
Current official facts require external verification. A search result’s relevance label is not a substitute for checking the governing source.
43. Worked project-resource case
An editor searches for ‘approved school-video logo’ and retrieves a brand discussion, a draft image and a file explicitly recorded as the current approved logo.
The brand discussion may be related but not suitable for the immediate editing action. The draft might be usable as historical context but not as current approved media.
Judge the result against the operational need, then verify the actual approval record. A file’s name containing approved should not be accepted without appropriate evidence.
44. A four-field judgement record
Use Query or Information Need, Document and Version, Relevance Grade, and Explanation. Add Date and Assessor when the evaluation is shared or repeated.
For a fictional query ‘equivalent fractions worked example’, a relevant document might be graded 3 with the explanation ‘Shows two values with a step-by-step numerical and visual comparison’.
A score without a rationale is harder to review. The explanation should refer to material actually present in the document.
45. Compare two assessors fairly
Give two learners the same search question, three short original documents and the four-level rubric. Ask them to grade independently and point to the supporting passages.
Where scores disagree, compare the precise information each thought necessary. Agree on whether the rubric needs an example or a clearer boundary.
The exercise teaches evidence-based classification, not a claim that a particular level of agreement proves the system is scientifically valid.
46. A negative test detects false relevance
Include a page with the right keywords but wrong subject sense—’scale’ in music when the question concerns map distance. The judge should reject or downgrade it under the intended context.
Also include a page with different wording but the correct concept, to test whether assessors can recognise useful semantic equivalence.
A balanced test checks both missed relevant items and attractive false matches.
47. Do not overfit the judgement scheme to one site
A local rubric designed only around one article template may reward that template while unfairly dismissing other valid explanations.
Test the categories across different content forms and legitimate user questions. A reader may need a table, diagram, exercise, direct answer or longer explanation.
Keep the information need central so the judgement does not become a disguised preference for a particular publishing style.
48. Relevance feedback can improve the next search
After a reader identifies relevant results, a search system can use terms from those documents to help reformulate the next query.
The Stanford chapter on relevance feedback and query expansion explains both explicit feedback and pseudo-relevance feedback.
Be careful: using top results as if they were relevant can amplify a wrong interpretation of the query. Confirm the intended sense before widening the search.
49. A controlled vocabulary helps evaluators communicate
If assessors use vague labels such as useful, kind of useful and maybe okay without shared definitions, later comparison becomes difficult.
A controlled set of grade labels with scope notes can help. But the labels’ existence does not make individual judgements correct.
The Controlled Vocabulary guide explains how to maintain consistent category meanings.
50. A crosswalk can help find relevant material
One educational library calls a topic ‘Reading Inference’, while another uses a narrower term for connecting missing relations across sentences. A crosswalk can route a search towards the suitable topic.
That relation does not automatically make every linked page relevant to the individual learner’s question. The content still needs inspection.
The Semantic Crosswalk guide explains how mapping strength and scope should be recorded.
51. A provenance record helps check an evaluation
A judgement based on an older article version may no longer match a page after substantial rewriting. A source and version record can explain why assessors previously gave a grade.
Keep track of the query, document version and relevant decision evidence. Do not overwrite historic judgements without noting what changed.
The Provenance Trail guide explores origin, revision and audit records.
52. What a small study library could measure
A tutor can begin with ten recurring search questions and a manageable collection. For each, identify the expected kind of useful resource and grade the first few results.
Look for repeated failure modes: wrong school stage, related but off-topic subjects, outdated official details, unclear titles or missing practice tasks.
Such a local audit can guide editorial improvements without being presented as a rigorous scientific comparison with large web search engines.
53. A staged review prevents false certainty
An assessor can mark a document Needs Inspection, Relevant, Partially Relevant or Not Relevant under the local process. An unresolved result should not be silently counted as definitely irrelevant unless the evaluation design explicitly requires that convention.
After content inspection, update the grade and record the basis. A second reviewer may be appropriate for contentious, high-impact or ambiguous cases.
Different statuses protect the distinction between a completed judgement and a document that merely appeared in the results.
54. Ten questions and short answers
1. Is keyword overlap sufficient for relevance? No. 2. Does relevance depend on the information need? Yes. 3. Can a factual error appear in a relevant document? Yes. 4. Can a correct article be irrelevant to one query? Yes.
5. Are binary and graded judgements both used? Yes. 6. Are qrels relevance records in TREC? Yes. 7. Can assessors disagree? Yes. 8. Is unjudged identical to proven irrelevant? No.
9. Does precision show the fraction of retrieved results judged relevant? Yes. 10. Does recall require the relevant population in a defined collection? Yes. 11. Can query expansion cause drift? Yes. 12. What is the practical goal? Judge whether a resource helps answer the actual question.
55. Frequently asked questions
Is a relevance judgement an objective fact? It follows a defined assessment task and criteria but includes subjective, context-dependent decisions. Does a high Google ranking prove relevance? No; the reader’s information need and the page’s actual content matter.
Can teachers use graded judgements? Yes, with a clear local rubric and examples. Is relevance the same as truth? No. Does a matching grade imply a pupil understands the article? No; learning requires separate evidence.
Why study this as vocabulary? It gives readers and editors a precise language for deciding whether a retrieved explanation serves a real goal rather than merely sharing a few words.
56. Research sources and next task
Core sources: Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, especially Chapter 8 on Evaluation; NIST How To TREC; and the TREC 2024 Overview on binary, graded and pooled judgements. TREC English Relevance Judgements documents one historical task-specific binary relevance policy.
The rubrics, queries, pupils and result counts in this article are constructed educational examples. They are not official TREC judgements or measured results from a live site.
Pick one search question that currently produces unsatisfactory results. Write what the reader actually needs, examine three results and label each Direct Answer, Partial Answer or Off-Topic with a quoted or paraphrased reason. Then distinguish any factual inaccuracies from relevance failures. You have used vocabulary to turn search frustration into an inspectable improvement task.