How Superintelligence Works | Agent Retrieval Coverage — Recall@K, Missing Evidence and Source-Aware Answers

An AI research assistant can produce a beautifully written answer while missing the document that would have changed its conclusion. Retrieval coverage asks whether the search stage found the relevant evidence—not merely whether the final response sounds coherent. For questions about school admissions, a top-ranked overview may be useful but insufficient if a separate official exception, effective-date notice or eligibility condition was not retrieved. Search relevance, evidence completeness and the correctness of a generated answer are related but different questions.

The practical method is to define which sources are relevant to the exact task, create independently reviewed query-document labels, measure top-K retrieval with clear denominators, test negative questions and check that the answer uses the retrieved evidence faithfully. This article works through an invented school-research benchmark, comparing recall@K and precision@K, and shows how a changed effective year can reverse which source is relevant. All results are fictional teaching data, not observed eduKate search performance or proof of artificial superintelligence.

Continue from How Super Intelligence Works. The Conflicting Evidence guide explains which authority governs after documents are found; API Pagination explains traversing a dataset completely. Retrieval coverage focuses on whether the ranking and selection process surfaced the documents needed to answer the actual user question.

Choose a useful route

• Evaluating a search assistant? Start with relevant labels, recall@K and precision@K.
• Concerned about missing official evidence? Read claim coverage, effective dates and negative queries.
• Learning Mathematics? Solve the five-result puzzle and calculate the alternative K trade-off.
• Building a RAG system? Compare chunk-level measures, authorised corpora and hybrid retrieval.

Contents

• 1. Define relevant evidence before scoring a ranker
• 2. Recall@K asks what was missed
• 3. Precision@K asks what filled the selected window
• 4. First relevant position matters for some jobs
• 5. NDCG handles graded usefulness and order
• 6. Distinguish a document from its chunks
• 7. Source authority and retrieval relevance are different
• 8. Access-filtered recall is a real security requirement
• 9. A fictional five-result retrieval puzzle
• 10. Topical hits cannot substitute for sufficient evidence
• 11. Use negative queries to test abstention
• 12. Check hybrid keyword and semantic retrieval separately
• Frequently asked questions
• Primary sources and evaluation methods
• Continue learning

1. Define relevant evidence before scoring a ranker

A retrieval benchmark cannot be meaningful until somebody defines relevance for each query. A document about Mathematics may be topically related yet useless for determining current subject selection at one school. Another document may include the decisive eligibility rule but use unfamiliar vocabulary and rank poorly. A good judgement checks the specific information need, audience, effective time and scope rather than similarity of the words alone.

Build test pairs of query and candidate source. For each query, establish which official documents and passages support the answer, which are merely background and which conflict or are obsolete. The relevance label may be binary for a simple teaching exercise, or graded when a document only partially supports the task. Preserve the method and reviewer because ground truth can itself contain mistakes.

An official notice is not always relevant to every school cohort. A genuine notice for a prior intake may be historical evidence but not the source to use for this year’s decision. A researcher should identify the effective date and relevant programme before marking it relevant. This prevents an evaluation set from rewarding the retrieval of authentic but inapplicable materials.

A model can assist in preparing candidate labels, but it should not independently certify that its own preferred result is sufficient. A qualified human or directly checkable source contract can establish the expected evidence. Otherwise the retriever and evaluator might share one erroneous assumption, producing a flattering score without finding the actual missing rule.

Return to contents or continue to Recall@K asks what was missed.

2. Recall@K asks what was missed

Recall at K is the share of the known relevant items that occur in the first K retrieved results. In a simple one-query example, suppose four documents have been independently judged relevant in the eligible corpus and the top five results contain three of them. Recall@5 is 3 ÷ 4 = 75%. The missed fourth relevant document matters even when the first result looks convincing.

This denominator is the number of known relevant items, not the number of documents returned. Confusing it with precision gives a different measure. Microsoft Azure’s information-retrieval guidance defines recall at K as retrieval of relevant items among the total relevant items. The score says something about the retrieval stage under the known labelled collection, not whether the generator later cited each passage correctly.

Recall is sensitive to the completeness of the reference labels. If the evaluation team has never discovered a relevant source, the denominator undercounts the actual need. A score of 100% against an incomplete label set can be misleading. Include diverse source discovery and reviewer challenge cases, then record which relevant documents were known at the evaluation cutoff.

More results can increase recall, but a larger K costs tokens and reviewer effort and may include unrelated or unsafe documents. The objective is not to copy the entire internet into the model’s context. Choose K according to task consequence, source availability, retrieval quality and the generator’s ability to use the evidence.

Return to contents or continue to Precision@K asks what filled the selected window.

3. Precision@K asks what filled the selected window

Precision at K is the number of relevant results among the first K, divided by K under the chosen convention for a fully populated result list. With three relevant documents in five returned results, precision@5 is 3 ÷ 5 = 60%. Recall@5 was 75% in the earlier example. Both can be true simultaneously because they use different denominators.

A system with high precision places few irrelevant pages near the top, reducing model distraction and unnecessary source checking. But one missing decisive source can still make the answer wrong for the user. A shorter result list can improve apparent precision while hurting recall. The correct balance depends on whether the user needs one good explanation or every applicable condition across multiple documents.

Rank ordering does not enter this simple precision@K formula beyond membership in the top K. A relevant document at rank five contributes the same to precision@5 as one at rank one, even though a generator may give earlier passages more attention. Complementary rank-sensitive metrics, such as reciprocal rank or normalized discounted cumulative gain, address aspects of ordering.

An empty or very short result list requires an explicit evaluation convention. Do not silently divide by a convenient number after a query returned fewer than K items, and do not report perfect precision for a question with no known relevant source merely because nothing was retrieved. Negative and zero-answer cases should be handled separately according to the test design.

Return to contents or continue to First relevant position matters for some jobs.

4. First relevant position matters for some jobs

Mean reciprocal rank, or MRR, rewards ranking a first relevant result high. For one query, reciprocal rank is one divided by the rank of the first relevant result; a first relevant result at position two gives one-half. If no relevant result occurs within the evaluated cutoff, the common truncated convention assigns zero. Average across queries to obtain MRR@K.

This is useful when the user needs any one valid source and the interface shows only the earliest results. A parent seeking one official general overview may benefit from the right source at rank one rather than buried at rank nine. But a task needing several separate eligibility conditions cannot be judged solely by where its first relevant document appears.

Suppose three fictional queries have first relevant results at ranks one, two and no relevant result within K. Their reciprocal ranks are 1, 1/2 and 0. MRR over the three is (1 + 0.5 + 0) ÷ 3 = 0.5. That average does not mean half of all relevant documents were found. It describes the observed position of the first hit under the chosen evaluation rules.

Microsoft’s retrieval guidance discusses MRR alongside precision and recall, and stresses testing positive and negative examples. Select metrics according to the user job. A summary requiring multiple factual clauses needs breadth and coverage checks even if MRR is excellent. A perfect first hit does not prove the rest of the evidence was retrieved.

Return to contents or continue to NDCG handles graded usefulness and order.

5. NDCG handles graded usefulness and order

Normalized discounted cumulative gain, or NDCG, can evaluate rankings when documents have graded relevance labels rather than only relevant versus irrelevant. It rewards placing more useful documents earlier and normalizes against the ideal ranking under a stated gain convention. Scikit-learn documents ndcg_score and the relevant mathematical assumptions. Exact values depend on the chosen relevance scale and rank cutoff.

For a school question, a current official admissions requirement might earn a higher relevance grade than a general educational overview. A related outdated guide could be background with limited grade or zero relevance for the actual decision. NDCG can show whether the ranker prioritizes the most useful material, although assigning grades consistently remains a real evaluation challenge.

Do not treat an automated relevance score as legal or institutional authority. The model could mark a visually polished unofficial page highly relevant, while the actual issuer’s current notice has the decisive governing text. Human and source-specific review still determine which claims can be made. The metric measures the ranking under a reference, not the truth of that reference.

Use rank-sensitive and set-coverage measures together when appropriate. NDCG can reward putting the best sources earlier, while recall at K shows how much relevant evidence was omitted. A model that retrieves one excellent document and misses a required appendix may perform well on one score while failing the user’s complete task.

Return to contents or continue to Distinguish a document from its chunks.

6. Distinguish a document from its chunks

RAG systems often split documents into smaller passages for indexing. If one long source produces ten chunks, the search results may contain several chunks from that same document. Counting each chunk as an independent official source can inflate apparent breadth. Label whether the retrieval metric counts passage relevance, unique document coverage or both.

A current education policy may require a table on one page and a definition on another. A chunk boundary might separate a qualification from the sentence it restricts, causing the retriever to find a misleading partial passage. Indexing should preserve enough context or neighboring passage relationships to make the result interpretable. The original source document must remain discoverable for authoritative checking.

Deduplicate where appropriate without discarding distinct evidence. Two identical snippets copied from one notice do not necessarily represent two corroborating institutions; two passages from separate current official issuers might genuinely support different clauses. Preserve source ID, revision, issuer, passage and effective scope. A model-generated citation with no such provenance does not prove evidence diversity.

Chunk-level labels are useful for diagnosing missed passages, but final-user completeness should remain tied to the real question. A search could retrieve many fragments and still omit the eligibility exception that changes the decision. Evaluate whether the needed claim-bearing passages entered the model context, not simply how many chunks received high similarity scores.

Return to contents or continue to Source authority and retrieval relevance are different.

7. Source authority and retrieval relevance are different

A page can be semantically close to a question while lacking authority for it. An unofficial forum may describe a school programme accurately but cannot replace the relevant official admissions notice for a consequential decision. Retrieval ranking should surface the source appropriate to the user’s purpose, and answer generation should qualify interpretations against that source.

Conversely, an official source may be authoritative but apply to a different year or programme. Search evaluation must check temporal and contextual applicability rather than treating every government-hosted result as universally relevant. An outdated official notice is still genuine historical evidence; it is not automatically the answer to a present-day requirement.

Conflicting sources require careful handling. A retrieval set can have high recall but contain two apparently inconsistent notices because their scopes differ. A model should identify the effective dates and responsible issuers before choosing. If a genuine unresolved conflict remains, abstention and referral can be correct. Fetching more pages is not always the solution to an authority dispute.

A source diversity measure should look through mirrors and syndicated copies. Five pages repeating the same original may appear to be five confirmations, when only one publisher supplied the underlying claim. Source lineage helps prevent false consensus. Ranking and provenance must work together to produce evidence a reader can challenge.

Return to contents or continue to Access-filtered recall is a real security requirement.

8. Access-filtered recall is a real security requirement

An agent serving several schools must not retrieve private documents simply because they are semantically relevant. The eligible search corpus for a given query includes only sources the authenticated user is allowed to access. Evaluate coverage under that authorized corpus, not by granting all documents to an offline judge and rewarding a retriever for disclosing restricted ones.

Suppose four school documents could help answer a topic question, but two belong to another tenant and are not permitted for the requesting parent. A system that correctly excludes them should not be penalized for failing to retrieve those prohibited documents. Its permitted-answer task may have a narrower scope or require an explanation that certain data are inaccessible. The relevant ground truth must match the actual authorised information need.

Permission checks should happen before private content enters the model context. Filtering restricted snippets only after generation can leave facts from them embedded in the answer. The Multi-Tenant Isolation chapter explains the broader data-flow boundary. Retrieval coverage metrics must not incentivise breaking it.

Record an effective policy version and source permissions alongside evaluation labels. If a teacher’s access expires, a previously relevant private document becomes ineligible for a new request. A cached retrieval result must be rechecked under current scope. Evaluation that ignores authorization changes can mistake correct protective denials for retrieval defects.

Return to contents or continue to A fictional five-result retrieval puzzle.

9. A fictional five-result retrieval puzzle

Imagine a question about an invented school’s current elective-subject process. The verified evaluation set identifies four genuinely relevant eligible documents. The retriever returns five candidates: a current overview, an unrelated past-year flyer, the official school subject table, an unofficial discussion page and a relevant exception note. Exactly three of those five are labelled relevant under the task.

Recall@5 is three retrieved relevant documents divided by four known relevant ones, giving 3/4 = 75%. Precision@5 is three relevant among five returned, giving 3/5 = 60%. The fourth relevant document was missing: suppose it describes an applicant cohort exception. The model’s final answer could therefore omit a condition that changes what the parent should do.

Now increase K to eight and suppose the missing relevant document appears at rank seven, while the extra results are otherwise irrelevant. Recall@8 becomes 4/4 = 100%, but precision@8 becomes 4/8 = 50%. The coverage improved while the fraction of relevant results declined. The example shows why neither metric alone defines the best retrieval configuration.

These values describe one invented query and its assumed relevant set. They are not a production benchmark. A separate evaluation must determine whether the generator actually noticed the rank-seven exception, cited it and gave the correct qualified answer. The input retrieval score does not certify successful reasoning over a long context.

Return to contents or continue to Topical hits cannot substitute for sufficient evidence.

10. Topical hits cannot substitute for sufficient evidence

Suppose an answer claims three separate things: the relevant subject is offered, the student meets an eligibility rule and the application deadline is current. One retrieved overview may support only the first claim. A model that cites it after every sentence may create the appearance of evidence while the two consequential conditions remain unverified.

Build a claim-to-source matrix. List each material assertion, its authoritative source, the applicable effective period and whether the retrieved passages contain it. A complete answer either supports each material clause or explicitly marks uncertainty and directs the reader to the responsible institution. Citation count alone is not evidence completeness.

Separate retrieval failure from generation failure. If the correct eligibility note was never retrieved, the retriever or indexing configuration needs investigation. If it was retrieved but the model ignored it, the problem is context use or reasoning. If the document was found and cited but misrepresented, the issue is claim support. Those errors require different repairs.

An answer can correctly refuse to make a current claim when applicable evidence was unavailable. It should not invent a deadline because an evaluation dashboard rewards answering rather than abstaining. The most trustworthy service goal is a justified user decision, not a guaranteed fluent response to every question.

Return to contents or continue to Use negative queries to test abstention.

11. Use negative queries to test abstention

Some test questions have no answer in the eligible corpus. A parent may ask for a programme the institution does not offer, or a private record the user cannot access. A retriever that always returns superficially related passages can prompt the generator to invent a plausible but unsupported response. Include such negative questions in evaluation.

For a negative query, ordinary recall can be undefined because there are no relevant items. Do not silently treat a zero denominator as perfect performance. Instead test the system’s ability to recognize lack of evidence, avoid unauthorised sources and communicate the absence of a verified answer. Report negative-query results separately under a declared method.

False positive retrieval can be more harmful in a high-stakes question than a moderate loss of recall. An obsolete school-policy page ranked first may cause a wrong application decision. The consequence and the expected refusal should be encoded in the benchmark before results are inspected. A correct abstention is an accepted service outcome, not a model embarrassment.

A negative reference set can be wrong too. The source corpus may have gained a new current notice since labels were created. Version the snapshot and evaluation date. When an official document is added, update the benchmark under controlled review rather than continue rewarding an agent for saying no answer exists.

Return to contents or continue to Check hybrid keyword and semantic retrieval separately.

12. Check hybrid keyword and semantic retrieval separately

Keyword search can find exact subject codes, document titles and rare proper nouns that semantic search misses. Semantic retrieval can help with paraphrases and questions that do not reproduce source wording. Hybrid search can combine these strategies, but each still requires source authorization and evaluation. A fused ranking is not automatically complete or current.

Test exact-code and paraphrased queries. A parent might type the official course code, while another asks about the subject in ordinary language. Their underlying permitted information need may match, and both should recover the decisive official documents. Source labels should use stable document identities so the evaluation can compare both paths.

Some systems apply a reranker only to the initial candidate pool. If the critical document was not retrieved into that pool, reranking cannot create it from nothing. A high-quality reranker may improve the position of known candidates while leaving a recall bottleneck unchanged. Measure first-stage candidate recall separately from final top-K relevance.

Index freshness is another source of misses. A newly issued notice may exist in the authoritative source but not yet in the search index. The system must distinguish an indexing lag from a query-language problem. A source freshness check or direct lookup may be appropriate for a current deadline. A model’s confidence is not evidence that the index contains every recent update.

Return to contents.

14. Build a source-label set that readers can audit

Choose a collection of test questions and independently assemble the relevant source passages before tuning the search algorithm. Record the school, subject, cohort, effective date, issuer and access scope. Relevance is conditional: a current rule for a different institution may be genuinely official yet inappropriate. A reviewer should be able to explain why each source was included or excluded, rather than merely assign a score based on familiar keywords.

An important test question may have two or three necessary passages. A course overview can explain the pathway, while a separate issuer notice establishes the current entry condition. If the benchmark records only the overview, a system can appear to achieve perfect recall while omitting the decisive eligibility clause. Another reviewer should challenge the label set with a deliberate search for missing exceptions.

Keep label versions. An official notice issued later can change which passages are applicable. The benchmark should say which date and available corpus it represents, and a correction should preserve the earlier record for comparison. A model performing well against last year’s benchmark does not automatically establish current advice for today’s school application.

Return to contents to choose another metric.

15. Separate first-stage candidate recall from final top K

Many retrieval systems create a broad candidate pool before applying a reranker. If the required eligibility document is missing from the initial candidate pool, the reranker cannot move it to the top because it never received it. Measure candidate recall at its own cutoff, then measure the final user-facing K. A low candidate recall points to indexing, query expansion or source coverage rather than a ranking-only defect.

Suppose a fictional search retrieves fifteen candidates that contain all four relevant documents, then a final reranker selects five but includes only three of them. Candidate recall@15 is 4 ÷ 4 = 100%, while final recall@5 is 3 ÷ 4 = 75%. The distinction suggests a ranking or truncation problem after retrieval rather than an absence of the documents from the corpus.

Change the example: the initial fifteen contain only two of the four relevant documents. Even a perfect reranker cannot reach recall@5 above 2 ÷ 4 = 50% using only that pool. Increasing reranker complexity or changing the model’s writing instructions cannot recover a source it never sees. The test should identify which stage constrained the evidence.

Return to contents to choose another metric.

16. A multi-query metric table makes comparisons possible

Take four invented queries with top-four retrieval outputs. The first has three relevant results among four and four relevant items in the reference set. The second has two among four, with two relevant in total. The third has one among four, with three relevant in total. The fourth has zero relevant among four, with one known relevant source. Their per-query recall@4 values are 3/4, 2/2, 1/3 and 0/1.

Macro-averaged recall@4 is (0.75 + 1 + 1/3 + 0) ÷ 4 ≈ 52.08%. Per-query precision@4 values are 3/4, 2/4, 1/4 and 0/4, giving a macro average (0.75 + 0.5 + 0.25 + 0) ÷ 4 = 37.5%. These are teaching arithmetic values chosen to show why different denominators produce different metrics.

A micro-averaged alternative over the invented labelled set counts six retrieved relevant items out of ten reference-relevant items, giving 6/10 = 60% recall. It differs from macro recall because queries contribute different numbers of relevant documents. A report needs to declare which averaging scheme was used and why, especially when a few complex questions have many relevant passages.

Return to contents to choose another metric.

17. Search result confidence is not source completeness

A similarity score or ranker confidence can help order candidates, but it is not a calibrated probability that the complete answer has been found. A document can be close in embedding space to a user’s words while omitting an exception, using a wrong effective year or belonging to another tenant. A system that turns high similarity into a final certainty statement hides the remaining evidence questions.

Consider an assistant that retrieves three strong passages and produces a clear current policy summary. A qualified reviewer then finds an omitted official appendix. The failure belongs partly to recall and partly to the final answer’s completeness. More fluent phrasing cannot recover the omitted clause. The right fix may be a query expansion, source-index repair or requirement to consult that appendix for the affected task category.

Uncertainty should be expressed in useful language: what is verified, which condition remains unconfirmed and where the person can check with the responsible institution. An assistant should not burden every parent with metric names in the final answer, but its underlying evaluation should make the limits inspectable by staff and engineers.

Return to contents to choose another metric.

18. Use source diversity without counting mirrors as authorities

Different search results may be mirrors or summaries of the same original notice. If five pages repeat one outdated paragraph, they do not constitute five independent institutional confirmations. Record the original issuer and source lineage where possible. Deduplicating by exact URL alone can miss copies on different hosts, while merging unrelated sources on textual similarity can erase genuinely independent conditions.

A school-specific rule may require a current school document plus broader ministry guidance. Those are different roles in the evidence chain, not interchangeable endorsements. Evaluate whether the two necessary issuers are represented, then check their applicable dates. A ranker should not fill all five top positions with derivatives of one overview while omitting a different official source.

Source diversity is not an invitation to include inaccessible private material. A user may legitimately be barred from another school’s internal notices. The authorised set remains the ceiling for search. Diversity describes independent permitted evidence for the task; it does not justify crossing tenant boundaries or quoting a restricted record because it happens to strengthen the answer.

Return to contents to choose another metric.

19. Evaluate the generated answer after retrieval

Retrieval metrics tell you about source selection, not how well the model read the chosen passages. The final answer might omit a key qualification, reverse a comparison or cite a page that doesn’t contain the claim. Use a claim-to-evidence check alongside top-K measures. For each material assertion, show which retrieved passage supports it and whether the cited source matches the relevant institution and time.

Microsoft’s RAG evaluation guidance distinguishes document retrieval from groundedness, answer relevance and response completeness. This separation is practical. A retriever might achieve strong recall but the model might ignore a source buried in a long prompt. Alternatively, the model might produce a correct-looking statement that was never supported by the retrieved context. Both need diagnosis rather than one blended score.

Define the user’s accepted outcome. A parent asking for a current deadline needs the right official date and any material exceptions, not a list of similar web pages. A safe answer may say that a current official source could not be verified, with a clear route to check. Measured retrieval performance supports that decision, but does not replace it.

Return to contents to choose another metric.

20. Fault diagnosis by which source was missing

Failure one: a required document exists in the source repository but not the search index. Repair ingestion and index freshness, then verify by stable ID. Failure two: the correct document enters the candidate pool but is ranked outside the final K. Repair reranking or selection while preserving the access boundary. Failure three: several chunks of one document crowd out another required source. Repair chunk grouping, document-level diversity or context allocation.

Failure four: an obsolete official notice outranks a current one. Repair effective-date and issuer-aware relevance checks, not merely a higher embedding similarity threshold. Failure five: a restricted tenant document appears in the model’s context. Repair permission-aware retrieval before generation; post-hoc answer filtering is insufficient. Failure six: the answer invents a missing exception despite correct retrieval refusal. Repair final groundedness and abstention gates.

Each failure needs an independent regression fixture. Save the query, expected current source set, accessible corpus and relevant output receipts without copying real children’s confidential records into a public test. Re-run the exact case and a changed-condition example after repair. A green summary produced by the model cannot establish which index and source version it actually used.

Return to contents to choose another metric.

21. A changed-context exercise for school subject selection

Imagine a fictional school publishes three sources: a public subject overview, an eligibility table and an exception notice. For a general question about offered subjects, the overview may be sufficient. For a student’s current eligibility decision, the table and exception notice can both be necessary. The same ranker and index may therefore need different acceptance rules depending on what the user asks.

Now the school issues a new table effective for next year’s intake. A question about last year’s policy should refer to the historical table; a current planning question should use the new one. A benchmark that marks both as equally relevant without effective-date distinctions rewards ambiguity. Keep source revision, cohort and date within the reference label and the retrieved context.

Change the access constraint: the parent’s account cannot see internal staff notes. The evaluator should not include those notes among the permitted relevant documents, even if they discuss the subject perfectly. The assistant can explain the public rule and refer the parent to an authorised staff member for the missing personal decision. Correct retrieval respects both relevance and access.

Return to contents to choose another metric.

22. What a reader should conclude from a retrieval score

A single high number can conceal an omitted material rule. Recall@K is useful when the labelled relevant collection is reasonably complete; precision@K helps diagnose irrelevant selected results; MRR rewards early first hits; NDCG accounts for graded position-sensitive relevance. None alone proves that a student’s individual education pathway was established correctly.

Ask what corpus, labels and reporting window produced the result. Were negative queries included? Did the scoring exclude documents the requesting user cannot access? Did labels include current official notices or only convenient sample pages? Did the final model cite the documents accurately? These are ordinary questions about evidence that matter more than an impressive chart title.

Builders can begin with ten representative permitted questions and a reviewed set of source IDs. Measure top-K recall and precision, inspect missed decisive passages, and add a final answer-completeness check. Use synthetic cases for confidential workflows. Scale only when the retrieval measures explain actual reader errors and guide a demonstrably useful correction.

Return to contents to choose another metric.

• 14. Build a source-label set that readers can audit
• 15. Separate first-stage candidate recall from final top K
• 16. A multi-query metric table makes comparisons possible
• 17. Search result confidence is not source completeness
• 18. Use source diversity without counting mirrors as authorities
• 19. Evaluate the generated answer after retrieval
• 20. Fault diagnosis by which source was missing
• 21. A changed-context exercise for school subject selection
• 22. What a reader should conclude from a retrieval score

Frequently asked questions

What does recall@K measure?

The share of known relevant, eligible documents that appear among the first K retrieved results under a specified reference set.

How is precision@K different?

It divides relevant top-K results by the number of considered results, focusing on the proportion of useful retrieved items.

Can high MRR prove an answer is complete?

No. It rewards an early first relevant result, even when a later required document is missing.

Should a private document count as missed evidence?

Not for a user who is not entitled to it. The eligible corpus and evaluation labels must respect current permissions.

What if the corpus has no relevant answer?

Use explicit negative-query and abstention evaluation, rather than dividing recall by zero.

Does a great retrieval score prove the generated explanation is true?

No. The model may ignore, misinterpret or miscite the retrieved evidence. Evaluate claims and final outcomes separately.

Primary sources and evaluation methods

Sources reviewed on 24 October 2026. Every fictional school, test query, retrieved result, relevance judgement and numerical metric here is an original teaching example, not a real production score.

• Microsoft Azure RAG Information Retrieval Guide: Precision@K, recall@K, MRR and positive/negative evaluation.
• Singapore Government RAG Playbook — Evaluation: Ranking metrics, groundedness and retrieval evaluation in public-service context.
• Scikit-learn NDCG Score: Graded relevance and order-sensitive normalized ranking measures.
• Google ML Glossary: Terminology for recall@K and ranked retrieval.
• NIST AI RMF Playbook — Measure: Validity and limits of sampled evaluation evidence.

Continue learning

Related guides: Super Intelligence curriculum; How X Works hub; Conflicting Evidence; API Pagination; Multi-Tenant Isolation; Cache Invalidation; Gold Standard of Education. A search result must be relevant, permitted, sufficiently complete and correctly used before it supports a user-facing decision.