A new search model launches.
Clicks go up.
Everyone celebrates.
Then support tickets reveal that users are clicking more because the first result is wrong and they must keep trying.
Search evaluation is the discipline of measuring whether the system actually helps the receiver complete the intended information task—not merely whether it returns results, earns clicks or improves one convenient metric.
This is the fourth pillar beneath How Search Works. The master owns the whole search mechanism. This article owns the verification loop: relevance judgments, candidate recall, ranking metrics, online experiments, latency, reformulation, click bias, subgroup performance and the world-return question of whether users actually found and used the right evidence.
Quick Read
Search evaluation has several layers. First-stage evaluation asks whether relevant documents enter the candidate pool at all, often using recall at K. Ranking evaluation asks whether better documents appear earlier, using measures such as reciprocal rank, mean reciprocal rank, discounted cumulative gain and nDCG. Precision and recall measure different error directions and remain useful for set-oriented retrieval. Offline evaluation requires representative queries and relevance judgments; online evaluation observes real behaviour through controlled experiments, reformulation, abandonment, latency and task completion. Clicks are useful signals but biased by position and presentation. A mature evaluation system therefore combines offline and online evidence, segments results by query type and receiver, tests freshness and permissions, and investigates which search stage caused a failure rather than reporting one undifferentiated score.
real tasks → representative query set → relevance judgments → candidate-recall tests → ranking metrics → latency/cost checks → controlled online experiment → task success/failure analysis → stage diagnosis → correction → repeat
Evaluation Starts by Naming the Search Job
Known invoice lookup.
Exploratory literature review.
Current train status.
Finding a suitable Primary Science lesson.
These should not share one success metric.
Evaluation becomes meaningful only after the receiver, task and acceptable failure mode are clear.
Relevance Judgments Create the Offline Reference
For a set of test queries, human assessors or a governed evaluation process judge which documents are:
- not relevant;
- partially relevant;
- highly relevant;
- authoritative/current where that matters.
Those judgments let retrieval and ranking models be compared without waiting for live users.
Binary Relevance Is Sometimes Too Crude
For the query how photosynthesis works primary 5:
- a Primary 5 explanation may be highly relevant;
- a JC biochemistry article may be scientifically related but pedagogically wrong;
- a page mentioning the word once may be barely relevant.
Graded relevance captures these differences better than a simple yes/no label.
Judgment Guidelines Must Be Explicit
Two assessors disagree.
Before blaming the assessors, ask whether the task definition was underspecified.
Guidelines should state:
- receiver level;
- query intent;
- freshness requirement;
- authority requirement;
- whether partial answers count;
- whether duplicates count independently;
- whether inaccessible documents are eligible.
Candidate Recall Must Be Evaluated Before Ranking
The judged best document is absent from first-stage top 100.
Final ranking fails.
That is not primarily a reranker error.
Candidate Generation owns this first-stage gate.
Measure recall@K:
Of the relevant documents we expect, how many are present inside the first-stage top K?
Recall and Precision Measure Different Error Directions
Recall: how much relevant material did we retrieve?
Precision: how much retrieved material was relevant?
A broad evidence search may favour recall.
A top-five product result list may favour precision.
Neither metric is universally “better.”
Precision@K Matches Limited Screen Space
If the user sees ten results, precision@10 asks how many of those ten are relevant.
It ignores ordering among the ten, so it should not be the only ranking metric.
Reciprocal Rank Rewards the First Useful Result
For a query with one dominant correct target:
RR = 1 / rank_of_first_relevant_result
Relevant at rank 1 → 1.0.
Rank 2 → 0.5.
Rank 10 → 0.1.
Mean Reciprocal Rank averages this across queries.
MRR Is Strong for Lookup-Like Tasks
Find the official policy.
Find invoice 8472.
Find the correct support article.
These often care heavily about how soon the first acceptable answer appears.
MRR is less expressive when many graded relevant documents matter throughout the ranking.
Discounted Cumulative Gain Rewards Graded Relevance Near the Top
DCG assigns greater value to highly relevant results and discounts value as rank gets deeper.
This matches a common user reality:
a highly relevant document at rank 2 is usually more useful than the same document at rank 50.
nDCG Normalises Against the Ideal Order
Different queries can have different numbers and grades of relevant documents.
Normalised DCG compares observed ranking gain with the best possible ordering for that judged query.
This makes nDCG useful for comparing ranking quality across a diverse query set with graded relevance.
One Metric Cannot Diagnose the Whole Search System
nDCG fell.
Why?
- query understanding chose the wrong entity;
- candidate recall dropped;
- reranker order worsened;
- permissions removed the correct result;
- index was stale;
- judgment set changed.
A top-level metric is an alarm, not a diagnosis.
Stage Metrics Create a Failure Map
- query interpretation accuracy;
- candidate recall@K;
- fusion contribution;
- reranker nDCG/MRR;
- freshness violation rate;
- permission leakage tests;
- latency by stage;
- final task success.
This converts “search is worse” into an actionable repair route.
Offline Query Sets Must Represent Real Traffic
Test only easy popular queries.
Model looks excellent.
Long-tail names, typos, multilingual queries and rare failure searches still collapse.
A representative evaluation set should cover:
- head queries;
- long tail;
- ambiguous queries;
- exact lookup;
- freshness-sensitive queries;
- multilingual queries;
- zero-result cases;
- permission-sensitive queries;
- high-consequence tasks.
Hard Queries Deserve Their Own Slice
An average metric can improve because common easy queries improve slightly while difficult important queries degrade badly.
Segment evaluation by intent, domain, language, device, geography and consequence where those dimensions matter.
Freshness Evaluation Needs Time-Indexed Judgments
A result judged correct in January may be stale in September.
Static test collections become misleading for current-status queries unless labels or source snapshots preserve their temporal context.
Evaluation should record:
- judgment date;
- source version;
- currentness requirement;
- whether the answer is historical or current.
Authority Evaluation Is Separate From Topical Relevance
A forum post can be extremely relevant to a government-policy query.
The official policy remains more authoritative for what the law/policy currently says.
Evaluation may need separate labels for:
- topical relevance;
- source authority;
- freshness;
- jurisdiction;
- evidence quality.
Collapsing all of these into one relevance grade can hide why the result is good.
Clicks Are Behaviour, Not Ground Truth
Users click rank 1 more often partly because it is rank 1.
This is position bias.
A higher click-through rate can therefore mean:
- better relevance;
- more attractive title;
- higher position;
- misleading snippet;
- curiosity;
- user confusion requiring extra clicks.
Clicks are evidence, not truth labels.
Dwell Time Is Ambiguous Too
Long dwell can mean:
- excellent detailed answer;
- confusing page that takes too long to parse;
- user left the tab open.
Short dwell can mean:
- instant answer;
- irrelevant result;
- accidental click.
Behavioural metrics need task context.
Reformulation Is Often Strong Failure Evidence
User searches:
secondary 3 chemistry equilibrium
Then immediately:
secondary 3 chemistry equilibrium simple explanation
The first results may have been topically relevant but at the wrong level.
Reformulation reveals a mismatch in receiver representation, not necessarily subject retrieval.
Abandonment Can Mean Success or Failure
User sees opening hours directly in the result and leaves.
Successful zero-click search.
User sees useless results and leaves.
Failed abandonment.
Interpret abandonment through the search surface and task.
Task Completion Is the Strongest Product-Level Metric
Did the student reach the right lesson?
Did the finance user open the correct invoice?
Did the traveller find an accessible entrance?
Did the researcher identify the authoritative paper?
When observable ethically and reliably, task completion closes the loop between ranking metrics and real usefulness.
Online A/B Tests Measure Causal Product Changes More Directly
Randomly assign eligible traffic to Search A or Search B.
Hold other factors stable.
Compare outcomes.
This can test whether a new reranker causes better user behaviour rather than merely correlating with different traffic.
A/B Tests Need Guardrail Metrics
Main metric improves.
Latency doubles.
Zero-result rate rises for one language.
Permission failures increase.
The launch is not automatically a success.
Guardrails can include:
- p95/p99 latency;
- error rate;
- zero-result rate;
- privacy/security incidents;
- subgroup quality;
- cost per query;
- reformulation rate.
Interleaving Can Compare Rankers Efficiently
Instead of sending separate users entirely different rankings, some search experiments interleave results from two rankers into one list and infer preference from user interactions.
This can detect ranking preference with fewer observations in appropriate settings.
The method still relies on behavioural assumptions and should not replace offline relevance analysis or task success.
Latency Is Part of Relevance in Interactive Search
A perfect answer delivered after 20 seconds can be practically worse than a very good answer in 300 ms for some tasks.
Search evaluation should include:
- median latency;
- p95/p99;
- timeout rate;
- stage breakdown;
- quality under fallback.
Cost Must Be Measured Beside Quality
Model B improves nDCG by 0.2% and multiplies GPU cost by ten.
Whether that is worthwhile depends on consequence and business/product value.
Search quality is constrained optimisation, not a metric-maximisation contest without resource limits.
Query Understanding Needs Its Own Evaluation Slice
Query Understanding can fail before retrieval:
- wrong typo correction;
- wrong entity;
- lost negation;
- wrong date parse;
- wrong filter;
- over-expansion.
Evaluation should label and measure those errors separately from ranking.
Reranking Needs Counterfactual Diagnostics
The correct result was candidate rank 2.
Reranker moved it to 20.
Clear reranker regression.
The correct result was never in top 1000.
Do not blame the reranker.
Multi-Stage Reranking owns the cascade; evaluation reconstructs where the item was gained or lost.
Permission Evaluation Is Adversarial, Not Only Relevance-Based
A search engine may have perfect relevance metrics while leaking private terms through autocomplete.
Include hostile cases for:
- private result leakage;
- snippets;
- facets;
- counts;
- semantic neighbours;
- revocation lag;
- caches.
Security correctness is a separate release gate.
Subgroup Evaluation Prevents Average-Score Blindness
English queries improve.
Malay queries degrade.
Global average rises because English traffic dominates.
The product has a hidden regression.
Segment metrics by important receiver and corpus groups rather than trusting one aggregate score.
Rare Critical Queries Need Oversampling
Emergency access.
Safety policy.
Legal authority.
These may represent tiny traffic volume but high consequence.
Evaluation sets should include enough of them to detect failure even if ordinary random traffic samples would barely contain any.
Evaluation Sets Drift
New products launch.
Vocabulary changes.
Users move to new intents.
Old queries become obsolete.
A static benchmark can become easier, less representative or actively misleading over time.
Refresh Evaluation From Real Failure Traffic
Add examples from:
- zero-result logs;
- frequent reformulations;
- support tickets;
- bad-click reports;
- failed tasks;
- permission incidents;
- new domains/languages.
The benchmark should learn from the world without becoming contaminated by only the easiest known fixes.
Regression Suites Protect Known Wins
Once a serious search failure is fixed, preserve it as a test case.
A new model should not reintroduce:
- wrong entity sense;
- lost identifier;
- stale canonical page;
- private autocomplete leak;
- known zero-result query.
Search quality improves cumulatively when fixes become permanent constraints.
World Return Is the Final Evaluation
Metric says search is excellent.
User opens the result.
It is the wrong current policy.
The metric suite missed the decision that mattered.
The deepest evaluation question is:
Did the receiver reach evidence that successfully resolved the real-world need?
A Better Search-Evaluation Model
task definition → representative queries → versioned relevance judgments → stage metrics → offline ranking metrics → latency/cost/security guardrails → controlled online test → task completion/reformulation → failure attribution → benchmark refresh → regression lock
A 30-Lens Search Evaluation Audit
- Receiver: who uses search?
- Task: what must search help accomplish?
- Query set: is traffic represented?
- Long tail: are rare queries covered?
- Judgments: who labels relevance?
- Guidelines: is relevance defined consistently?
- Grading: binary or multi-level?
- Freshness: do labels have time context?
- Authority: is source ownership separately judged?
- Permissions: are inaccessible documents excluded from relevance?
- Candidate recall: is relevant material in top-K?
- Precision: how much retrieved material is relevant?
- Recall: how much relevant material was found?
- MRR: how early is the first useful result?
- nDCG: is graded relevance ordered well?
- Interpretation: are query-understanding errors measured?
- Reranking: can gains/losses be attributed to the cascade?
- Latency: p50/p95/p99?
- Timeout: how does fallback affect quality?
- Cost: what resources buy the gain?
- Clicks: are position/presentation biases understood?
- Dwell: is duration interpreted cautiously?
- Reformulation: did users have to repair the search?
- Abandonment: successful answer or failed search?
- Task success: did the user complete the job?
- A/B design: is causal comparison valid?
- Guardrails: security, errors and subgroup performance?
- Subgroups: what does the average hide?
- Drift: is the benchmark still representative?
- World return: did ranked output resolve the actual need with current authoritative evidence?
Laboratory 1: MRR vs nDCG
Create two rankings: one puts the first relevant result at rank 1 but orders later graded results poorly; another puts the first relevant result at rank 2 but orders several highly relevant documents well. Compare what MRR and nDCG reward.
Laboratory 2: Candidate or Ranker?
For ten failed queries, record whether the judged best document entered first-stage top 100 and where the reranker placed it. Attribute failures before changing the model.
Laboratory 3: Click Trap
Design an experiment where a sensational but weak result gets more clicks than a precise answer because its title is more attractive. List the additional metrics needed before calling it better search.
For Primary Readers
If you ask a helper to find the right book, do not judge them only by how many books they bring. Ask whether the right book is there and how quickly you can find it in the pile.
For Secondary Readers
Distinguish candidate recall, precision, reciprocal rank and graded ranking quality, then connect each metric to a different search failure.
For Advanced Readers
Model search evaluation as multi-level validation of a staged retrieval policy under imperfect relevance judgments and behavioural observation. Offline metrics estimate retrieval/ranking quality under a test distribution; online experiments estimate product effects under live traffic; neither substitutes for authority, security or real task-resolution checks.
Common Misconceptions
- “More clicks mean better search.” Position, presentation and user confusion can increase clicks.
- “One relevance metric is enough.” Candidate recall, ordering, latency, cost and task success expose different failures.
- “A high offline score guarantees online improvement.” Benchmark distribution and live user behaviour can differ.
- “Average quality protects everyone.” Important languages, query types or high-consequence tasks can regress underneath a rising global mean.
- “A stable benchmark remains valid forever.” queries, corpus, products and user needs drift.
Research Corridor
- Stanford — Introduction to Information Retrieval — precision, recall, ranked retrieval and evaluation foundations.
- Elastic — Semantic Reranking — current reranking architecture and score use.
- Elastic — Retrievers — current staged retrieval and compound-ranker framework.
- eduKateSG — How Search Works.
Frequently Asked Questions
What is search evaluation?
It is the process of measuring whether retrieval and ranking help users find the right eligible information efficiently, using relevance judgments, stage metrics, online behaviour and real task outcomes.
What is the difference between MRR and nDCG?
MRR emphasises the rank of the first relevant result, which suits lookup-like tasks. nDCG evaluates graded relevance throughout a ranked list while discounting lower positions.
Why are clicks not enough?
Because clicks are affected by rank position, presentation, curiosity and confusion. They should be combined with other behavioural, relevance and task-completion evidence.
Final Thought: The Metric Is Useful Only If It Protects the Human Job
Search can win a benchmark and lose the user.
The evaluation system is doing its job when a higher score means the receiver is more likely to reach the right eligible evidence, sooner, with fewer hidden failures and a reliable route to correction.
SEARCH · FOUR PILLAR LEGS
Return to How Search Works, or continue through Query Understanding, Candidate Generation and Multi-Stage Reranking. Return to the Information & Representation Hub.