A search system owns one hundred million documents.
The expensive reranker can inspect only two hundred.
Which two hundred get invited into the room?
Candidate generation is the first retrieval gate that turns a huge searchable collection into a small enough set of plausible results for richer ranking.
This is the second pillar beneath How Search Works. The master owns the whole query-to-ranked-result mechanism. This article owns the first-stage recall problem: how lexical, semantic and structured retrievers gather plausible documents quickly enough that later ranking has something worth judging.
Quick Read
Candidate generation aims for high recall under strict latency and cost limits. A lexical retriever can use inverted-index evidence such as BM25. A vector retriever can retrieve nearest neighbours in an embedding space. Structured filters can constrain eligibility. Hybrid search can run several retrievers and combine their result sets, for example with Reciprocal Rank Fusion. But every top-K cut creates a recall ceiling: if the relevant document ranks 501st in a retriever that returns only 500, later stages cannot recover it. Good candidate generation therefore balances breadth, diversity, eligibility, freshness, latency and the downstream reranker’s capacity.
interpreted query → parallel first-stage retrievers → eligibility filters → top-K per branch → union/fusion → deduplicate → candidate pool → richer reranking
Candidate Generation Is a Recall Contract
The first-stage system does not need to put the perfect answer at rank 1.
It does need to put the perfect answer somewhere inside the pool passed downstream.
This creates the central first-stage question:
Did every answer worth serious consideration survive the first cut?
The Candidate Pool Sets a Hard Ceiling
Candidate set contains documents A–Z.
The truly best answer is document Q.
Excellent.
If Q is missing, no downstream scoring model can rank it.
reranking quality ≤ candidate recall.
Lexical Retrieval Is a Strong First Stage
Names.
Identifiers.
Rare phrases.
Technical terms.
Exact wording.
These are where an inverted index shines.
Inverted Index & Postings owns the underlying data structure.
BM25 Is Often a Candidate Generator, Not the Final Judge
A lexical scoring model can produce a strong first ordering using term frequency, rarity and document-length effects.
That list may then be reranked using:
- semantic models;
- freshness;
- authority;
- business rules;
- receiver context;
- expensive query-document interaction.
First-stage ranking is therefore often an efficient approximation to “worth examining further.”
Semantic Retrieval Opens a Different Door
User asks:
plants make food from sunlight
A lesson titled Photosynthesis may share few exact words.
Vector retrieval can recover candidates that are conceptually close despite vocabulary mismatch.
Vector Retrieval Has Its Own Recall Approximation
At large scale, vector systems often use approximate nearest-neighbour structures rather than comparing the query with every vector exactly.
Approximation reduces latency and cost.
It can also miss a true nearest neighbour.
Candidate recall therefore depends on vector-index configuration as well as embedding quality.
Top-K Is a Budget
Retrieve top 10.
Fast and narrow.
Retrieve top 10,000.
High potential recall and expensive downstream work.
The right K depends on:
- collection size;
- query difficulty;
- first-stage quality;
- reranker capacity;
- latency budget;
- required recall;
- number of retrieval branches.
K Is Not the Same Across Query Types
Exact invoice lookup may need one or a few candidates.
Open research question may need hundreds or thousands.
Ambiguous query may need more breadth because several senses must remain alive.
One fixed K can under-retrieve hard queries and waste computation on easy ones.
Hard Filters Can Improve Candidate Quality
User asks for papers after 2024.
Filtering before expensive retrieval avoids spending candidate slots on ineligible older documents.
But only if the filter is truly hard.
A Wrong Hard Filter Destroys Recall Completely
User says “recent” and the system interprets it as exactly 30 days.
The authoritative answer is 45 days old.
It never enters the candidate pool.
Query Understanding owns deciding whether a constraint is hard, soft or ambiguous before candidate retrieval executes it.
Permissions Are Hard Eligibility, Not a Late Cosmetic Filter
A private document can be a perfect semantic match and still be ineligible.
Candidate generation should operate within the authorised search space or enforce permission filtering before any lower-trust downstream stage sees restricted content.
See Permission-Aware Indexing.
Hybrid Retrieval Protects Against Different Blind Spots
Lexical branch:
- excellent exact names;
- strong technical terms;
- good identifiers;
- weak vocabulary mismatch.
Semantic branch:
- good conceptual similarity;
- good paraphrases;
- weaker exact distinctions;
- model-dependent neighbourhoods.
Run both and the union can cover failure modes neither solves alone.
Candidate Union Increases Recall—and Pool Size
Lexical top 200.
Vector top 200.
After deduplication, perhaps 330 unique candidates remain.
The reranker must now handle 330 items.
Hybrid breadth is useful only when downstream capacity exists.
Reciprocal Rank Fusion Combines Ranks Without Requiring Comparable Scores
Current Elasticsearch documentation describes Reciprocal Rank Fusion (RRF) as a way to combine multiple result sets whose relevance indicators need not share the same score scale.
Each candidate receives contribution based on its rank inside each child result list.
This is useful because BM25 and vector similarity can have fundamentally different numeric scales.
Fusion Is Still a Candidate Decision
RRF’s rank_window_size or equivalent per-branch windows determine which child results are even considered.
Increase the window and potential recall rises at a performance cost.
Candidate-generation hyperparameters are therefore not merely performance tuning; they set evidence availability downstream.
Score Fusion Requires Calibration or Normalisation
Lexical score 14.2.
Vector cosine 0.81.
Adding raw numbers is meaningless without a model for making their scales comparable.
Rank fusion avoids part of this problem; linear score combination requires explicit normalization and weighting.
Structured Retrievers Can Supply Exact Candidates
Known ISBN.
Known invoice.
Known primary key.
Do not force every query through fuzzy full-text/vector retrieval when a structured lookup can produce the exact candidate directly.
Pinned or Rule-Based Candidates Can Be Legitimate
A query for a safety policy may need the current official policy pinned above commentary.
A navigation query for a known product may route directly to the official product page.
Current search frameworks expose rule/pinned retriever concepts precisely because retrieval can incorporate governed business or authority knowledge.
Rules should remain visible and versioned so editorial intervention is not mistaken for organic relevance evidence.
Candidate Diversity Protects Against Redundant Pools
Top 100 candidates.
Ninety are near-identical mirrors of the same article.
The pool has size 100 and effective evidence diversity near 11.
Canonicalisation or deduplication before expensive reranking can free candidate budget for distinct evidence.
Duplicate Removal Has an Epistemic Benefit
Ten syndicated copies should not look like ten independent candidate sources.
Candidate generation should preserve:
- canonical owner;
- copy/derivation relationship;
- best representative;
- alternate access route where needed.
Freshness Can Influence Candidate Eligibility
Query:
current MRT disruption
A beautifully relevant 2023 article is not useful candidate evidence for today’s status.
For time-sensitive intents, candidate generation may narrow the time window or prioritise fresh indexes/sources before ranking.
Freshness Filters Can Also Hide Necessary History
Query:
why did this policy change?
Historical versions are necessary evidence.
Intent should determine temporal candidate scope rather than a universal “newest only” rule.
Candidate Generation Can Be Multi-Hop
Search finds a company.
The company links to products.
A product links to manuals.
In graph or knowledge systems, the candidate route can expand through bounded relationships rather than one flat lexical/vector search.
Each hop expands recall and cost—and can compound relevance drift.
First-Stage Latency Must Stay Predictable
A fast average with 5-second tail latency creates poor interactive search.
Candidate systems should monitor:
- p50/p95/p99 latency;
- per-retriever time;
- timeout rate;
- candidate count;
- recall at K;
- filter selectivity;
- shard/partition imbalance.
Timeouts Can Quietly Reduce Recall
Lexical branch returns.
Semantic branch times out.
The user still gets results.
The hybrid candidate pool is now missing one retrieval family.
Search should expose partial-retrieval health internally and avoid evaluating such runs as if all candidate sources were present.
Candidate Generation Is Where Search Cost Is Controlled
Expensive cross-encoder over 100 million documents is impossible for an interactive request.
Cheap retrieval reduces the problem to hundreds.
The third pillar, Multi-Stage Reranking, owns how the system spends richer computation on the survivors.
Evaluation Must Measure Recall Before Final Ranking
Final nDCG is poor.
Was the reranker bad?
Or did the right result never enter candidate top-K?
The fourth pillar, Search Evaluation, separates first-stage recall failures from ordering failures.
A Better Candidate-Generation Model
interpreted query → hard eligibility → exact/lexical/semantic/graph retrieval branches → top-K/window per branch → union/fusion → canonical deduplication → candidate diversity/health checks → reranker handoff
A 30-Lens Candidate Generation Audit
- Corpus: what collection can contribute candidates?
- Interpretation: which query representation enters retrieval?
- Eligibility: which hard filters apply?
- Permissions: which content is authorised?
- Lexical: what inverted-index retrieval runs?
- Exact: what known-ID lookup runs?
- Semantic: what vector retrieval runs?
- Approximation: can ANN miss neighbours?
- Graph: are relationship expansions used?
- Branch K: how many candidates per retriever?
- Global K: how many survive to reranking?
- Recall: how often is the judged relevant item present?
- Hard-filter error: what correct answers can be excluded?
- Fusion: union, RRF or score combination?
- Score scaling: are heterogeneous scores comparable?
- Window: which rank cutoff limits fusion?
- Deduplication: are mirrors consuming candidate budget?
- Canonical owner: which source represents each duplicate cluster?
- Diversity: are several plausible interpretations represented?
- Freshness: should time narrow the pool?
- Authority: are governed official candidates pinned/routed when appropriate?
- Latency: what p95/p99 budget applies?
- Timeout: what happens when one retriever fails?
- Shard: can partition imbalance reduce recall?
- Cost: how much work per candidate branch?
- Downstream capacity: how many docs can reranker process?
- Query difficulty: should K adapt?
- Observability: can candidate sources be inspected?
- Failure attribution: can recall failure be separated from rank failure?
- World return: did the candidate set include the current authoritative evidence needed for the actual task?
Laboratory 1: Top-K Ceiling
Create a ranked first-stage list where the best judged result is rank 51. Compare K=50 and K=100. Explain why the same reranker has zero chance in the first configuration.
Laboratory 2: Lexical + Semantic Union
Use one query with an exact rare term and a paraphrased concept. Build lexical and semantic top-10 lists, union them, deduplicate and identify which relevant items each branch uniquely recovered.
Laboratory 3: Duplicate Budget
Create a top-20 pool containing ten mirrors of one source. Canonicalise them into one owner and use the freed nineteen slots to admit new evidence. Compare effective evidence diversity.
For Primary Readers
You have a shelf with 1,000 books but can carry only 20 to the teacher. Candidate generation is choosing the 20 books most worth checking before the teacher reads them carefully.
For Secondary Readers
Explain why first-stage retrieval should maximise useful recall under a candidate budget and why no reranker can choose a missing document.
For Advanced Readers
Model candidate generation as constrained high-recall retrieval over heterogeneous searchable representations. The first stage optimises coverage under latency and downstream-capacity budgets; candidate cutoffs, ANN approximation, filters and fusion windows define the maximum achievable quality of every subsequent ranker.
Common Misconceptions
- “The reranker can fix a weak first stage.” It cannot rank documents that were never retrieved.
- “More K is always better.” Larger pools cost more latency and reranking capacity and can add noise.
- “Semantic retrieval replaces lexical retrieval.” Exact names and identifiers remain strong lexical/structured jobs.
- “Hybrid scores can simply be added.” Different retrieval scores may live on incompatible scales.
- “A pool of 100 means 100 independent candidates.” duplicates can consume most of the budget while adding little evidence diversity.
Research Corridor
- Elastic — Retrievers — current first-stage, compound and reranking retrieval framework.
- Elastic — Reciprocal Rank Fusion — fusion of multiple top-document sets.
- Stanford — Introduction to Information Retrieval — lexical retrieval, recall and ranked-search foundations.
- eduKateSG — How Search Works.
Frequently Asked Questions
What is candidate generation in search?
It is the first-stage retrieval process that selects a manageable high-recall pool of plausible documents from a much larger searchable collection.
Why is candidate recall important?
Because later rerankers can reorder only the candidates they receive. Missing the relevant result in first-stage top-K creates an unrecoverable ceiling for that query.
Why combine lexical and semantic retrieval?
They have complementary strengths: lexical retrieval preserves exact terms and identifiers, while semantic retrieval can recover conceptual matches expressed with different wording.
Final Thought: The Most Important Ranking Decision May Happen Before Ranking
The expensive judge sees only who reached the courtroom.
Candidate generation is trustworthy when the first cut is fast without becoming so narrow that the right evidence is silently denied the chance to compete.
SEARCH · FOUR PILLAR LEGS
Return to How Search Works, or continue through Query Understanding, Multi-Stage Reranking and Search Evaluation. Return to the Information & Representation Hub.