A lightweight search stage can inspect millions of documents.
A powerful cross-encoder cannot.
So search does not ask the most expensive judge to examine everyone.
It narrows first.
Multi-stage reranking is the cascade in which cheap broad retrieval selects plausible candidates and progressively richer models spend more computation on smaller sets where better judgment can still change the final order.
This is the third pillar beneath How Search Works. It is deliberately narrower than the site’s broader How Lossy Works | Ranking, which owns the general transformation from score/distance to order. This article owns the search-specific cascade: retrieval window, fusion, rescoring, semantic reranking, freshness/authority adjustments and final presentation order.
Quick Read
A modern search system often ranks in stages because the best scoring model is too expensive to apply to the whole corpus. First-stage retrievers such as BM25, vector kNN or structured lookup produce top candidates. Compound retrievers can fuse result sets—for example Reciprocal Rank Fusion. A second-stage reranker can then inspect query and document together using a cross-encoder or another rich model. Current Elasticsearch semantic-reranking documentation explicitly describes cross-encoders as more accurate but more computationally intensive than bi-encoder similarity, which is why they are suited to reranking smaller candidate sets. Later stages can also incorporate freshness, authority, permissions, diversity or task-specific signals. The cascade works only if each earlier stage preserves enough recall for later stages to improve the order.
query → broad retrievers → top-K candidates → fusion → cheap score → smaller window → cross-encoder/richer reranker → freshness/authority/diversity constraints → final top-N
Why Not Use the Best Model First?
Suppose a cross-encoder needs to read query + document text jointly.
One hundred million documents.
One interactive query.
Even excellent per-document latency becomes impossible at corpus scale.
The cascade solves this by spending expensive judgment only after cheap retrieval has reduced the search space.
Every Stage Trades Breadth for Judgment
Stage 1 sees millions cheaply.
Stage 2 sees hundreds more carefully.
Stage 3 may inspect dozens with richer source context.
Stage 4 presents perhaps ten.
The architecture is a sequence of shrinking candidate sets and increasing judgment cost.
Candidate Generation Is the Non-Negotiable Input
If the correct result never appears in the first-stage top-K, every later stage can be perfect and still fail.
Candidate Generation owns this recall ceiling.
reranker intelligence cannot exceed candidate availability.
Fusion Can Be an Intermediate Ranking Stage
Lexical retriever produces one order.
Vector retriever produces another.
Structured or rule-based retriever may contribute additional candidates.
A fusion layer combines these branches before the expensive reranker.
RRF Combines Ranks Rather Than Raw Score Scales
Current Elasticsearch documentation describes Reciprocal Rank Fusion as combining two or more child retrievers by their rank positions, without requiring the relevance indicators to use the same numeric scale.
This is useful because:
- BM25 score is not cosine similarity;
- vector score may be query-dependent;
- business or rule-based branches may not have natural comparable scores.
Rank fusion creates a common intermediate ordering without claiming the underlying signals mean the same thing.
Fusion Window Size Is a Quality/Cost Dial
RRF and similar fusion systems operate over a finite window from each child retriever.
Too small and useful branch-specific candidates disappear before fusion.
Too large and latency/memory rise while low-quality tail results flood the next stage.
Rescoring Means “Use a Better Function on Fewer Results”
A first-stage lexical score might depend on term statistics and fields.
A later rescorer might incorporate:
- phrase proximity;
- business value;
- freshness;
- authority;
- query-document semantic interaction;
- receiver-specific features.
The late stage can afford richer evidence because it processes far fewer documents.
Cross-Encoders Read Query and Document Together
Current Elastic semantic-reranking documentation distinguishes cross-encoders from bi-encoders.
A bi-encoder produces query and document representations separately and compares them later.
A cross-encoder processes query and document text jointly, allowing richer interaction between:
- word order;
- negation;
- phrase relationships;
- query-specific context;
- fine semantic distinctions.
That Richness Costs Computation
Elastic’s current documentation describes cross-encoders as more accurate for semantic reranking but more resource-intensive than bi-encoder similarity.
This is exactly why reranking happens late.
late-stage model quality is purchased with early-stage selectivity.
Rerank Window Is Another Irreversible Cut
Candidate pool = 1,000.
Cross-encoder sees top 100 only.
A relevant result at candidate rank 101 cannot benefit from the richer model.
Every cascade boundary creates its own recall ceiling.
Window Size Should Follow Query Difficulty
Exact identifier query:
small rerank window.
Ambiguous research query:
larger window may be justified.
Adaptive cascades can spend more computation on difficult queries rather than charging every query the worst-case cost.
Early Exit Can Save Work
Known invoice ID matches one authorised record exactly.
There may be no reason to invoke an expensive semantic reranker.
A cascade should be able to stop when additional judgment cannot plausibly improve the decision.
Exact Evidence Can Override Fuzzy Relevance
User requests invoice 8472.
A semantically related accounting guide should not outrank the exact invoice.
Reranking should preserve hard identity constraints rather than allowing a learned model to reinterpret lookup as topical similarity.
Freshness Can Be a Late Reranking Signal
Two pages answer the same operational query.
One is three hours old.
One is three years old.
If the user asks for current status, freshness should alter the order.
If the user asks for historical context, the older page may be exactly right.
Freshness is therefore intent-relative evidence, not a universal boost.
Authority Can Be a Separate Signal
Official transport operator notice.
News commentary.
Social repost.
All may be topically relevant.
For current service status, the official operator can deserve priority because it owns the operational fact—not because authority is a universal relevance score for every question.
Canonicalisation Should Happen Before the Final Screen
Five mirrors rank in the top ten.
The user sees apparent agreement that is actually one source copied repeatedly.
Reranking should cluster duplicates or derivations so final ordering rewards independent usefulness rather than copy count.
Diversity Can Be a Deliberate Final-Stage Objective
User asks a broad exploratory question.
Top ten all say the same thing.
A diversity-aware stage can trade a small amount of individual relevance for broader coverage of:
- subtopics;
- perspectives;
- source types;
- time periods;
- product variants.
Diversity is useful when the task is exploration; it can be harmful for exact lookup.
Personalisation Belongs Late Unless It Changes Eligibility
Shared evidence should remain available.
Receiver context can reorder among eligible candidates according to:
- language;
- reading level;
- location;
- known preferences;
- current task.
But personalisation should not turn receiver preference into a substitute for source authority or factual correctness.
Permissions Must Precede Lower-Trust Rerankers
An external model reranks candidate documents.
If restricted documents are removed only afterward, protected text has already crossed the boundary.
Permission-Aware Indexing owns the wider information-flow rule.
Reranking Scores Are Not Truth Probabilities
A cross-encoder score may be more comparable across queries than raw embedding similarity under some models, but it still represents model-estimated relevance under its training/objective.
The master keeps the governing distinction:
ranking score ≠ truth probability.
Calibration Matters When Scores Become Thresholds
A system says:
Only send candidates to the LLM if reranker score > T.
Now score stability and calibration matter operationally.
A threshold learned on one domain may reject good candidates or admit poor ones after corpus/query drift.
Search Reranking Is Not Evidence Evaluation
A page can be highly relevant to the query and factually wrong.
Reranking says “this looks useful for the request.”
How Evidence Works owns “how strongly should this claim be believed?”
These jobs may exchange signals but should not collapse into one score.
Search Reranking Is Also Not the General Theory of Ranking
The broader Lossy | Ranking article owns how multidimensional difference becomes an order and what distance information is lost.
This article owns a specific operational architecture:
cheap first-stage retrieval → progressively more expensive query-specific judgement.
Cascades Need Stage-Level Observability
Final result is wrong.
Which stage failed?
- query interpretation;
- candidate recall;
- fusion;
- reranker;
- freshness boost;
- authority policy;
- deduplication;
- presentation?
Log candidate ranks and stage transitions so failure can be attributed rather than blamed vaguely on “search.”
A/B Testing Only the Final Ranker Can Mislead
Reranker B improves top-10 metrics.
But candidate generator changed simultaneously.
You cannot know which stage caused the gain.
Search experiments should isolate changes or use factorial/controlled designs where multiple stages change.
Latency Budgets Should Be Allocated by Marginal Value
Stage 1 costs 20 ms and recovers 98% of relevant documents.
Stage 2 costs 30 ms and improves top-10 order substantially.
Stage 3 costs 500 ms and barely changes results.
The last stage is economically weak even if it is mathematically sophisticated.
spend computation where it changes the decision.
Tail Latency Matters More Than Average
Average rerank time: 80 ms.
p99: 2 seconds.
Interactive users experience the tail.
Monitor stage-specific p95/p99 and timeouts, not only mean latency.
Timeouts Need a Degradation Policy
Cross-encoder times out.
Options:
- return fused first-stage order;
- fall back to lexical ranking;
- return partial rerank;
- fail the query.
The right fallback depends on consequence and product expectations.
Fallback state should be observable in evaluation logs.
Rerankers Drift When Queries or Corpus Change
A reranker trained on product search begins serving scientific literature.
Its relevance judgments may no longer transfer.
Evaluation must segment by query/domain and monitor changes over time rather than assuming one benchmark remains representative forever.
Search Evaluation Owns the Final Test
Did the richer cascade actually improve user tasks?
The fourth pillar, Search Evaluation, owns offline judgments, ranking metrics, online experiments and task success.
A Better Multi-Stage Reranking Model
eligible high-recall candidates → branch fusion → cheap first order → rerank window → rich query-document interaction → freshness/authority/diversity/business constraints → final order → source verification → evaluation feedback
A 30-Lens Search Reranking Audit
- Need: what search task is being ranked?
- Candidate source: which retrievers feed the cascade?
- Candidate recall: is the right answer present?
- Fusion: how are branches combined?
- Fusion window: which child ranks survive?
- First score: BM25, vector, rule or hybrid?
- Rerank window: how many candidates get expensive judgment?
- Cross-encoder: is query/document interaction used?
- Model version: which reranker produced the score?
- Domain: was it trained/evaluated on this query family?
- Exact identity: can fuzzy ranking override exact lookup improperly?
- Freshness: is time relevant to this intent?
- Authority: does official ownership matter?
- Permissions: were candidates authorised before reranking?
- Deduplication: do copies occupy final slots?
- Diversity: is exploration breadth required?
- Personalisation: what receiver-specific ordering is applied?
- Score meaning: is relevance being confused with truth?
- Calibration: are thresholds stable enough for use?
- Stage attribution: can an error be located?
- Latency: what does each stage cost?
- Tail latency: what is p95/p99?
- Timeout: what fallback occurs?
- Early exit: can exact/easy queries skip expensive stages?
- Adaptive budget: do hard queries receive more work?
- Cost: does the stage improve enough to justify resources?
- Drift: has corpus/query distribution changed?
- Evaluation: which offline/online metric moved?
- Source return: can the final result reach authoritative evidence?
- Receiver success: did the final order improve the real task?
Laboratory 1: Two-Stage Search
Create 50 candidate titles with cheap lexical scores. Choose the top 10 and apply a slower manual query-document relevance judgment. Compare final order with first-stage order.
Laboratory 2: Rerank Window Failure
Place the truly best result at candidate rank 21. Compare rerank windows of 20 and 50. Show why the same expensive model succeeds only in one configuration.
Laboratory 3: Cost Curve
Estimate quality gain and latency for reranking 20, 100 and 500 candidates. Identify the point where additional computation produces little useful movement in the final top 10.
For Primary Readers
First, a helper quickly chooses 20 books from a huge library. Then a teacher carefully reads those 20 and places the best five on top. The teacher is slower but sees far fewer books.
For Secondary Readers
Explain why search uses a cascade: broad cheap retrieval first, then richer scoring on fewer documents. Identify the recall loss at each window boundary.
For Advanced Readers
Model multi-stage reranking as sequential resource allocation over a shrinking candidate set. Each stage optimises expected ranking utility conditional on surviving candidates, while window sizes define irreversible recall constraints and compute budgets determine the marginal depth of query-document interaction.
Common Misconceptions
- “Use the best model on every document.” The computational cost makes this impractical at large scale.
- “A better reranker fixes candidate recall.” It cannot rank absent documents.
- “Cross-encoder score is truth probability.” It is a relevance-model output under a specific objective and domain.
- “Freshness always deserves a boost.” Historical queries can prefer older evidence.
- “More stages always improve search.” Additional stages can add latency, cost and new failure modes without meaningful task gain.
Research Corridor
- Elastic — Semantic Reranking — current cross-encoder reranking concepts and trade-offs.
- Elastic — Retrievers — current standard, RRF, rescorer and text-similarity reranker framework.
- Elastic — Reciprocal Rank Fusion.
- eduKateSG — How Search Works.
Frequently Asked Questions
What is reranking in search?
Reranking means taking an already retrieved candidate set and applying a different, often richer scoring model to reorder some or all of those candidates.
Why use cross-encoders for reranking?
Because processing query and document together allows richer query-specific interaction than comparing separately encoded embeddings, although the extra computation makes cross-encoders better suited to smaller late-stage candidate sets.
How large should a rerank window be?
Large enough that relevant candidates usually survive into the richer stage, but small enough to meet latency/cost limits. The optimum depends on first-stage recall, query difficulty and reranker capacity.
Final Thought: Expensive Judgment Is Valuable Only After Cheap Search Has Protected the Right Possibilities
The cascade works by knowing when not to think deeply yet.
Multi-stage reranking is intelligent when every extra unit of computation is spent on candidates that still have a realistic chance to change the user’s decision.
SEARCH · FOUR PILLAR LEGS
Return to How Search Works, or continue through Query Understanding, Candidate Generation and Search Evaluation. Return to the Information & Representation Hub.