Cross-encoder reranking is one of the most important ideas in modern search engines, semantic search, information retrieval, vector search and retrieval-augmented generation (RAG): retrieve a manageable candidate set quickly, then spend a much stronger relevance model on that shortlist. The architecture exists because the most accurate query-document judgment is usually too expensive to apply to every document in a large corpus. So modern retrieval systems separate candidate generation, dense retrieval, hybrid search, ranking and reranking into stages, each with a different computational budget.
The central problem is also a problem of lossy information retrieval. A cross-encoder can jointly read the query and a candidate document, model token-level interactions, understand negation and qualifiers, and make a richer relevance judgment than a single-vector embedding comparison. But it can judge only the candidates that survive first-stage retrieval. If BM25, dense retrieval, approximate nearest-neighbour search, HNSW, IVF, hybrid retrieval or another candidate generator removes the correct evidence before reranking, the cross-encoder never sees it. Later intelligence cannot recover evidence that an earlier stage has already discarded.
This article explains how cross-encoder reranking works, why reranking improves search relevance, how retrieval recall limits reranking, why top-k candidate size matters, how cross-encoders differ from bi-encoders and late-interaction retrieval, how RAG pipelines lose evidence, and how latency, relevance and retrieval accuracy trade against one another. The governing idea is simple: search becomes affordable by making coarse decisions early and expensive judgments late—but every early narrowing step is a bet about what the future will not need.
The Short Answer: What Is Cross-Encoder Reranking?
A cross-encoder reranker receives a query and one candidate document together. Instead of producing one independent vector for the query and another independent vector for the document, it allows the two texts to interact inside the model before a relevance score is produced.
That joint processing is expensive and powerful.
It is expensive because every query-document pair must be processed separately. If a corpus contains ten million documents, scoring all ten million pairs for every query is usually impractical.
It is powerful because the model can notice relationships that independent embeddings can blur: whether a qualifier applies to the right noun, whether a sentence negates the proposition in the query, whether the requested date or jurisdiction matches, whether an apparent semantic neighbour actually answers the question, and whether a document contains the exact evidence rather than merely discussing the same topic.
So a retrieval pipeline usually performs cheaper search first, perhaps retrieving 20, 100, 500 or 1,000 candidates, and then asks the cross-encoder to reorder only those survivors.
One-sentence answer: cross-encoder reranking is lossy at the system level because its superior relevance judgment is applied only after an earlier stage has irreversibly reduced the corpus to a shortlist.
Why This Article Exists in the How Lossy Works Library
At first glance, reranking looks like an accuracy technique rather than a lossy technique.
The reranker itself can improve relevance. It often restores distinctions that a coarse embedding or lexical score missed.
But the architecture surrounding it is a funnel.
The full corpus becomes a candidate set. The candidate set becomes a reranked list. The reranked list becomes a handful of passages. Those passages may become the context supplied to a generator or the results shown to a user.
At every transition, possibilities disappear.
The interesting question is therefore not simply whether the reranker is accurate.
What evidence remained eligible to receive accurate judgment?
That is a quintessentially lossy question.
The Retrieval Funnel
Imagine a corpus containing 50 million passages.
A query arrives.
A practical system might perform something like this:
- 50,000,000 passages exist.
- A lexical or vector index rapidly identifies 2,000 plausible candidates.
- A stronger retrieval stage reduces those to 200.
- A cross-encoder reranks the 200.
- The top 20 receive deeper processing.
- The top 5 enter a RAG context window.
- The generator answers from those 5.
The final answer may look like one act of intelligence.
It is actually the endpoint of several irreversible selection operations.
Each stage asks a cheaper question than the next.
Each stage also controls what the next stage is allowed to know.
Candidate Generation Is Not Reranking
This distinction is foundational.
Candidate generation asks:
Which tiny fraction of the corpus deserves expensive attention?
Reranking asks:
Given the candidates that survived, which are actually most relevant?
The two jobs must remain separate in diagnosis.
If the correct document never enters the candidate set, improving the reranker cannot fix the failure.
If the correct document is present but ranked poorly, the problem may genuinely belong to the reranker.
Many retrieval systems become difficult to improve because these two errors are collapsed into one metric called “search quality”.
Bi-Encoders: Fast Because Query and Document Separate Early
A bi-encoder typically encodes the query and document independently.
The document representation can be computed before any user asks a question. Millions of document vectors can therefore be stored in an index.
When a query arrives, only the query needs fresh encoding. The system searches for nearby document vectors.
This is extraordinarily scalable.
The cost is early commitment.
The document must decide how to represent itself before it knows the future query.
That is the central compression problem explored in How Lossy Works | Single-Vector Embeddings.
Cross-Encoders: Expensive Because Query and Document Meet Early
A cross-encoder delays the relevance decision until query and document can interact richly.
Imagine the query asks:
Which policy applies to students who transferred after the registration deadline?
A semantically similar document about student registration may retrieve well under a dense embedding.
But the actual answer may depend on the interaction between transferred, after, and deadline.
A cross-encoder can process those query terms jointly with the candidate text and judge whether the relevant exception really exists.
The price is that the document cannot be reduced to one reusable query-independent relevance score.
Why Cross-Encoders Are Usually Rerankers
If the corpus contains 10 million documents and every query requires 10 million expensive forward passes, search latency becomes unacceptable.
So cross-encoders typically arrive after a cheaper first stage.
This division of labour is not a weakness.
It is a rational allocation of computation:
- cheap models preserve breadth;
- expensive models provide depth;
- the candidate budget decides where breadth becomes depth.
The whole architecture is a rate–distortion bargain over search.
Late Interaction Sits Between Bi-Encoders and Cross-Encoders
Late-interaction systems such as ColBERT keep multiple token-level vectors and delay part of the query-document interaction until scoring.
They preserve more local evidence than one-vector bi-encoders while retaining more precomputation than a full cross-encoder.
That middle position matters.
Retrieval architectures are not simply “fast” or “accurate”.
They choose when query-dependent interaction occurs.
The earlier the system commits to a compressed query-independent representation, the cheaper broad search becomes.
The later it commits, the more local evidence can survive.
Read How Lossy Works | Multi-Vector Retrieval for that middle architecture.
A Three-Way Mental Model
Think of three architectures as three points on a continuum.
| Architecture | Document representation | Query-document interaction | Typical strength | Typical cost |
|---|---|---|---|---|
| Single-vector bi-encoder | One vector | Very late / simple similarity | Scalable broad retrieval | Lower local fidelity |
| Multi-vector late interaction | Many local vectors | Late token-level matching | Fine-grained retrieval | Larger index and search cost |
| Cross-encoder | Joint pair representation | Inside the model | Rich relevance judgment | High per-pair compute |
None is universally superior.
The correct architecture depends on corpus scale, latency, hardware, relevance complexity and the cost of missing evidence.
The Most Important Equation Is Not a Neural Equation
For a reranking system, end-to-end success is constrained by candidate recall.
If a first-stage retriever places the relevant document somewhere inside the top 100 candidates 92% of the time, no reranker operating only on those 100 can achieve successful retrieval on more than that 92% through reordering alone.
The reranker may improve precision dramatically inside the 92%.
It cannot touch the missing 8%.
This simple ceiling is often forgotten because reranking models are more glamorous than candidate-recall audits.
Recall@k Is the Door Metric
Suppose one relevant document exists for each query.
Recall@100 asks whether that document appears somewhere in the first 100 candidates.
At this stage, position 1 and position 100 are both survivals.
The reranker can still work with either.
If the document is at position 101 and only 100 candidates are passed onward, it is functionally absent.
A one-position difference becomes an epistemic cliff.
Candidate Budget Is a Loss Budget
Increase the candidate count and first-stage recall usually improves.
The reranker must then score more pairs.