A system has one billion possible answers.
The best model can examine only one thousand carefully.
So the system asks cheaper questions first.
Then better questions later.
That is coarse-to-fine retrieval.
Quick Read
Coarse-to-fine retrieval is a multi-stage search architecture in which inexpensive, lower-fidelity mechanisms rapidly remove most of a corpus before progressively more accurate and expensive models examine the surviving candidates. The stages can include filters, lexical retrieval, vector ANN search, lightweight scoring, reranking and final verification.
One-sentence answer: coarse-to-fine retrieval is lossy because early stages discard most candidates using cheaper representations and weaker models, betting that the correct item will survive long enough to reach the expensive stages.
Why Not Use the Best Model on Everything?
Because the best model is often too expensive.
A cross-encoder can compare a query and document richly but may be far too slow to run across millions of documents.
A vector index can retrieve thousands quickly but with less precise query-document interaction.
The architecture therefore spends computational precision gradually.
Each Stage Has a Different Resolution
The first stage may ask only whether terms match or vectors are nearby.
The next may examine richer metadata.
The final stage may read the entire query and candidate together with a large learned model.
Resolution rises as candidate count falls.
Early Errors Are Irreversible
If the right candidate is removed in stage one, stages two through five cannot recover it.
This is the central asymmetry of a cascade.
Later intelligence cannot repair missing evidence.
Candidate Generation Is One Stage; Coarse-to-Fine Is the Whole Architecture
Candidate Generation owns the first major shortlist.
Coarse-to-Fine Retrieval owns the staged allocation of representational and computational fidelity across the entire retrieval path.
The distinction prevents collision: one is a stage; the other is the cascade design.
Recall Must Stay High Early
Early stages should usually favour recall over precision.
It is acceptable to admit many false candidates if downstream stages can reject them.
It is much more costly to exclude a true candidate that can never return.
Later Stages Can Spend Precision
Once the candidate set is small, richer models can examine pairwise interactions, context, instructions and nuanced relevance.
The system effectively converts saved breadth into deeper reasoning.
Coarse-to-fine architecture is therefore a resource exchange between how many things are considered and how carefully each surviving thing is considered.
Stage Budgets Form a Funnel
One billion → one million → ten thousand → one thousand → one hundred → ten.
Every narrowing step should have a reason.
The funnel should not merely be aesthetically neat. Its widths should reflect measured recall, latency and downstream compute.
Hybrid Retrieval Protects the Funnel
A lexical route can recover exact names, numbers and rare terms.
A vector route can recover paraphrases and semantic neighbours.
A graph or LSH route can make vector search fast.
Combining routes before aggressive narrowing can preserve more kinds of relevance.
Reranking Is Not Retrieval
Retrieval decides which items remain eligible.
Reranking spends more compute to order those survivors.
A reranker can transform quality inside the candidate set.
It cannot resurrect excluded items.
A Cascade Can Hide Where the Failure Happened
A final answer is wrong.
Was the correct item absent from the first retrieval?
Was it present but dropped by a second-stage filter?
Was it ranked poorly later?
Without stage-by-stage receipts, all failures can look like one generic “search error”.
Latency Should Be Allocated by Marginal Value
Adding 5 ms to the first stage may improve recall enough to save the whole system.
Adding the same 5 ms to an already strong final reranker may barely change output.
A mature cascade allocates compute where the next unit of latency produces the largest improvement in end-to-end quality.
Coarse-to-Fine and IVF Share the Same Shape at Different Scales
IVF first selects coarse cells and then scans vectors inside them.
Coarse-to-fine retrieval generalises that pattern across multiple heterogeneous stages.
One is an indexing mechanism.
The other is an architectural principle.
High-Stakes Search Needs a Wider Funnel
If missing a relevant result is expensive—medical evidence, legal precedent, safety analysis—the first stages should be more conservative.
Efficiency still matters.
But recall requirements rise because early exclusion carries higher consequence.
Education: Good Thinking Often Goes Coarse to Fine
A student first identifies the broad topic.
Then retrieves several plausible methods.
Then compares the exact conditions and commits to one.
This is an analogy, not retrieval engineering. The common lesson is that early stages should narrow enough to make reasoning possible without narrowing so hard that the correct route disappears.
A CivDJ View: Preserve Receipts at Every Funnel
Forward: does the final system meet relevance and latency targets?
Backward: at which stage did every missed relevant item disappear?
Rotate: do rare queries, languages, new items and minority topics survive each stage at comparable rates?
The cascade becomes trustworthy when every loss has an owner and every exclusion can be traced.
When Coarse-to-Fine Retrieval Is Exactly the Right Loss
Use it when the corpus is enormous, high-fidelity models are expensive, early stages can preserve high recall, and stage-by-stage evaluation shows that computational savings do not create unacceptable blind spots.
When Coarse-to-Fine Retrieval Becomes Dangerous
- Early stages optimise precision instead of preserving recall.
- Stage budgets are chosen independently rather than end-to-end.
- Hybrid routes are collapsed into one brittle retriever.
- Failures are measured only at the final output.
- Rare items disappear early and never receive expensive scoring.
- Latency cuts are made without tracing which stage loses the correct answer.
A Practical Coarse-to-Fine Audit
- Stages: what does each stage know and discard?
- Widths: how many candidates survive each transition?
- Recall: where do relevant items first disappear?
- Cost: how much latency does each stage consume?
- Routes: are multiple retrieval geometries represented?
- Tails: which query families need wider funnels?
- Receipts: can a failure be traced stage by stage?
Continue Through eduKateSG
Continue with How Lossy Works | Candidate Generation, How Lossy Works | Inverted File Indexing, and How Lossy Works | Graph Search Budgets.