Search feels instantaneous partly because many systems refuse to recompute the same expensive work every time.
Retrieval caching stores reusable outputs from earlier search work — query results, candidate sets, feature computations, embeddings, document representations or other intermediate states — so later requests can reuse them instead of repeating the entire pipeline.
The gain is speed and lower computational cost. The risk is staleness. A cache can return yesterday’s result after the underlying index, permissions, prices, availability or document state has changed.
This article sits beneath How Search Works, How Indexing Works and the broader latency owner How The World Works | Latency. The narrow question is: which search work can be safely reused, for how long, and what must invalidate it?
1. A Cache Trades Computation for Stored State
Without caching, repeated identical or similar queries can force the system to repeat tokenisation, retrieval, feature calculation, ranking and rendering work.
A cache stores a prior result under a key. When the same key returns, the system can reuse the stored state if it is still valid.
2. Query-Result Caches Are the Most Visible Form
A frequent query can be mapped directly to a ranked result set. This is efficient when the query is common and the underlying corpus changes slowly.
It becomes risky when freshness matters. Breaking news, inventory, prices and live availability can invalidate a cached result quickly.
3. Candidate Caches Reuse Earlier Retrieval Without Freezing the Final Ranking
A system can cache the broad candidate set while allowing later-stage ranking to incorporate fresh user context, recency or policy.
This provides a middle ground: expensive first-stage retrieval is reused while final judgment remains more current.
4. Representation Caches Reuse Embeddings and Features
Document embeddings, query embeddings and ranking features can be expensive to compute. If the source document has not changed, the representation can often be reused.
The cache key should therefore include enough version state to distinguish a current representation from one generated from an old document, old model or old processing pipeline.
5. Time-to-Live Is the Simplest Freshness Rule
A cached item can be declared valid only for a fixed period. After the time-to-live expires, the next request recomputes the result.
This is easy to operate but imperfect. A result can become stale one second after being cached, or remain perfectly valid long after the timer expires.
6. Event-Based Invalidation Is More Precise
If the underlying index changes, an event can invalidate dependent cache entries immediately.
The challenge is dependency tracking. A changed document may affect thousands of queries whose cached rankings included it, or whose rankings would now include it.
Precise invalidation can therefore become expensive enough that systems combine event triggers with shorter time windows and lazy refresh.
7. Stale-While-Revalidate Protects Latency
Some systems serve a slightly stale result immediately while recomputing a fresh version in the background for later requests.
This is reasonable only when brief staleness has low consequence. It is inappropriate when permissions, safety, financial state or legal status changed and an old result would be harmful.
8. Permissions Must Be Part of the Cache Key
A cached result generated for one user cannot automatically be reused for another if their access rights differ.
Permission-aware retrieval requires the cache to preserve the security boundary. Otherwise caching can become an information-leak path.
The specialist owner remains Permission-Aware Indexing.
9. Personalisation Complicates Reuse
If ranking depends on location, language, history, device, age restrictions or account state, one global cached result can become wrong for many receivers.
Systems can cache common intermediate work and apply personalisation later, or partition caches by the context dimensions that materially change the result.
10. Cache Stampedes Are a Failure Mode
When a popular cache entry expires, thousands of simultaneous requests can all try to recompute it at once.
Locks, request coalescing, jittered expiry and background refresh can prevent one expired entry from creating a sudden load spike.
11. Worked Example: eduKateSG Site Search
A frequent search such as “Primary 6 vocabulary” may be expensive if the system repeatedly retrieves and reranks a large estate.
A cache can reuse the broad candidate set while still allowing newly published articles, user-specific context and canonical owner rules to affect the final ranking. The cache accelerates the path without becoming the authority for what is current.
12. Worked Example: Live Availability
A restaurant search can cache static information such as name, cuisine and neighbourhood for a long time. Real-time table availability should have a much shorter validity window or bypass the stale cache entirely.
The correct cache boundary follows the rate at which the underlying fact changes and the consequence of serving stale state.
13. A Retrieval-Caching Checklist
- Identify expensive retrieval stages worth reusing.
- Choose cache keys that include relevant query, model, index, permission and context state.
- Set freshness rules according to source volatility.
- Use event invalidation where consequence justifies it.
- Allow stale-while-revalidate only where brief staleness is acceptable.
- Protect permission boundaries.
- Avoid over-partitioning until cache reuse disappears.
- Prevent stampedes at expiry.
- Measure hit rate, latency, staleness and receiver task success together.
14. Read the Mechanism Forward, Backward and Sideways
Forward: expensive search work → cache entry → repeated request → validation → reuse or refresh. Backward: start from a stale answer and trace which invalidation boundary failed to notice the source change. Sideways: compare performance engineer, content owner, security owner and user. The fastest cached answer is not useful if it is stale, unauthorised or routed to the wrong owner.
Retrieval caching works when yesterday’s computation can be reused without pretending yesterday’s world is still current.
Continue through How Hybrid Retrieval Works, Incremental Indexing and Freshness and the How X Works hub. Next: result diversification — why a useful results page needs coverage of distinct plausible intents, not ten versions of the first answer.