A search index receives one new document.
Should it rewrite the entire index?
Now another document arrives.
Rewrite everything again?
At scale, that would be absurd.
Segment-based indexing keeps writes cheap by creating new immutable searchable pieces, then pays the organisational debt later by merging them into fewer larger segments.
This is the second pillar beneath How Indexing Works. The master owns indexing from source to searchable state. This article owns the storage lifecycle beneath that state: immutable segments, refresh, flush, delete markers, merge policy, compaction, reclamation and why a search index is often a collection of indexes rather than one continuously rewritten structure.
Quick Read
Lucene-style indexes are organised into segments. New indexed documents are buffered, written into new segments and made searchable without rewriting older segments. Elasticsearch’s current near-real-time documentation describes a Lucene index as a collection of segments plus a commit point; a refresh writes and opens new segments so recent operations become searchable. Updates commonly behave as delete-old-plus-add-new at the index layer. Old segments therefore accumulate deleted documents and many small files. Background merges combine segments, discard deleted documents and compact searchable state. Merge policy is a trade-off: merge too little and searches must inspect many segments; merge too aggressively and indexing consumes excessive I/O and CPU.
buffered writes → new immutable segment → refresh/searchable → more segments + delete markers → merge selection → read old segments → write merged segment → publish new segment set → reclaim obsolete files
A Segment Is a Searchable Mini-Index
A segment can contain its own:
- term dictionary;
- postings;
- stored fields;
- doc values;
- points/numeric structures;
- vector structures;
- deletion state and metadata.
A query over the logical index can search several segments and merge their candidate results.
The first pillar, Inverted Index & Postings, owns what the lexical structures inside those segments do.
Immutability Simplifies Concurrent Search
A reader opens a segment.
Another thread indexes new documents.
If the old segment remains unchanged, the reader does not need to watch individual postings mutate underneath it.
New state arrives as another segment or as separately managed live/deletion metadata.
Immutability moves complexity from in-place mutation into segment publication and lifecycle management.
The Indexing Buffer Is Not Yet the Searchable Index
Documents can first accumulate in memory.
The indexing system analyses them and builds structures before writing a segment.
Until a search-visible refresh occurs, a successfully accepted write may not be visible to ordinary search yet.
Refresh Opens New Searchable Segments
Elastic’s current Near Real-Time Search documentation explains that buffered documents can be written to a new Lucene segment, placed in the filesystem cache and opened for searching without requiring a full disk commit.
In Elasticsearch, that operation is called a refresh.
search visibility ≠ durable commit.
Refresh and Flush Solve Different Problems
A refresh makes recent index changes searchable.
A flush/commit lifecycle concerns durable persisted index state and transaction-log management.
The details vary by engine, but the conceptual split is important:
- visibility;
- durability;
- recovery.
Do not use one timestamp to pretend these are the same event.
Frequent Refresh Creates Many Small Segments
Refresh every tiny batch.
Search sees changes quickly.
But the index can accumulate many small segments.
Each query may need to open/search more structures and merge more partial results.
Freshness has a structural cost.
Updates Are Often New Documents Plus Old Deletions
A source document changes from version A to version B.
Instead of rewriting the old immutable segment in place, the index can:
- mark the old indexed document as deleted/not live;
- index the new version into a newer segment;
- let search suppress the deleted version;
- physically reclaim the old bytes later during merge.
The logical update becomes a lifecycle over multiple physical structures.
Deletion Visibility Can Precede Storage Reclamation
Remove a private document from results now.
The bytes may still exist inside an old segment file until compaction rewrites that segment without the deleted document.
This is a crucial distinction:
not retrievable ≠ physically reclaimed.
Security policy may require immediate search suppression while storage reclamation proceeds asynchronously.
A Merge Reads Several Segments and Writes a New One
Take segments S1, S2 and S3.
Build a new segment S4 containing the live indexed state that should survive.
Publish S4 into the index’s active segment set.
When no active reader needs the obsolete files, remove them.
This is compaction by rewrite rather than by in-place mutation.
Merging Reclaims Deleted Space
A segment contains 1,000,000 indexed docs.
300,000 are logically deleted.
Search ignores them.
The underlying segment may still occupy space for them.
A merge can rewrite only the live 700,000 documents into new segment structures and eventually reclaim the old storage.
Merging Also Reduces Search Fan-Out
Query ten segments.
Each can produce candidate matches.
The search layer combines them.
If those ten become two larger segments, there are fewer independent structures to inspect and fewer partial result sets to merge.
This can improve search efficiency—at the cost of the I/O required to perform the merge.
Merge Policy Is an Economic Policy
Which segments should merge?
How large should the resulting segment become?
How much deletion waste justifies rewriting?
How much background I/O can the system spend without hurting indexing or query latency?
A merge policy converts these competing costs into operational decisions.
Aggressive Merging Can Starve Foreground Work
Merge huge segments continuously.
Disk bandwidth fills.
CPU rises.
Cache churn increases.
Foreground search or indexing latency may suffer.
Compaction is maintenance work competing for the same physical machine.
Weak Merging Creates Segment Debt
Never merge enough.
Small segments accumulate.
Deleted bytes remain unreclaimed.
File descriptors, metadata and query fan-out increase.
Indexing saved work by postponing organisation.
Segment debt is that postponed work becoming visible.
Large Segments Are Not Free Either
A very large segment can be efficient to search.
But rewriting it to reclaim a small amount of deleted space can be expensive.
Merge policy therefore tends to balance segment size tiers, deletion ratios and available resources rather than seeking one giant perfect file.
Merges Rebuild Internal Statistics and Structures
New segment.
New internal document IDs.
Rewritten postings.
Repacked stored fields and doc values.
Potentially rebuilt vector/search structures.
This is why internal docIDs must never be confused with public stable resource identity: compaction can renumber or reorganise internal representation while the logical resource stays the same.
Stable External Identity Survives Segment Churn
Article URL/canonical ID remains stable.
Segment location changes.
Internal docID changes.
Search result can still route to the article because the canonical identity is stored or recoverable separately from the physical segment layout.
Refresh Timing and Merge Timing Are Independent
A new segment can become searchable immediately after refresh.
It may not be merged for minutes, hours or longer depending on policy and load.
Users can therefore search a structurally messy index while maintenance gradually compacts it underneath them.
Segment Search Makes Near-Real-Time Practical
If every new document required rewriting a global monolithic structure before search could see it, low-latency updates would be far more expensive.
Segment publication lets the system expose new searchable state cheaply and defer expensive consolidation.
The third pillar, Incremental Indexing & Freshness, owns when source changes become visible and how freshness is measured.
A Force Merge Is Not Ordinary Housekeeping
Some systems expose a force-merge operation that asks for aggressive consolidation.
This can be useful for largely read-only historical indexes.
It can be expensive or counterproductive on actively written indexes because future updates create new segments again and large forced segments can be costly to rewrite.
Maintenance operations should follow workload state rather than aesthetic desire for “one segment.”
Merging and Sharding Solve Different Scales
A shard can contain many segments.
Merging reduces segment fragmentation inside a shard/index partition.
Sharding distributes index capacity across partitions/machines.
Do not treat segment count and shard count as interchangeable scaling knobs.
Replicas Carry Their Own Segment State
Distributed search systems keep copies for availability and read capacity.
Replication and segment lifecycle must keep logical index state aligned even if physical merge schedules differ or files are copied/shared through engine-specific mechanisms.
The logical consistency contract matters more than identical physical byte layout at every instant.
Snapshots Observe Segment Immutability as an Advantage
Immutable segment files are naturally compatible with incremental snapshot ideas because unchanged files can be reused instead of copied again.
Once a new merged segment replaces older files, future snapshots can refer to the new structure.
Storage immutability becomes useful not only for search concurrency but also for backup mechanics.
Delete Is a Security Event Even Before Merge
A user’s permission is revoked.
A private resource is removed.
The search layer must stop exposing it at the policy boundary immediately enough for consequence.
Waiting for physical merge reclamation would be unacceptable.
The fourth pillar, Permission-Aware Indexing, owns the distinction between physical presence and authorised retrievability.
Search Results Can Change After Merge Without Semantic Change
A healthy merge should preserve the logical searchable content and ranking semantics expected under the same index state.
But implementation details, internal doc IDs, floating-point tie behaviour or statistics timing can occasionally affect edge cases.
Critical retrieval systems should therefore test observable search behaviour across maintenance, not assume storage reorganisation is invisible by definition.
Index Corruption Recovery Depends on Durable Source Ownership
If the index becomes corrupted, the safest repair may be to rebuild from:
- canonical database;
- source documents;
- event log;
- trusted snapshot.
This is why the master insists:
index ≠ canonical owner.
Segments are disposable/reconstructible representation, however expensive rebuilding may be.
Reindexing Is a Controlled New Segment Universe
Change tokenisation, field mappings or embedding model.
Old and new documents should not remain indefinitely under incompatible representation rules.
A controlled reindex can build an entirely new index generation from canonical sources, validate it and swap traffic after acceptance.
That is larger than a normal background merge: it changes the representation contract itself.
Compaction Has a Write-Amplification Cost
A document may be indexed once but its bytes can be rewritten several times as its segment participates in progressively larger merges.
Total storage bytes written can therefore greatly exceed final index size.
This write amplification is the price of combining cheap incremental writes with compact long-lived search structures.
Read Amplification Moves the Other Way
Many tiny segments can increase search work.
Merge them and query fan-out falls.
So indexing design balances:
- write amplification;
- read amplification;
- space amplification;
- freshness latency.
There is no free setting that minimises all four simultaneously.
A Better Segment-Lifecycle Model
source updates → indexing buffer → immutable segment → refresh/search visibility → live docs + tombstones → segment debt → merge policy → background compaction → new segment generation → old-file reclamation → continuous freshness/latency monitoring
A 30-Lens Segment and Merge Audit
- Buffer: where do writes live before segment creation?
- Segment: what immutable structures are written?
- Refresh: when does the segment become searchable?
- Commit: when does durable state advance?
- Flush: what recovery/log lifecycle applies?
- Small segments: how quickly do they accumulate?
- Query fan-out: how many segments does search inspect?
- Update: how is old indexed state superseded?
- Delete marker: when does a document stop being live?
- Physical bytes: when are deleted bytes reclaimed?
- Merge policy: which segments are selected?
- Merge size: how large may outputs become?
- Deletion ratio: when does waste justify rewrite?
- I/O budget: how much bandwidth can maintenance consume?
- CPU budget: how much compression/rebuild cost?
- Write amplification: how often are bytes rewritten?
- Read amplification: how many structures must a query touch?
- Space amplification: how much obsolete data remains?
- DocID churn: can internal IDs change safely?
- Canonical identity: what stable ID survives merges?
- Vector structures: what must be rebuilt during merge?
- Force merge: is the index sufficiently read-only to justify it?
- Shard: is segment maintenance being confused with distribution?
- Replica: how is logical consistency maintained?
- Snapshot: how does immutability help backup?
- Security delete: does access stop before physical reclaim?
- Reindex boundary: is representation contract changing rather than merely compacting?
- Failure: what if a merge is interrupted?
- Recovery: can the index rebuild from canonical data?
- World return: do search results remain correct across segment lifecycle?
Laboratory 1: Five Segments Become Two
Represent five small segments with paper cards. Mark several documents deleted, then merge three segments into one new card containing only live documents. Track which old physical cards can be discarded after readers release them.
Laboratory 2: Refresh Versus Merge
Create a timeline where new documents become searchable every second but merges occur every few minutes. Explain why a document can be searchable long before its segment is compacted.
Laboratory 3: Write vs Read Amplification
Compare two fictional policies: frequent aggressive merges versus rare merges. Predict which has more background writes and which forces searches across more segments.
For Primary Readers
Imagine writing new index cards into small piles instead of rewriting one giant book. Later you combine several piles into one tidy pile and throw away cards marked “old.”
For Secondary Readers
Explain why immutable segments make frequent writes easier and why merges are needed later to reduce small-segment and deleted-document overhead.
For Advanced Readers
Model segment-based indexing as an append/compact storage regime in which immutable per-segment search structures trade in-place write complexity for background merge amplification. Operational optimisation balances searchable freshness, segment fan-out, deletion density, I/O budget and long-run space efficiency.
Common Misconceptions
- “One search index is one physical file.” Lucene-style indexes commonly consist of multiple segments.
- “A refresh is the same as a durable commit.” Search visibility and durable commit/recovery are distinct lifecycle concerns.
- “Deleting a document immediately erases its bytes.” logical deletion can precede physical reclamation until merge.
- “More merging is always better.” aggressive merges consume I/O and CPU and can hurt foreground workload.
- “Internal docIDs must stay stable forever.” physical compaction can reorganise internal IDs; stable canonical resource identity must live above that layer.
Research Corridor
- Elastic — Near Real-Time Search — segment search, filesystem cache and refresh.
- Apache Lucene Documentation — current segment/index implementation APIs and concepts.
- Elastic — Merge Settings — operational merge behaviour and resource control.
- eduKateSG — How Indexing Works.
Frequently Asked Questions
What is an index segment?
It is an immutable or mostly immutable searchable sub-index containing a subset of indexed documents and the structures required to retrieve them.
Why do segments need merging?
Merging reduces the number of segments, compacts searchable state, discards logically deleted documents and reclaims storage, while trading background I/O and CPU for lower long-run search/space overhead.
Does deleting a document immediately remove it from disk?
Not necessarily. Search can stop treating it as live before old segment bytes are physically reclaimed during later merges or cleanup.
Final Thought: Fast Writes Borrow Organisation From the Future
The segment lets today’s document become searchable without reorganising yesterday’s entire index.
Merging is the repayment schedule: the system periodically turns many cheap immutable additions into a cleaner searchable state without surrendering the canonical identity above the storage layer.
INDEXING · FOUR PILLAR LEGS
Return to How Indexing Works, or continue through Inverted Index & Postings, Incremental Indexing & Freshness and Permission-Aware Indexing. Return to the Information & Representation Hub.