RAG chunking is one of the foundational design decisions in retrieval-augmented generation, semantic search, vector databases, embeddings and information retrieval. Before a RAG system can retrieve evidence from a long document, it usually divides that document into smaller retrieval chunks: paragraphs, fixed token windows, sections, sentences, semantic units or recursively split passages. The choice of chunk size, chunk overlap, semantic chunking and document segmentation directly affects retrieval recall, embedding quality, context preservation, vector search precision and the evidence eventually supplied to a language model.
The difficulty is that a document was not written to satisfy a vector database. Meaning crosses boundaries. A claim may sit in one chunk while its exception sits in the next; a table may be separated from its caption; a mathematical theorem may be separated from its assumptions; a pronoun may lose its antecedent; a policy rule may be retrieved without the paragraph defining who it applies to. Chunking in RAG is therefore lossy information architecture: every cut creates a possibility that the retrieval system will see only part of a meaning that was originally continuous.
This longform explains how RAG chunking works, fixed-size versus semantic chunking, recursive text splitting, chunk overlap, parent-child retrieval, contextual retrieval, document structure, embedding dilution, retrieval precision and recall, chunk boundaries, tables and code, multilingual documents, metadata, reranking, context windows and evidence reconstruction. The governing principle is straightforward: the best chunk is not the smallest or largest unit; it is the smallest retrievable unit that preserves enough surrounding structure for the receiver to interpret the evidence correctly.
Quick Read: What Is RAG Chunking?
Chunking is the process of dividing a larger source into smaller units that can be indexed, retrieved and passed into later stages of a RAG pipeline.
A 100-page report might become 500 passages.
A textbook might become sections and subsections.
A code repository might become functions, classes or files.
A transcript might become speaker turns or topical segments.
One-sentence answer: RAG chunking is lossy because it replaces one continuous source with independently retrievable fragments, preserving local evidence while weakening or severing some relationships that crossed the chosen boundaries.
Why Chunk at All?
Why not embed an entire document?
Because one document can contain many topics, details and local answers.
A single-vector representation of a long document can dilute a short decisive passage inside a much broader semantic average.
Chunking gives that local passage its own retrievable representation.
It is a deliberate move from document-level meaning to local evidence units.
The Boundary Is the Hidden Model
When engineers discuss retrieval models, they often focus on embeddings, vector indexes and rerankers.
But the chunk boundary is already a model of meaning.
A fixed 500-token splitter assumes that approximately 500 tokens form a useful retrieval unit.
A paragraph splitter assumes paragraph boundaries correlate with semantic boundaries.
A heading-aware splitter assumes document authors organised sections meaningfully.
A semantic splitter assumes changes in embedding geometry reveal changes in topic.
Every splitter carries a theory of where one idea ends and another begins.
Fixed-Size Chunking: Predictable and Blind
Fixed-size chunking divides text after a chosen number of characters, words or tokens.
Its strength is operational simplicity.
Index sizes are predictable. Embedding batches are regular. Context budgets are easy to estimate.
Its weakness is semantic blindness.
The 500th token may occur halfway through a sentence, halfway through a proof or between a rule and its exception.
Recursive Splitting Tries Better Boundaries First
A recursive splitter can try large natural boundaries first—sections, paragraphs, sentences—and use smaller boundaries only when the resulting unit remains too large.
This preserves more authored structure than blind fixed windows.
But it still operates under a size constraint.
A coherent section that exceeds the budget will eventually be divided.
Semantic Chunking: Let Meaning Decide the Cut
Semantic chunking attempts to detect changes in topic or representation and place boundaries where meaning shifts.
This can produce chunks of unequal size that better follow the source’s conceptual structure.
The new problem is that semantic change is itself model-dependent.
An embedding model may consider two adjacent paragraphs similar even when a small legal or scientific distinction between them is decisive.
The splitter becomes only as sensitive as the representation used to detect change.
Small Chunks Improve Local Retrieval
Suppose a 30-page document contains one sentence answering a narrow question.
Embed the whole document and the sentence may barely influence the representation.
Embed the paragraph containing it and the evidence becomes much more prominent.
Small chunks reduce evidence dilution.
Small Chunks Also Lose Interpretation
A sentence can be locally relevant and globally misleading.
“Students may submit after Friday.”
That looks clear.
The previous sentence might say, “Only students granted an approved medical extension are covered by the following exception.”
Retrieve only the first sentence and local precision has destroyed scope.
Large Chunks Preserve Scope
Larger chunks keep neighbouring definitions, examples and qualifications together.
This is valuable for legal clauses, scientific methods, narrative reasoning and instructional material where interpretation depends on context.
The price is retrieval dilution and context cost.
There Is No Universal Best Chunk Size
A single number such as 256, 512 or 1,000 tokens cannot be optimal across every corpus.
A dictionary definition, legal contract, mathematics proof, source-code file, transcript and research paper contain different natural semantic units.
Chunk size should therefore be treated as a corpus-and-task parameter.
Chunk Overlap: Paying Twice to Protect the Boundary
Overlap repeats some content across adjacent chunks.
A 500-token chunk with 100-token overlap might share its final 100 tokens with the next chunk.
This protects evidence near the boundary because it can appear with context on both sides.
The cost is duplicated index content, repeated retrieval and more tokens downstream.
Overlap Is Not Free Recall
More overlap can improve boundary coverage.
It can also flood top-k retrieval with near-duplicate chunks from the same source.
Five retrieved passages may appear diverse while actually containing the same paragraph repeated across neighbouring windows.
Overlap therefore protects local evidence and can reduce source diversity at the same time.
Deduplicate After Retrieval
One repair is to identify heavily overlapping retrieved chunks and collapse or diversify them before context construction.
This preserves overlap during indexing while preventing duplicated evidence from consuming the entire final context budget.
The system keeps redundancy where it protects recall and removes redundancy where it wastes receiver capacity.
Parent-Child Retrieval Separates Search Unit From Reading Unit
A powerful design uses small child chunks for retrieval and larger parent sections for final context.
The small child makes local evidence easy to find.
Once matched, the system expands back to a larger parent that restores surrounding explanation.
This is a return path.
It acknowledges that the best unit for finding evidence need not be the best unit for understanding evidence.
The Parent Is a Recovery Master
In a parent-child design, the child chunk is an access derivative.
The parent retains richer context.
The original document remains the master.
This hierarchy is safer than treating the small retrieval chunk as if it were a complete evidence object.
Contextual Retrieval Adds Lost Surroundings Back Into the Chunk
A chunk can be enriched with document title, section heading, nearby summary, entity names or a short description of where it sits in the source.
This contextual prefix helps an embedding or lexical retriever understand a fragment whose meaning would otherwise be ambiguous.
For example, “The limit is 30 days” becomes much more retrievable when prefixed with “Refund policy — international customers — cancellation window”.
The retrieval representation becomes slightly larger in order to repair context lost during segmentation.
Metadata Is Compressed Context
Document title, author, date, jurisdiction, product, chapter, subject, source type and access permissions can all travel with the chunk as metadata.
Metadata gives a fragment coordinates in the larger information system.
Without it, identical sentences from different contexts can become indistinguishable.
Headings Should Usually Survive
Headings are compact semantic labels written by the source author.
Discarding them during chunk extraction can remove high-value context at very low token savings.
A heading such as “Exceptions”, “Contraindications”, “Limitations” or “For Secondary 3 Students” can radically change interpretation of the paragraph beneath it.
Lists Need Structural Integrity
A list item may depend on an introductory sentence.
“The following cases are excluded:” is the semantic parent of every bullet below it.
If the bullets become independent chunks without the parent statement, their polarity can invert.
Chunking should preserve list headers and enough hierarchy to keep the bullets interpretable.
Tables Are Not Paragraphs
A table derives meaning from row and column relationships.
Split rows into independent text fragments and the header can disappear.
“42” means nothing without knowing whether the column is age, price, percentage, score or sample size.
Table-aware chunking should preserve headers, units and enough neighbouring structure to interpret each value.