How RAG Reads a Corpus
The indexing axis of RAG, strategy by strategy. Why chunking is the quiet make-or-break decision, and how to decouple the unit you search from the unit you read.
The indexing axis of RAG: why chunking is the quiet make-or-break decision, and the ladder of strategies, each with an everyday analogy and interactive visualizer: fixed-size chunks, structure-aware splitting, sentence-window retrieval, parent-document (small-to-big), summary indexing, RAPTOR's hierarchical tree, and contextual and late chunking. How to decouple the unit you search from the unit you read.
Before you can look anything up in a book, someone had to decide what goes on each index card. Cut the text into cards that are too big and each one is a blurry mix of topics; cut them too small and you lose the thread that made a passage make sense. Chunking is that cutting, and it quietly sets the ceiling on how well retrieval can ever work.
This is part of a series on RAG. The last piece was about how you match a query to your data. This one is about what you decided to match against in the first place, which is set long before any query arrives, at indexing time. It is the least glamorous part of RAG and quietly the most decisive. Each strategy below gets an everyday analogy first, then the mechanism, then a demo.
The tension at the heart of chunking
You cannot embed a whole document as one vector and expect good retrieval; the signal gets averaged into mush. So you split documents into chunks. But now you face a tension that never fully resolves: the unit that retrieves well and the unit that reads well are not the same size.
Small chunks embed precisely, because a short passage is about one thing, so its vector is sharp. But a small chunk handed to the model may be too little context to answer from. Big chunks carry plenty of context to read from, but they embed poorly, because a long passage is about many things and its vector smears across all of them. Every indexing strategy below is a different way to escape this bind.
You can feel the tension directly. Drag the chunk size and overlap below and watch the same document break into more, sharper pieces or fewer, fuller ones:
A vector database stores embeddings and finds the nearest ones to a query. Before anything can be stored,
anything can be stored, the source documents must be split into smaller passages called chunks. Each chunk becomes
chunks. Each chunk becomes a single vector. If a chunk is too large it mixes several topics and
mixes several topics and its vector turns blurry. If it is too small it loses the surrounding context
loses the surrounding context that made it meaningful. Overlap keeps a sentence from being cut in half at
cut in half at a boundary.
Fixed-size chunking
The laziest way to cut a book into index cards is to cut every two hundred words, no matter what falls on the boundary. Quick, but you will slice a recipe in half and split a definition from the word it defines.
That is fixed-size chunking, the default everyone starts with: cut every N tokens, with some overlap between neighbours so a sentence straddling a boundary is not lost. It is trivial to implement and genuinely fine for a first pass. Its flaw is exactly the book-cutting one: it slices with no respect for meaning, cutting tables in half, severing a claim from its condition, splitting a definition from the term it defines.
One clean document, before any cutting.
Structure-aware chunking
A better cut follows the seams the author already put there. Cut on the chapter, heading, and paragraph breaks, and every card is a whole thought instead of an arbitrary window.
Structure-aware chunking is the first upgrade: split on the document's own boundaries, headings, paragraphs, sentences, list items, so each chunk is a coherent unit rather than an arbitrary slice of tokens.
The same document, before any cutting.
Decouple the search unit from the read unit
The real breakthrough is to stop assuming the chunk you search has to be the chunk you read. Once you separate them, several strategies fall out of the same idea: the vector you match on is not the text you read from. It is the highest-leverage idea in RAG indexing, and it costs almost nothing.
Sentence-window retrieval
Index each sentence on its own so the match is pinpoint, but when one hits, hand over the sentences on either side of it too. A quote always makes more sense with the lines around it.
Sentence-window retrieval does just that: index individual sentences so matching is precise, and when a sentence matches, return it plus a window of the sentences around it, so the model gets the context the bare sentence lacked.
Each sentence is indexed on its own, so the match is sharp: only the sentence about the 30-day return matches.
Parent-document, or small-to-big
Search by the small, sharp label on the drawer, but once you find the right one, pull out the whole folder inside. You match on something tiny and precise, then read something large and complete.
Parent-document retrieval indexes small child chunks, but when a child matches, hands the model its larger parent chunk or whole section. You search small and read big.
Small child chunks are indexed and matched precisely: the “30-day defective return” child is the hit.
Summary indexing
A library catalogue card is a one-line summary, not the book itself. You search the cards, and when one matches, you go and fetch the actual book off the shelf.
Summary indexing works the same way: index a short summary of each document, and when a summary matches, retrieve the full source. Good when documents are long and topically distinct.
Only a short summary of each document is indexed, so matching is fast and topical: the IT policy summary wins.
RAPTOR: retrieve at the right altitude
Small-to-big helps within a document. But some questions are not about a passage at all; they are about a whole section, or the whole corpus. "What are the main risks in this report?" has no single chunk that answers it, because the answer is spread across many. Flat chunking simply cannot serve that query.
A good book lets you read at any zoom: a one-line chapter summary, a section, or the exact paragraph. RAPTOR builds that ladder. It clusters the chunks, like sorting a pile of index cards into topic piles, summarizes each cluster into a higher-level node, one summary card per pile, then clusters and summarizes those, and repeats until it has a single root card that summarizes everything.
Now retrieval can land at any altitude. A detailed question matches a leaf chunk. A broad question matches a summary node that already condensed the relevant span. The same index serves both the specific and the thematic, because it holds the corpus at multiple levels of abstraction at once.
The same index holds the corpus at every level of abstraction, so a broad, whole-section question matches a summary node that already condensed the relevant span.
Contextual and late chunking
Two more refinements are worth knowing, both about the same problem: a chunk that has lost the thread of where it came from.
Contextual chunking
A scrap of paper that just says "the rate rose to 4 percent" is useless until you staple a note on top naming the company and year. Contextual retrieval, covered in the retrieval piece, prepends a short situating blurb to each chunk before embedding it, so the chunk is not an orphaned fragment.
On its own the chunk is an orphan. Its embedding is ambiguous, so the query that should find it does not.
Late chunking
Think of being handed one page torn from a novel and asked what it means. On its own, "she finally agreed" is a mystery: agreed to what? Now imagine reading the whole novel first, then turning to that same page. The words are identical, but now you know who "she" is and what she agreed to. Late chunking does the second thing.
Instead of cutting the page first and embedding each scrap blind to the rest, it reads the whole document first, then decides the cuts. Late chunking exploits long-context embedding models: run the whole document through a long-context embedder first, then pool the token embeddings into per-chunk vectors, blending each section's word-by-word impressions into one impression that still remembers the rest of the page. Each chunk's vector is informed by the entire document, so it keeps context that naive chunking throws away, without adding any extra text to the chunk. A related idea, sometimes called LongRAG, simply uses much larger retrieval units now that long-context models can read them.
Naive chunking cuts first and embeds each scrap on its own, so the chunk's vector never learns the context that surrounded it.
The rule of thumb
Start with structure-aware chunking and a decoupled search-versus-read unit; that alone gets most systems most of the way. Add RAPTOR when your users ask broad, whole-document questions that no single passage can answer. Add contextual or late chunking when your chunks are losing the thread of where they came from.
Retrieval and indexing are the static half of RAG: decided before any question arrives. The next piece is about the dynamic half, what the system does at question time when one retrieval is not enough.
