Vectorless RAG

The default recipe for RAG is simple: chop a document into chunks, turn each chunk into a vector, and at query time fetch the chunks whose vectors look most similar to the question. It works well for a lot of things. But for a long, structured document, it throws away the one thing that made the document usable in the first place: its structure.

Picture a 200-page employee handbook and one question: how long do I have to return a defective laptop? One way to answer it is to run a photocopier over the whole book, keep every page that seems related to the question, and read that shuffled stack out of order. The other way is to open the table of contents, see "Equipment and Returns," turn to "Defective Devices," and read the one paragraph that answers you, in place. Vector search is that photocopier, except it matches by meaning rather than by exact words. Vectorless RAG is the table of contents.

Similarity is not relevance

A face in a crowd can look a lot like your friend's and still belong to a stranger. Resemblance is not identity. Vector RAG retrieves by similarity: it finds the chunks whose embeddings sit closest to the question's embedding. Most of the time, close in meaning is close enough. But similarity and relevance are not the same thing, the resemblance can be strong and the passage still be the wrong one, and that gap is the whole story here.

A passage about "paid time off, accrued in days" is genuinely similar to "how many days do I have," yet it is the wrong answer. Surface similarity pulled in a look-alike. And there is a second, quieter cost. To make a document searchable by vectors at all, you first had to chunk it, which cuts each section out of its home and discards the hierarchy that told you what belonged where. Once "Defective Devices" is just a floating 300-word chunk, the fact that it lived under "Equipment and Returns" is gone.

What vectorless RAG does instead

Vectorless RAG keeps the document whole. Instead of chunk-then-embed, it builds a table-of-contents tree over the document: chapters, sections, and subsections, each with a short summary and its page range. There is no vector database, and nothing is cut into embedded retrieval chunks. (The document may still be read in page-sized groups while the tree is built, but the unit you retrieve is a section of the tree, not an arbitrary chunk.)

At query time, a language model navigates that tree the way a human expert would. It starts at the top, reads the section titles and summaries, decides which branch is most relevant, and descends, one level at a time, until it reaches the section that actually answers the question. Retrieval becomes reasoning over structure rather than similarity over fragments. The clearest current example is an open-source project called PageIndex, which describes itself as "vectorless, reasoning-based RAG": no vector DB, a hierarchical tree index, and retrieval through tree search that mirrors how a person finds an answer in a long document.

The whole pipeline has two phases. Once per document it builds the tree. Then, for every question, it reasons down that same tree to the answer. There is no embedding step in either phase.

Here is how that tree gets built from a raw document. Notice that nothing is embedded along the way, and the document is never cut into retrieval chunks:

The handbook, both ways

Take the laptop question against that handbook. Section 4 is "Equipment and Returns," with subsections 4.1 Company Laptops, 4.2 Returning Equipment, and 4.3 Defective Devices. The 30-day return rule lives in 4.3.

The vector way finds 4.3, but it also grabs 2.3 Paid Time Off (it matched "days"), 4.1 Company Laptops (about issuing a laptop, not returning it), and 5 Leaving the Company (returning equipment when you quit). Four results, out of order, each one stripped of where it sat in the book. The right passage is in there, buried in noise and stripped of context.

The vectorless way walks the tree: from the root it picks "Equipment and Returns" over compensation and workplace policy, then inside it picks "Defective Devices" over the general laptop and returns sections, and reads the 30-day clause in place, with the exact path that led there. Try it both ways:

Where it can fail

Vectorless RAG is only as good as the structure it builds. When there is no usable table of contents, the model is guessing the hierarchy, and a scanned or badly formatted PDF, or hidden tables and appendices, can send it confidently down the wrong branch. Some questions do not suit tree navigation at all: comparing many sections at once, finding a clause repeated across the whole document, or pulling a small detail that never made it into a section summary. There a vector or hybrid retriever can still find the needle faster. And however you retrieve, judge it by more than whether the answer sounds right: check that it opened the right section and cited the right pages, and watch cost and latency, since every query spends model calls. A later part of this series goes deep on evaluating retrieval.

When to reach for it

Vectorless RAG is not a replacement for vector search. It is a different retrieval shape for a different kind of document.

• Reach for it when the document is long and structured, and the structure carries meaning: financial filings, legal contracts, technical manuals, regulatory documents, medical literature, academic texts. Here, navigating the hierarchy beats matching fragments, and keeping a passage in context matters.
• Stay with vector or hybrid search for huge, messy corpora, real-time support knowledge bases, web-scale search, and noisy documents, or anywhere plain keyword plus vector retrieval is already good enough. Tree navigation is reasoning-heavy at both ends: building the tree takes many model calls up front, and every query spends more model calls reasoning through it, where a vector lookup costs almost nothing once the index exists.

The two are often combined rather than chosen between. On its own, vectorless retrieval is clearest inside one long document; to aim it at the right document out of many you either add a document-selection step first (often vector search), or build a higher-level tree over the whole corpus, a table of contents for the whole shelf rather than for a single book, so you first pick the right book and then navigate inside it, which is what PageIndex's newer File System layer does with the same tree search. The common shape is vectors on the outside to pick the document, tree navigation on the inside to find the passage.

The takeaway

Vectors are not dead. The lesson underneath vectorless RAG is the one worth keeping: in retrieval, what you actually want is relevance, and for a structured document, relevance often lives in the structure. Similarity is a cheap and useful proxy for it, not the same thing. And notice what vectorless retrieval does not do: it does not remove the guesswork, it moves it out of vector space and into the model's reasoning, which is why it buys accuracy at the price of more model calls. Put plainly: vectorless RAG is not RAG without retrieval, it is retrieval through the document's structure instead of through embedding similarity. When you have shredded a document to search it and the answers keep coming back out of context, that is the signal to stop shredding and start navigating.

This is part six of the RAG series. Vectorless RAG is newer in packaging than in spirit: structure-aware retrieval also shows up as the hierarchical summaries of RAPTOR in How RAG Reads a Corpus, and as graph navigation in Beyond the Vector. For the vector side of the coin, see How RAG Finds Things.