RAG That Thinks
The control-flow axis of RAG, one pattern at a time. Moving from retrieve-once to loops that reason and retrieve again, decide whether to retrieve at all, grade and correct their own results, and finally hand the whole thing to an agent.
The control-flow axis of RAG, pattern by pattern, each with an everyday analogy and an interactive visualizer: iterative and multi-hop RAG, adaptive RAG, FLARE, Self-RAG, corrective RAG (CRAG), speculative RAG, and agentic RAG. How each adds a loop, and then adds judgment to the loop, and how to stop at the least autonomous pattern that answers your questions.
A careful researcher does not answer from the first source they grab. They skim it, judge whether it is any good, and if it is thin they go back and search again before writing a single word. Most RAG just grabs once and answers. This is RAG that stops to check its sources and retrieves again when they are weak.
This is part of a series on RAG. The earlier pieces fixed retrieval and indexing, the static half. This one is about the dynamic half: the control flow. Naive RAG retrieves exactly once and then answers, and for a lot of questions that is simply not enough. The techniques here add a loop, and then add judgment to the loop. We will climb them one at a time, each with an everyday analogy, the real mechanism, and a demo.
Read that spectrum left to right as increasing autonomy. Each step buys the ability to answer harder questions and costs latency, tokens, and predictability. The skill is stopping at the least autonomous point that actually answers your questions.
Iterative and multi-hop
A detective does not crack a case from one clue. Each answer points to the next question: find who owned the car, which tells you where they worked, which tells you who they called. You cannot even ask the last question until the earlier answers are in hand.
Some questions are like that, and no single retrieval can answer them, because answering them requires a fact you can only search for after you have found an earlier one. "Who succeeded the person who founded the company that built X" is a chain: find X's builder, then its founder, then their successor. Iterative RAG runs the loop multiple times, retrieving, reasoning about what it now knows, and retrieving again for the next hop. The retrieval is no longer a preamble to thinking; it is interleaved with it.
Hop 1: retrieve “What company built the iPhone?”, get “Apple”, then feed that into the next question.
Adaptive: retrieve only when it helps
You do not open a reference book to answer "what is 2 plus 2." You reach for it only when a question is actually beyond what you already know. A good researcher asks first: do I even need to look this up?
Retrieval has a cost, and not every question needs it. Asking a model to retrieve before answering "what is 2 plus 2" wastes time and can actively hurt, by dropping irrelevant context into the prompt. Adaptive RAG puts a decision before retrieval: does this query even need external knowledge, and if so, how much. Simple queries go straight to the model; hard ones get the full multi-step treatment. The router works like a triage nurse, sending a scraped knee to a bandage and a chest pain to the full workup, sizing the effort to the question.
Skip retrieval. the model already knows this; retrieval would only add noise and latency.
FLARE: retrieve for the words you are unsure of
Now picture writing an essay from memory. Your pen moves smoothly through the parts you know, until you reach a specific date or figure you are not certain of. There, and only there, you stop, look it up, and keep writing. You never re-check the words you already trust.
FLARE, forward-looking active retrieval, is that instinct woven into generation itself. As the model writes its answer, whenever it is about to produce a span it is not confident about, it pauses, retrieves on that span, and continues. It retrieves when and only when it is about to say something it is unsure of, which is exactly when grounding is worth the cost.
Confident words are written straight out; retrieval only fires on the shaky ones.
Self-RAG: grade yourself as you write
Picture a meticulous writer who keeps checking their own work in the margin: do I need a source here? Is this quote actually relevant? Does my sentence really follow from it? And when the answer is no, they rewrite on the spot, before anyone sees the draft.
Self-RAG trains the model to emit special reflection tokens as it works: little marker words it writes with the same pen, in-line, not in a separate pass, the way our writer scribbles "check this" or "does not follow" in the margin mid-sentence. Some decide whether retrieval is needed, and others grade whether a retrieved passage is relevant and whether the generated answer is actually supported by it. The model critiques itself in-band and can retrieve more or revise when its own grades come back poor.
Special reflection tokens decide whether to retrieve, whether a passage is relevant, and whether the answer is actually supported. A “no” on the last one sends it back to revise, in-band, before anything is shown.
Corrective RAG (CRAG): recover from weak retrieval
A good editor does not run a story built on thin sourcing. If the material looks weak, they send the reporter back out, to check another source or re-interview, before a word goes to print. Better a delay than a confident falsehood.
Corrective RAG (CRAG) adds a lightweight evaluator over the retrieved documents. If they look strong, it proceeds. If they look weak or ambiguous, it does not just plough ahead; it falls back, typically to a web search or a query rewrite, and tries to correct the retrieval before generating. The core idea is that bad retrieval should trigger a recovery, not a confidently wrong answer.
Speculative RAG: draft several, verify, pick
When a hard question lands in a group, a few people each jot a quick answer from whatever they happened to read, and then the most knowledgeable person reads all of them and picks the best. That is often faster and better than one person grinding it out alone.
Speculative RAG works the same way. A smaller model drafts several candidate answers from different retrieved subsets in parallel, and a larger model verifies and picks the best, trading some extra retrieval for better answers without a long serial loop.
A small model drafts several candidate answers in parallel, each from a different retrieved subset of the documents.
Agentic RAG: retrieval as a decision
Give a capable assistant a hard question and they do not follow a fixed script. They decide: is this in our own files, or do I need the web, or the database, or should I ask around? They pick the right place, check what they got, and go somewhere else if it comes back thin.
At the far end of the spectrum, retrieval stops being a step in a pipeline and becomes one tool among several that an agent decides how to use.
An agentic RAG system plans: it can break a question into sub-questions, choose which source to hit for each (a vector index for documents, a web search for fresh facts, a SQL database for numbers, a knowledge graph for relationships), grade what comes back, and loop, trying another source or refining the query when a result is weak. Router RAG, the pattern of classifying a query and directing it to the right retriever or index, is the simplest version of this idea; full agentic RAG is the general one.
The agent routes the question to the vector index and grades the result: weak, the answer is not in the indexed documents. So it does not plough ahead.
This buys a great deal of capability and is genuinely harder to operate. Every added loop is more latency, more tokens, more places to go wrong, and less predictable behaviour. An agent that can retrieve five times can also spin five times. Which is precisely why the control loop deserves the same engineering discipline as any other agent loop: budgets, termination conditions, and a way to test the trajectory, not just the final answer.
Where to stop
Match the control flow to the questions. If single-shot retrieval answers your users, adding a self-correcting agent just makes it slower and flakier. If your questions genuinely require chaining facts across sources, no amount of clever retrieval or chunking substitutes for a loop. Climb the spectrum only as far as your hardest real question forces you to.
The last piece asks what happens when even a smart loop over flat vectors is not enough, because the knowledge itself has a shape that vectors flatten: Beyond the Vector.
