RAGOps: RAG Is a Distributed System, Not a Demo
Why a working RAG demo is the easy 10 percent: the corpus changes under you, so production RAG needs a data lifecycle, deletion that actually deletes, a plan for the day you swap the embedding model, and a trace for every answer.
Part seven of the RAG series, and the first on running RAG in production. A demo retrieves from a fixed pile of documents and answers one question. Production retrieves from a corpus that changes every hour, so the hard part is not the answer, it is keeping the index true to the source: continuous ingestion and incremental re-indexing for freshness, deletion that removes a document's chunks so the model cannot cite what no longer exists, a migration plan for the day you change the embedding model and every old vector becomes meaningless, and a trace on every answer so you can see why it said what it said. Grounded in the RAGOps paper. Everyday analogies, a hand-drawn diagram, and interactive visualizers.
The previous six parts built up RAG piece by piece: retrieval methods, chunking and indexing, the control flow of iterative and agentic RAG, graph and multimodal variants, and the vectorless approach. All of it assumed a fixed pile of documents you retrieve from. This part is about what changes when that pile is alive: a corpus that grows, gets edited, and gets deleted while real users ask questions against it. That is the difference between a RAG demo and RAG in production.
Think about the difference between cooking one dish for a photo and running a restaurant kitchen. The photo needs one perfect plate, once. The kitchen has to serve that dish at 8pm on a Friday when three deliveries arrived late, one supplier changed a recipe, a crate of tomatoes has gone off and must be pulled before it reaches a plate, and a health inspector could walk in at any moment. The recipe is the easy part. The hard part is freshness, stock rotation, throwing out what has spoiled, and being able to show, for any plate, exactly what went into it.
A RAG demo is the photo. You point it at a folder of PDFs, ask a question, and it answers. RAG in production is the kitchen. The corpus changes every hour, and the job is not "get one good answer," it is "keep the retrieved knowledge true to a source that never sits still, and prove it." The research community has a name for this discipline: RAGOps. As the RAGOps paper (Xu et al., 2025) puts it, RAGOps "extends LLMOps by incorporating a strong focus on data management to address the continuous changes in external data sources." That sentence is the whole article. Everything below is what "a strong focus on data management" actually costs you.
Two clocks: the demo has one, production has two
Here is the mental shift, in one picture. A demo has a single clock: a user asks, the system answers. You build the index once, up front, and never think about it again. Production has two clocks running at very different speeds. There is the fast clock of serving (a query comes in, you retrieve and generate, in under a second) and the slow, never-stopping clock of the data (documents arrive, change, and disappear all day). The RAGOps paper makes exactly this point: data management runs at a "much higher frequency, potentially continuously," than updates to the query pipeline.
Almost every production RAG failure is really a failure to keep those two clocks in sync. The serving side answers confidently from an index that the data side let go stale. So the rest of this article is four ways the data clock gets out of sync with reality, and what you do about each. Freshness (the index falls behind new writes). Deletion (the index still holds what the source deleted). Drift (you change the embedding model and every stored vector silently stops meaning what it used to). And observability (you cannot fix any of the above if you cannot see, for one answer, what was retrieved and why).
Freshness: the index has to chase a moving corpus
Imagine a library where new books arrive daily but the card catalog is only retyped once a year. For most of the year, the catalog lies: it lists books that were returned and misses books on the shelf. A RAG index built once and never updated is that stale catalog. If your support docs changed this morning and the index was built last week, the model retrieves last week's answer and states it with full confidence.
The fix is not to rebuild the whole index every time one document changes, which for a large corpus is slow and expensive. It is incremental updating: when a document is added or edited, you re-chunk and re-embed only that document and update only its entries in the index. The RAGOps paper calls this "updating retrieval sources," and is precise about it: "Incremental chunking and embedding are performed on newly added or modified data. Indexing is also updated accordingly." Before that, the pipeline detects the change at all, using mechanisms the paper lists: webhooks, change data capture, polling, and file watchers.
Worked example. A 50,000-document knowledge base. At 2:14pm, someone edits one policy page. A file watcher fires; the pipeline re-chunks that one page into, say, 8 chunks, embeds those 8 chunks, and upserts them into the index, replacing the 8 old ones. Total work: 8 embeddings, not 50,000. By 2:14pm and a few seconds, a user asking about that policy gets the new answer. That is freshness: the catalog chases the shelf, one book at a time.
The index is built once. The policy page and its 8 chunks in the index agree: both v1, fresh.
Deletion: the index must forget what the source forgot
A shredded document is not gone if a photocopy is still in the filing cabinet. When a source document is deleted, whether because it expired, was wrong, or a person asked to be forgotten, deleting the original is not enough. Its chunks are still sitting in the vector index, and the retriever will happily pull them and the model will cite them, quoting a document that officially no longer exists. This is one of the most common and most dangerous production RAG bugs, because it turns "we deleted that" into a lie the system tells on your behalf.
Deletion has to propagate: when a document leaves the source, every chunk derived from it must be removed from the index in the same operation. In a real system each chunk carries the id of its parent document, so deletion becomes "find every chunk whose parent is document D, and purge it," not "hope nobody asks." The RAGOps paper keeps versioning and access-control metadata alongside each data entry (there, so that retrieval respects permissions); the same per-document bookkeeping is what lets a delete find every chunk it needs to remove. This is standard production discipline more than a headline result, which is exactly why it is so easy to skip.
Worked example. A customer invokes their right to be forgotten under a privacy law. Their record is one source document that was chunked into 5 pieces months ago. A correct pipeline looks up document D's 5 chunk ids and deletes exactly those 5 vectors from the index, in the same transaction that removes the source. A broken pipeline deletes the source row and leaves the 5 chunks behind, so next week the model answers a question using that person's supposedly-erased data. Same corpus, and one of these is a compliance incident.
A customer record is one source document, chunked into 5 pieces months ago and stored in the index.
Drift: the day you change the embedding model, every old vector lies
Suppose two people index the same library, but one measures every book's position in meters from the door and the other in feet from the window. Both catalogs are internally fine. Mix them, ask "what is near position 10," and you get nonsense, because 10 means different things in each system. An embedding model is the ruler that turns text into a vector. Every vector in your index was measured with one specific model. The day you upgrade to a better embedding model, that new model is a different ruler, so a query embedded with the new model and a document embedded with the old one no longer live in the same space, and their similarity scores are meaningless.
This is why you cannot just swap the embedding model in place. Changing it means re-embedding the entire corpus with the new model and rebuilding the index, then cutting over. The RAGOps paper tracks exactly this risk as a module-level signal, listing "embedding drift" alongside cosine similarity in its monitoring table. Drift is not only a deliberate upgrade, either: if a provider silently updates a hosted embedding model behind the same name, your new vectors quietly stop matching your old ones, and retrieval quality decays with no code change on your side. Pinning the model version, and re-embedding on purpose when you change it, is the defense.
Worked example. You have 1,000,000 chunks embedded with model v1, and v2 is released with better retrieval scores. You cannot query v2 against a v1 index. The migration is: stand up a new index, embed all 1,000,000 chunks with v2 (a real batch job with a real bill), verify retrieval quality on a test set, then switch traffic over and retire the v1 index. Skipping the re-embed and pointing v2 queries at the v1 index does not error out; it just quietly returns worse matches, which is worse, because nothing tells you it broke.
Query and document are both embedded with model v1, the same ruler. They land close together, so the match is real.
Observability: you cannot fix an answer you cannot trace
If a dish comes back to the kitchen, "the food was bad" is useless; "the salmon on table 6, cooked at 7:52, from the batch delivered Tuesday" is something you can act on. RAG answers fail the same way. "The bot gave a wrong answer" is not debuggable. What you need, for any single answer, is the full trace: what the query was rewritten to, which chunks were retrieved and with what scores, which the reranker kept, and what the model did with them. Without that trace, every bug is a guess.
The RAGOps paper builds this in as two of its five system qualities. Observability tracks, per stage, signals like retrieval relevance and injection-attack detection on the way in, and context adherence, toxicity, and usefulness on the way out. Traceability requires "versioning, retrieval routing, grounded source" and documenting the "input and output data of each component." In practice this is a per-request trace, the same idea as distributed tracing in any microservice system, applied to the RAG pipeline: one trace id follows the query through query-processing, retrieval, rerank, and generation, and records the numbers at each hop (retrieval latency, mean reciprocal rank of the retrieved set, and a faithfulness score on the final answer, all metrics the paper names).
Worked example. A user reports a wrong answer. You open its trace. You can now see the failure is at a specific hop: maybe retrieval pulled the right chunk but the reranker dropped it, or retrieval never found it (a chunking or freshness problem), or both were fine and the model ignored the context (a faithfulness problem). Each of those has a different fix, and the trace is what tells you which one you are looking at. The next part of this series is entirely about the metrics on that final hop.
A user reports a wrong answer. 'The bot was wrong' is not debuggable. Open its trace instead.
The rest of the kitchen: fallback, versioning, cost
Three more operational realities that do not each need their own picture, but that separate a demo from a system. Fallback: retrieval will sometimes return nothing useful, and the system needs a defined behavior for that (say "I do not have that information" rather than letting the model fill the silence with a guess). This is general production practice more than a result of the paper; the closest the RAGOps paper comes is its guardrails, placed in the pipeline as input, dialog, retrieval, and output rails that can reject or modify a response before it reaches the user. Versioning: the corpus, the chunking rules, the embedding model, and the prompt are all things that change, and traceability means every answer can be tied back to the versions that produced it, so a regression is diagnosable. Cost and latency: every extra step (query rewriting, reranking, multiple retrievals) adds tokens and milliseconds, and in production those add up to a real bill and a real p99, which is why the paper treats latency and resource usage as first-class monitored signals, not afterthoughts.
What actually runs
There is no single "RAGOps" product; it is a discipline assembled from parts. The vector store (Pinecone, Weaviate, Milvus, pgvector, and others) handles incremental upserts and deletes by document id. Orchestration frameworks (LangChain, LlamaIndex) wire the ingestion and serving pipelines. Tracing and evaluation tools (LangSmith, Arize Phoenix, TruLens, Ragas) provide the per-request traces and the quality metrics. Change detection rides on the boring, proven machinery the RAGOps paper lists: webhooks, change data capture, and file watchers. None of it is exotic. The discipline is in wiring the slow data clock to the fast serving clock so they never drift apart, and in being able to prove, for any answer, what went into it.
This part was the operational shape of production RAG: keep the index true to a moving corpus, delete what the source deleted, re-embed when the ruler changes, and trace every answer. It kept pointing at one question it did not answer: how do you actually measure whether retrieval and generation are any good? The paper named the metrics (faithfulness, mean reciprocal rank, hallucination rate) without defining them. The next part does exactly that: how to judge a RAG system, one metric at a time, with a worked example for each.
Sources: RAGOps: Operating and Managing Retrieval-Augmented Generation Pipelines (Xu et al., 2025).
