You Cannot Improve What You Cannot Measure
Operating an LLM system means making a fuzzy, non-deterministic thing measurable.
LLM systems are hard to operate because outputs are open-ended and non-deterministic. LLM ops and eval is the discipline that makes them measurable: evals as tests, tracing, regression gates, and a flywheel that turns production failures into your next test set.
Software teams catch bugs two ways. Unit tests run before you ship, over a fixed set of cases you already know about. Monitoring watches real traffic after you ship, catching the surprises the tests never imagined. And every bug that slips through becomes a new test, so it can never slip through the same way twice. Evaluating an LLM system works exactly like that, and this piece is about the two halves and the loop between them.
Most software has a clear notion of correct: the function returns the right value or it does not. LLM systems do not. Outputs are open-ended and non-deterministic, the same input can produce different results, and "good" is often a judgment call. That is what makes them hard to operate, and it is the whole reason LLM ops and eval exists as a discipline. You cannot eyeball your way to reliability at scale. You have to make the fuzzy thing measurable.
The first move is to turn "is it good" into concrete, graded cases. That splits naturally into two kinds of evaluation with two different jobs.
Offline eval runs before release. It is a fixed, curated dataset you grade against, cheap and repeatable enough to run in CI on every change, and its job is to gate the release. It catches the regressions you already know about. Its weakness is that it is blind to anything not in the set.
Online eval runs in production. It samples and traces real traffic, watches metrics and human feedback, and alerts on drift. Its job is to catch the unknown: the inputs you never imagined and the slow degradation you would otherwise miss. Its weakness is that it is noisy and always after the fact.
You need both, and the reason is that they cover each other's blind spots. Offline eval stops old bugs from coming back. Online eval finds the new ones. And the two connect into a loop that is the real engine of improvement.
This flywheel is the point. Every failure that slips into production should not just be fixed; it should become a case in the dataset that you can never regress on again. Over time the dataset stops being a synthetic benchmark and becomes a memory of every way your system has actually broken. That is what makes a fuzzy system get more trustworthy instead of drifting.
Step through one turn of the loop to see how a production surprise becomes a permanent test:
Two cautions from doing this in practice. First, using a model to grade other models' outputs is useful but it is not ground truth; anchor on real signals wherever you can, whether a tool actually returned that value or a test actually passed, and treat model-graded scores as a helpful proxy rather than a verdict. Second, you cannot debug what you cannot see. Tracing every step, every model call, every tool result, is not optional; it is the difference between "the agent did something weird" and knowing exactly which step went wrong.
This is also where a lot of my own work lives. AgentTrustCI is offline eval for agent behavior: it turns "only approved tools, grounded answers, expected sequence" into regression tests that gate the build, so a behavior you fixed stays fixed. ToolContractGate is closer to the online side: it checks every tool call against a versioned contract at runtime and refuses the ones that have drifted. Neither improves the model. Both make a non-deterministic system operable.
The uncomfortable truth is that the ground keeps moving under you: the model gets new versions, your prompts change, your tools change. Eval is not a one-time score you hit and forget. It is the instrument that keeps a moving system honest, and the flywheel is how you make each failure the last time you see it.
