How to Actually Evaluate an LLM System

Picture grading a stack of exams. Some answers you check in a second because they are multiple choice: the box is either right or it is not. Some are short essays that need a rubric to score fairly. And a few are open-ended enough that only a person reading carefully can really judge them. Evaluating an LLM system is that same job, and the whole game is choosing the right grader for each kind of answer.

If you build with LLMs for any length of time, you hit the same wall: the thing works in the demo, and you have no idea whether it works in general. Normal software has an obvious oracle. The function returns 4 or it does not. An LLM system has no such oracle. The output is open-ended, the same input can produce different text, and "good" is often a matter of judgment. Eval is how you get out of that bind. It is the practice of turning "seems fine" into a number you can track, defend, and improve.

Underneath the jargon, an eval is three things: a dataset of cases, a grader that scores each case, and a number you get by aggregating the scores. That is it. Everything hard about eval is hidden inside those three words, and this guide is about what actually goes in each.

The one idea: the grader is the eval

People ask "what type of eval should I use," but the real question is how you are going to grade. The grader is where all the difficulty and all the choices live. There are four broad families, and they trade off along one axis: cheap and strict on one end, rich and human on the other.

• Exact and heuristic checks are code: string equals, a regex, a JSON schema, a unit-test-style assertion. They are deterministic, free to run, and unambiguous. Use them wherever the output has structure, a classification label, an extracted field, a piece of valid JSON. Their weakness is that they are blind to meaning. Two correct answers phrased differently will look like a pass and a fail.
• Reference-based metrics compare the output to a gold answer with some similarity measure, from simple overlap to embedding distance. They help when you genuinely have a known right answer to compare against. The trap is believing the score: high overlap with the reference is not the same as being correct, and a right answer worded freshly can score low.
• LLM-as-judge uses a model to grade another model's output against a rubric, either scoring it directly or picking the better of two responses. This is what makes it possible to measure open-ended quality at scale, and in practice it is the workhorse for anything subjective. The thing to never forget is that a judge is a proxy, not ground truth. It has biases, it can be gamed, and it needs to be validated against something real.
• Human review is people grading to a set of guidelines. It is the closest thing to a source of truth for judgment calls, and it is how you calibrate everything else. It is also slow, expensive, and impossible to run on every change, so you spend it where it counts: building the gold set and auditing the judge.

Click through the four to compare what each one buys you:

The practical answer is that real systems mix all four. Strict checks guard the structure, a judge scores the quality, and a trickle of human review keeps the judge honest. Choosing an eval is really choosing the cheapest grader that still captures what you mean by good.

Two more ways to slice it

Beyond how you grade, two other distinctions are worth naming, because people use the word "eval" for all of them.

What you evaluate. You can evaluate a single component in isolation, does the retriever fetch the right documents, does the classifier label correctly, or you can evaluate the whole system end to end, did the user get a good final answer. Component evals are sharper and easier to debug; end-to-end evals are what actually matters to the user. You want both, for the same reason you want unit tests and integration tests. For agents, the end-to-end version also means grading the trajectory, the sequence of steps, not just the final output.

When you evaluate. Offline eval runs before release on a fixed dataset and gates the ship. Online eval runs in production on real traffic and catches what the dataset missed. I have written about that split and the flywheel between them separately; this guide stays on the mechanics that both share.

How to run one, step by step

Here is the whole procedure. The steps are simple to list and the first two are where all the real work is.

1. Collect cases. Gather real inputs, and where you can, the right answer for each. Twenty cases that reflect real usage beat two thousand synthetic ones. This dataset is the single most valuable asset in the whole exercise.
2. Define pass. For each case, decide how it gets graded, using the spectrum above. This is where you are forced to say, concretely, what "good" means. If you cannot write it down, you do not yet know it.
3. Run. Execute the system across every case and capture the outputs, along with the intermediate steps if you can, because you will want them when you debug.
4. Score. Apply each grader to its case and get a pass, a fail, or a number.
5. Aggregate. Roll the scores into one headline figure, usually a pass rate, and then slice it by category. The average is where the truth hides; a single number can look healthy while one whole bucket is quietly broken.
6. Gate and track. Set a threshold, wire the eval into CI so a regression blocks the change, and watch the trend over time. An eval that is not run automatically stops getting run.

Then close the loop. Every failure that reaches production should come back as a new case in the dataset, so the same bug can never regress unnoticed again. Over time the dataset stops being a synthetic benchmark and becomes a record of every way your system has actually broken, which is exactly what makes the number trustworthy.

The failure modes worth knowing

A few things quietly ruin evals, and they are worth watching for.

• A dataset of easy cases. If you build your set from what is convenient to test, your score measures convenience, not quality. Grow it from real failures.
• Trusting the judge blindly. An LLM judge is a measurement instrument, and instruments need calibration. Spot-check it against human labels, and be suspicious when it agrees with you a little too readily.
• One number for everything. A 90 percent pass rate can hide a category sitting at 40. Always report the slices, not just the mean.
• Grading only the final answer. For anything multi-step, a right answer reached by a wrong path is a bug waiting to happen. Grade the trajectory too.

The takeaway

An eval is a dataset, a grader, and a number, and the craft is almost entirely in the first two. Pick the cheapest grader that captures what you actually mean, build the dataset out of real cases rather than convenient ones, slice the result so it cannot lie to you, and wire it into CI so it keeps running. Do that and you convert an unmeasurable, non-deterministic system into something you can steer. Skip it and you are shipping on vibes, and vibes do not survive contact with real users.