22 July 2026
Evaluating LLMs in production: building an eval suite that actually matters
How to build useful LLM evals in production: task success, groundedness, latency, cost, golden datasets, LLM judges, and CI.
Public benchmarks measure something different from production
When I deploy an LLM for a client, I look at public benchmarks with interest, then I put them aside quickly. They are useful for understanding broad model capabilities: reasoning, code, math, instruction following. They help narrow the model shortlist. But production asks a more specific question: does this system, with this prompt, these tools, this data, and this interface, complete the real task for this user?
Most failures I see happen in that gap. The model ranks well on a public leaderboard, but the agent calls the wrong tool. The RAG pipeline retrieves a nearby document, but misses the contractual source. The answer sounds fluent, but skips a business constraint. Latency feels fine in a demo and painful inside the daily workflow. A useful eval suite starts with a simple premise: the measured object is the full system, from data to interface.
The four metrics I want to track together
I almost always start with four metric families. The first is task success: did the user get the expected outcome, in the right format, with the right actions performed? For a support assistant, that might mean identifying the right procedure and drafting a reply that can be sent. For an internal agent, it might mean creating the right ticket, filling the required fields, and asking for approval at the right moment.
The second is groundedness: is the answer actually supported by the available sources? The third is performance: p50 latency, p95 latency, time by step, timeouts, retries. The fourth is cost: tokens, tool calls, reranking, judges, cache hit rate. These dimensions have to be read together. A very grounded answer that is too slow can miss the product. A fast and cheap answer that degrades task success costs more than it shows on the API invoice.
Build a golden dataset that looks like the field
The heart of the eval suite is a golden dataset. I prefer building it from real traces: user questions, support conversations, tickets, decision examples, human corrections, refusal cases. Then I clean it, anonymize it when needed, and turn each example into a test contract. The dataset should include the request, the allowed context, expected sources, success criteria, and sometimes a reference answer. It should also include boring cases: ambiguous requests, contradictory documents, stale data, different permission levels, and questions where the right behavior is to ask a follow-up question.
A good golden dataset starts small. Fifty to one hundred well-chosen examples are more useful than a thousand rows nobody understands. I classify them by risk family: retrieval, extraction, synthesis, tool action, compliance, tone, cost, latency. After every production incident, I add the smallest example that would have caught it. The dataset becomes operational memory for the product, instead of a forgotten QA archive.
Use an LLM judge with a calibrated scope
LLM judges are convenient because they can evaluate criteria that are hard to encode: complete answer, relevant citation, business instruction followed, faithful summary. I use them with a human-calibrated scope. An LLM judge inherits biases, varies with its prompt, sometimes rewards longer answers, and can confuse persuasive style with truth. It has to be treated as an imperfect measurement tool, calibrated against human examples.
My method is pragmatic. I first code deterministic assertions: valid JSON, required fields present, mandatory source, expected refusal, latency threshold, token budget, correct tool call. Then I add LLM judges for semantic dimensions, with a simple rubric and annotated examples. I keep a human-reviewed sample to measure agreement between judge and expert. When agreement drops, I adjust the judge prompt or narrow its scope. The judge helps evaluation scale, while the team keeps responsibility for defining quality.
Run evals like regression tests
An eval suite that lives in a notebook rarely protects production for long. I want it in the development cycle: locally on a fast subset, in CI on critical cases, then as a scheduled job on a broader set. Every important change should produce a readable diff: success rate, groundedness, latency, cost, improved examples, degraded examples. The goal is to make every tradeoff explicit before deployment.
I also like versioning everything that affects the result: model, system prompt, tool prompt, index, chunking strategy, retrieval parameters, output schema, and orchestration code. When a score moves, the team should be able to explain what changed. In CI, I only block critical regressions: a clear drop on core tasks, excessive cost increase, groundedness below threshold, or a violation of a product safety behavior. For everything else, I leave a report that helps the team decide.
A silent regression the evals caught
A concrete example: on an internal assistant, we replaced a model with a faster and cheaper version. Manual tests looked good. Answers were shorter, cleaner, and latency dropped. The eval suite still flagged a drop on questions involving recent policies. The issue came from a detail: the new model followed the concision instruction better, but often cited the first retrieved source instead of reconciling two documents when a policy had recently been updated.
This regression could have lived for weeks. It kept the interface healthy, avoided server errors, and gave plausible answers. The fix was straightforward: adjust the synthesis prompt to require date comparison, change reranking to favor active versions, and add ten contradictory-policy cases to the golden dataset. Cost per answer increased slightly, but the rate of answers grounded in the right version recovered. That is exactly the kind of tradeoff I want to see before users discover it.
The eval suite that actually matters
A good eval suite is an operating system for quality. It connects real examples to measurable criteria, separates quality dimensions, exposes cost, and turns incidents into regression tests. It also makes conversations with business teams clearer: here is what the system can do, here is what is improving, here is what costs more, and here are the cases where human validation still has value.
My advice is to start small and make the loop mandatory. Take the twenty tasks that matter most, write the success criteria, add expected sources, measure latency and cost, then replay that set on every change. From there, the suite grows with the product. The best LLM systems I have seen are the ones whose team can measure, explain, and improve quality week after week.