Start Here
An LLM feature without an eval is a feature whose quality is somebody's opinion. This guide is about replacing the opinion with a number you can defend: how to build the set, grade it, judge with a model without being fooled by it, and turn a score into a ship decision. Start here for the frame, the vocabulary and the running example.
Why evals, and why now
Traditional software has tests because its behaviour is deterministic: the same input gives the same output, so one passing case proves something. An LLM system gives a distribution of outputs, and a distribution cannot be checked by a single example. It can only be sampled and measured. That one fact changes the engineering: the artefact that tells you whether the system works is not a test file but an eval, a labelled set run many times, scored by graders, and summarised with an interval around the number.
Most teams discover this after shipping. A prompt change that looked better on the five examples in the pull request turns out to have broken a category nobody tried. A model upgrade quietly changes how refusals are phrased and a downstream parser starts failing. An agent passes the demo and fails one time in four in production because nobody ran it four times. None of these are exotic. They are what happens when a probabilistic system is managed with deterministic habits.
This guide is for engineers who ship LLM features and need numbers they can defend to a reviewer, a product owner or a regulator. It explains the mechanism behind each part of an eval well enough that you can design one for a system you have not met. If you want the interview-shaped version, AI Engineer Interview Prep 07 and 08 cover that and link back here for depth.
The decisions an eval serves
An eval is not a score. It is a measurement built to make a specific decision, and the decision determines everything about its design: what goes in the set, how it is graded, how big it must be, and what number ends the argument. Four decisions cover almost everything a team does:
| Decision | Question the eval answers | What that demands of the design |
|---|---|---|
| Ship or hold | Is this prompt, model or retrieval change at least as good as what is running, on every slice that matters? | Paired comparison on the same items, per-slice results, a pre-agreed threshold, an interval not a point |
| Choose | Which of these models or configurations should we build on? | The same items for every candidate; cost and latency measured alongside quality |
| Permit | May this feature handle this category of request without a human? | A set dense in that category and its dangerous neighbours; safety checks that fail closed |
| Alert | Has quality in production moved enough that someone should look? | A production sample graded continuously, proxies that track quality, drift detection |
When an eval is argued about, it is almost always because two people are using it for different decisions. A set built to choose a model is a poor set for alerting, and a benchmark score, built to compare models across the industry, serves none of the four for your system. Chapter 01 develops this.
The vocabulary
The field reuses words loosely. These are the meanings this guide holds to:
| Term | Meaning here |
|---|---|
| Item | One input to the system, with whatever is needed to grade the output: a reference answer, a rubric, a set of required facts, or a programmatic check. Also called a case or a task. |
| Eval set | A fixed collection of items, versioned like code, with each item's source and slice recorded. |
| Slice | A labelled subset of items that share a property worth reporting separately: question type, language, customer segment, risk level. |
| Grader | Whatever turns an output into a score: exact match, a program, a model with a rubric, or a person. |
| Judge | A model used as a grader. "LLM-as-judge" is this. |
| Offline eval | Run on the eval set, before a change ships. Repeatable. |
| Online eval | Run on production traffic, after shipping: graded samples, proxy metrics, user feedback. |
| Regression | A slice or item that was passing and now fails, whatever the aggregate did. |
| Trajectory | For an agent, the full sequence of model calls, tool calls and results, not only the final answer. |
| Trial | One attempt at an item. A sampled system answers differently on repeated trials, so agent items are run several times (chapter 06). |
The eval stack a team needs
No single eval does every job. Teams that get this right run four layers, each cheap enough to run at its cadence and each catching what the others miss:
The arrows matter as much as the layers. A production failure becomes an item in the decision set. A human disagreement with the judge becomes a calibration case. A fast check that keeps catching the same thing becomes a change to the prompt or the schema. An eval stack that does not feed itself goes stale within a quarter, and a stale set reports a quality the system no longer has.
At Harbourline, grading 1,000 items with a judge model costs a few dollars and twenty minutes; a wrong coverage decision reaching a customer costs a regulatory conversation. Most teams under-invest in evals because the first number is visible on an invoice and the second is not, until it is.
The running example: Harbourline Insurance
Harbourline is a fictional mid-size property-and-casualty insurer, about two million policies in the US and UK, shared with the other AI Engineering guides. Three systems are under evaluation here:
| System | Volume | Eval assets | Cost of a wrong output |
|---|---|---|---|
| Help-centre RAG assistant | ~13k sessions/day | ~400 labelled items in five slices: limits and deductibles, exclusions, procedures, definitions, identifiers | A wrong deductible: a complaint and a refund |
| Claims-intake agent | ~6k claims/day | ~150 trajectories with reference tool sequences; tools: policy lookup, coverage rules, fraud flags, adjuster routing | A wrong coverage decision: a regulatory issue |
| Customer-service agent | ~40k sessions/day | Read tools and confirmed write tools (address, payment); safety assertions on every trajectory | An unconfirmed write: real harm, hard to undo |
The uneven stakes are the point. The same eval design would be wasteful on the first system and negligent on the third, and the chapters say which choices move with the stakes.
The chapters, in order
Section B · Foundations
| Chapter | What it settles |
|---|---|
| 01 What an Eval Is For | Offline and online, capability and regression, why benchmarks do not transfer, the stack |
| 02 Building the Eval Set | Sampling by slice, synthetic and red-team items, labelling and agreement, size, PII, staleness |
| 03 Graders | What exact, programmatic, model and human grading can each see; rubrics two people score alike |
| 04 LLM-as-Judge | Position, verbosity and self-preference bias with evidence; calibration against humans; when not to |
Section C · Decisions
| Chapter | What it settles |
|---|---|
| 05 The Ship Decision | Paired comparison, intervals and set size, per-slice blockers, non-quality gates, CI wiring |
| 06 Evaluating Agents | Trajectories, tool-call correctness, budgets, partial credit, mocked environments, pass^k |
| 07 Evaluating RAG | Retrieval apart from generation; context relevance, faithfulness, answer relevance, abstention |
| 08 Model and Prompt Changes | What changes silently, regression suites, canaries and shadow traffic, rehearsed rollback |
Section D · In production
| Chapter | What it settles |
|---|---|
| 09 Online Signals and Drift | Proxies and their biases, drift in inputs and models, what to check when offline and online disagree |
Retrieval measurement in its own right lives in Production RAG, chapter 06; chapter 07 here builds on it.
Takeaway
- An LLM system produces a distribution, so its quality is sampled and measured, never proved by one example.
- An eval is built for a decision: ship, choose, permit or alert. The decision fixes the set, the grader, the size and the threshold.
- A score without an interval and without per-slice results is not yet a decision.
- Four layers at four cadences, feeding each other, keep an eval honest; a set that does not take in production failures goes stale within a quarter.
- Match the rigour to the cost of error: the same design can be wasteful on one surface and negligent on another.
Begin with the question every other chapter depends on: what is this eval for? → What an Eval Is For