What an Eval Is For

Start from the decision

A team at Harbourline wants to change the help-centre assistant's system prompt so it answers exclusion questions more carefully. The first question is not "what metric?" It is "what will we do differently depending on the number?" If the answer is "merge the prompt or not", the eval is a ship decision, and it needs the same items run on both versions, a result per slice, and a threshold agreed before anyone sees the result. If the answer is "tell the product owner whether the assistant is good enough to stop showing a human-handoff button", it is a permit decision, and it needs a different set: dense in the dangerous category, graded so that a confident wrong answer counts far more than a refusal.

The reason to be strict about this is mechanical. An eval is a sample of items, a grader that turns outputs into scores, and a rule that turns scores into an action. Each of the three is chosen by the action. The sample has to over-represent the cases whose failure changes the action. The grader has to see whatever distinguishes a correct output from an incorrect one for that action. The rule has to say, in advance, what result triggers what. Take the action away and you are left with a number that anyone can argue means anything, which is the usual state of a team that says "we have evals" and cannot say what they have decided with them.

Four decisions cover nearly everything: ship or hold a change, choose between candidates, permit a feature to act without a human, and alert on a production shift. They differ in how often they happen, how much error each tolerates, and whether the person asking can wait. A ship decision happens on every prompt edit and must finish in minutes. A choose decision happens once a quarter and can take a day. A permit decision happens once per feature but carries the heaviest consequences. An alert decision happens continuously and has to work on traffic nobody labelled.

Four axes of an eval

Evals are classified on four axes at once. Most confusion comes from using one word, such as "regression test" or "online eval", as if it picked a point on all four.

Offline and online

An offline eval runs on a fixed set before a change ships. It is repeatable: the same items and the same grader give the same answer tomorrow, which is what lets you compare two versions. Its weakness is that the items are a snapshot. An online eval runs on live traffic after (or alongside) shipping: graded samples, proxy signals such as escalations and re-asks, and user feedback. It sees the real distribution, including inputs nobody thought to write down, but it is not repeatable. Two versions never see the same traffic unless you split it deliberately, and the labels arrive late or never. The two are complements. Offline evals decide whether a change may reach users; online evals report whether the system you shipped is behaving as the offline numbers promised. Chapter 09 covers what to check when they disagree.

Capability and regression

A capability eval asks "can the system do this at all?" It is built from hard items the current system mostly fails, and its job is to show progress. A regression eval asks "did anything that used to work stop working?" It is built from items the system passes, and its job is to stay green. They need different sets and different thresholds. A capability set that the system passes 95% of the time has stopped measuring capability. A regression set is only useful if it is dense in the things you would hate to break: at Harbourline, the identifier slice (policy numbers, claim references, plan codes) is cheap to get wrong in a prompt change and costly when it happens. The two sets age differently as well. Capability items migrate into the regression set as the system learns them.

Unit-like and end-to-end

A unit-like eval checks one narrow property with a deterministic check: the response parses as the expected schema, every cited document id exists, a refund amount is within the plan limit, no write tool was called without a confirmation turn. These are cheap, exact, and run in seconds. An end-to-end eval runs the whole system, including retrieval, tools and generation, on a realistic input and judges the outcome. It is slower and noisier but it is the only one that tells you whether the parts compose. The usual mistake is to have only the second kind, so that a broken citation resolver shows up as a vague drop in a judge score two weeks later, rather than as a red check on the commit that broke it.

Per-item and aggregate

An aggregate says "86% pass". A per-item view says which items failed, and how the failures cluster. Decisions are made on aggregates and per-slice aggregates; diagnosis is done per item. Keep both. A reported aggregate that cannot be traced back to the item-level rows that produced it will not survive the first sceptical reviewer.

Why benchmark scores do not transfer

Public benchmarks answer a real question: which general-purpose model is broadly stronger, as measured on a shared test? That is the "choose" decision at the industry level. It does not answer the choose decision for Harbourline, and it answers none of the other three, for three separate reasons.

Distribution. A benchmark is a sample from a distribution someone else cared about: exam questions, coding puzzles, trivia, short-form maths. Harbourline's help-centre traffic is about deductibles on a specific policy form, whether a clause excludes flood damage, how to file a claim from a phone. A model's rank on an academic knowledge test says little about whether it reads a UK home policy schedule correctly, and the error you care about, quoting the wrong limit with confidence, is not what the benchmark counts. Two models a few points apart on a leaderboard can be in either order on your slice, and only your own items can tell you which.

Contamination. Test items that appear in training data inflate scores. Deng et al. (2023) probed this directly: they masked a wrong answer option in multiple-choice test items and asked models to fill it in, a task that has no business succeeding unless the model has seen the item. They reported exact-match rates of 52% for ChatGPT and 57% for GPT-4 on MMLU items under that test, which is memorisation, not knowledge. Your own labelled set is less exposed than a famous public one, but it is not immune. If you paste eval items into a prompt for a hosted model, or publish them, they can leak the same way. Keep the held-out part of the set out of any text that leaves the building. Chapter 02 covers the split.

Format. Benchmarks usually grade multiple choice, short answers or code that runs. Harbourline's assistant writes a paragraph, cites policy documents, and sometimes should decline to answer. A score on a format where the correct output is one letter does not predict behaviour on a format where the correct output is a paragraph with a number in it and a citation that must resolve. It also hides whatever the benchmark does not measure at all: latency, cost per answer, refusal calibration, behaviour under the system prompt you actually use.

The eval stack

No single eval serves four decisions. Teams that run this well keep four layers, each cheap enough to run at its own cadence, and each catching what the cheaper layers above it cannot. The start page sketches them; this is what each one is for and what it costs.

Layer 1 exists because many failures are not matters of quality. A citation that points at a deleted help article, a response that fails the JSON schema the front end expects, a write tool called without a confirmation turn: each is a deterministic fact about one output. Check them on every commit, with no model in the loop, and treat any violation as a build failure. These checks are the cheapest in the stack and the ones teams skip most often, because they feel too simple to be evals. Chapter 03 lists what a programmatic check can see.

Layer 2 is the one this guide spends most of its time on. It is where a change is accepted or rejected, so its numbers need statistics behind them (chapter 05). Its size is a trade between precision and the speed of the loop. Too small and it cannot see a four-point regression in one slice. Too large and engineers stop running it before small edits, and then the layer is decorative.

Layer 3 catches the cases layer 2 cannot imagine. The assistant handles about 13k sessions a day; sampling 1% gives 130 a day, enough to see a shift within a week even at modest per-day noise, and cheap enough to grade with a judge and rules. It is also where you meet the input distribution as it is, including new products, seasonal spikes and phrasing nobody wrote an item for.

Layer 4 keeps the other layers honest. People read a stratified sample, and every case where a person and the judge disagree is either a bug in the rubric, a bug in the judge, or a new kind of item. Without this layer a judge drifts away from what users and regulators would call correct, and nobody notices because its score stays steady. The cadence is slow because the layer is expensive; it should still not be skipped.

The arrows are the point. A production failure becomes an item in the decision set. A repeated failure in the decision set becomes a deterministic check. A human disagreement becomes a rubric correction. A stack without those arrows is four disconnected reports.

Cost of the eval, cost of the error

Eval spending should follow the cost of being wrong, not the cost of being measured. The two are rarely compared because one arrives on an invoice and the other arrives as a complaint, an escalation or a letter. Harbourline's three systems make the contrast plain.

The arithmetic is lopsided. Grading 1,000 items with a judge model costs a few dollars and about twenty minutes (an estimate; price it with your own model and token counts). Having two people label those 1,000 items costs days of salaried time. A regulatory finding about a coverage decision costs far more than either, and the 6k claims a day mean a defect that affects 0.5% of claims touches about thirty people a day before anyone notices. Because the cost of labelling people is the largest eval line item, use them where they matter most: the permit decisions and the calibration of the judge, not every ship decision.

The principle flips when the error is cheap and the volume is small. A weekend prototype serving ten internal users does not need a 400-item stratified set; twenty hand-picked items you read yourself on every change are a better use of a day. The test is whether a person can still read every output. When they can, read them. When they cannot, you need the stack.

✓ Check yourself

Answer from memory before opening each one.

1. Two engineers report different scores for the same prompt change on the same model: one says "88% correct, good enough", the other says "no improvement". Both are looking at the same run. What is most likely going on, and what do you ask them?

   Show answer

   They are answering different questions: one is reading an absolute level for a launch ("permit") decision, the other a before-and-after difference for a ship decision. Ask what action each number triggers, then check the set and grader suit that action. See Start from the decision.

2. The claims agent's regression set has been green for six months, with 98% of items passing. Is that good news? What do you do with the set?

   Show answer

   Not necessarily. A regression set is supposed to stay green, so the real question is whether it still covers what you would hate to break and whether production failures have been added. A set that never changes and never fails has stopped sampling the product. Add failures from layer 3 and layer 4 and re-weight the slices toward current traffic. See Four axes and The eval stack.

3. A vendor shows that a model ranks first on a well-known knowledge benchmark and proposes you switch the help-centre assistant to it. Give two independent reasons the rank may not hold on Harbourline traffic, and say what you would run instead.

   Show answer

   The task distribution differs (policy-specific questions, long grounded answers with citations); the items may be contaminated, as masked-option tests show for some public sets; the output format differs. Run the shortlisted models on your decision set, by slice, paired, with cost and latency alongside. See Why benchmark scores do not transfer.

4. A citation resolver silently starts returning 404 for 8% of links after a content migration. Which layer should have caught it, and what did you pay for the layer you caught it in instead?

   Show answer

   Layer 1, on the commit or deploy that changed the content. If it surfaces only as a slow decline in a judge score in layer 2 or 3, you pay in days of delay and in the analysis needed to trace a vague quality drop back to a mechanical cause. See The eval stack.

5. A manager proposes cutting human review in layer 4 to save money, since the judge "agrees with itself every week". Argue for or against, and say what would change your answer.

   Show answer

   Against, for any system where a wrong output is costly: self-agreement says the judge is stable, not that it is right. Human review is what detects a judge drifting from the standard users and regulators apply. It could shrink for a low-stakes surface (the help-centre assistant) if earlier calibration showed high agreement and a small stratified audit continued. For the claims agent it should not. See Cost of the eval.

Takeaway

• An eval is a sample, a grader and a rule that maps scores to an action; name the action first, because it fixes the other three.
• Evals vary on four axes at once: offline or online, capability or regression, unit-like or end-to-end, per-item or aggregate. Say where yours sits on each.
• Public benchmark scores differ from your task in distribution, may be contaminated, and grade a different output format. Use them to shortlist, never to decide.
• A team needs four layers: deterministic checks per commit, a paired decision set per change, a graded production sample, and periodic human review.
• The layers must feed each other: production failures become items, repeated failures become checks, and human disagreements correct the rubric.
• Spend on evals in proportion to the cost of an error, and use human labour where a wrong judge would do the most harm.

The decision set is the layer that does the most work, and it is only as good as its items and labels. → Building the Eval Set

Further reading

• Define success criteria and build evaluations — Anthropic documentation. Good for the principle of mirroring the real task distribution, including edge cases, and for worked examples of matching grader to criterion.
• Investigating Data Contamination in Modern Benchmarks for Large Language Models — Deng, Zhao, Tang, Gerstein, Cohan, 2023 (NAACL 2024). Good for how contamination is probed and why a high public score can reflect memorisation.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023. Good as the origin of the model-grading practice that layers 2 and 3 rely on; chapter 04 uses it in depth.
• Demystifying evals for AI agents — Anthropic, January 2026. The vocabulary this guide shares (task, trial, grader, transcript, outcome), the capability-versus-regression split, and pass@k against pass^k.

For the interview framing of the same material, see AI Engineer Interview Prep 07 and 08; for retrieval measurement in its own right, Production RAG 06.