The Eval Set and the Ship Decision

The question, and what is being scored

The interviewer says: "You changed the prompt for our help-centre assistant and the new version scores higher. How do you decide whether to ship it?" Variants: "How would you build an eval set for this?" and "How do you know your judge model is right?" All three are the same question.

Harbourline's help-centre assistant answers policy and procedure questions from documents. It handles a share of the roughly 40k chat sessions a day, and a wrong deductible quoted to a customer becomes a complaint. The interviewer is scoring four things:

• You treat a score as an estimate with error, not a fact.
• Your items come from real traffic, sliced by what matters, not from whatever was easy to write.
• You know how your grader fails, and you checked it against humans.
• Your decision rule exists before you see the result, and it includes cost, latency and the worst slice, not only the mean.

Chapter 01 assumed you had a labelled set to measure recall@k. This chapter is where that set comes from. For the shorter cheat-sheet version of the same ideas, see Evals in the compliance guide.

Where the items come from

An eval set predicts production only if it is drawn from production. Anthropic's guidance says to mirror the real task distribution and to include edge cases such as irrelevant or overly long input, harmful input and ambiguous cases. Three sources cover that, and each fills a gap the others leave.

• Sampled production logs, stratified by slice. Do not sample uniformly. A uniform sample of help-centre chats is mostly "how do I download my certificate", and the rare, expensive questions never appear. Sample per slice: topic (deductibles, cancellations, claims process, flood and subsidence), region (US or UK wording differs), and channel. Take enough from each slice to read its result on its own.
• Synthetic items for rare cases. A question about a policy form issued in 2019, or an exclusion that fires in one county, occurs a few times a month. Have a model draft candidates from the source documents, then have a person edit and approve each one. Synthetic items test coverage. They do not tell you the real mix, so never use them to estimate an overall rate.
• Red-team items. Prompt injection in the customer's text, requests to reveal the system prompt, attempts to get a coverage promise. These are written by hand and each has an expected safe behaviour.

Every item stores the input, the context the system saw (so a changed retriever does not invalidate it), a reference answer or a rubric, its slice tags, and where it came from. Labelling follows. Two people label a subset, and you compute agreement beyond chance, usually Cohen's kappa. If agreement is low, the rubric is vague, and no grader will fix that, because the grader inherits the ambiguity. As a working rule, I rewrite the rubric until two people agree on most items before I trust either label.

How many items

The size follows from the size of the difference you need to see. A pass rate measured on n items has a standard error of about √(p(1−p)/n). At p = 0.85 the 95% interval is roughly:

That is the origin of "50 is a smoke test and 300 to 1,000 is a decision set". It is practice, not law. Anthropic's own engineering guidance says 20 to 50 tasks drawn from real failures is a good start, because early changes have large effects. The two statements agree: a small set finds big problems, and a large set is only needed when the changes you are choosing between are small.

Two refinements matter in an interview. First, the interval above is for one system measured alone. You are comparing two systems on the same items, and that comparison is much tighter, as the next sections show. Second, slices cut the effective size. A slice of 40 UK cancellation questions has the ±10-point problem even inside a 500-item set. Decide in advance which slices carry the cost of a wrong answer and size those to be readable.

Graders, and how the judge model fails

Use the cheapest grader that can be right, in this order.

1. Exact or programmatic. Does the answer contain the deductible figure, the right policy section ID, a refusal phrase? Fast, free, deterministic. It fails on paraphrase, so check for the fact, not the wording.
2. Model-graded with a rubric. A judge model scores each answer against a written rubric: "states the correct deductible for the policy in context; does not state any other figure; cites a section." Use binary or small-scale questions, and grade each dimension in its own call, so one weak dimension does not blur the others.
3. Human. The ground truth for calibrating the other two, and the grader for anything where the rubric cannot be written down.

The judge is a model, so it has biases. The paper that named them, Zheng et al. 2023 ("Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"), found three: position bias (in a pairwise comparison, a preference for the first or second answer regardless of content), verbosity bias (longer answers win), and self-enhancement bias (a judge favours text that its own model family wrote). It also reported that a strong judge agreed with human preferences over 80% of the time, about as often as humans agree with each other. Both facts matter. The judge is usable, and it is usable only after you measure it.

Calibration is a short procedure. Take 100 items, have humans label them, run the judge, and compute agreement and the confusion matrix. Then run the known tests. Swap answer order and check the verdict holds. Pad a correct answer with filler and check the score does not rise. Grade with a model from a different family than the one that produced the answers, which Anthropic's docs also recommend. Re-run the calibration whenever the rubric or the judge model changes, because either change moves the scores.

From a score to a ship decision

Write the rule before you run the comparison. A rule written after you see the numbers is a rationalisation. Mine has four parts.

1. Paired comparison on the same items. Score old and new on identical inputs. What you analyse is the per-item difference. Most items are right under both versions, or wrong under both, and they cancel. Only the items that flip carry information, so the interval is far tighter than two separate intervals. Evan Miller's 2024 paper on error bars for evals makes the same case: analyse differences between models directly.
2. An interval on the difference. Bootstrap over items (resample the per-item differences with replacement) and read the 2.5th and 97.5th percentiles. If the work is clustered, such as several questions about one policy, resample the clusters.
3. Per-slice checks with a stated tolerance. A slice that drops beyond its tolerance blocks the change until someone has read the flipped items. With a dozen slices, one will look bad by chance. So a flagged slice triggers reading, not an automatic veto, except on slices where a wrong answer is a regulatory problem. There the tolerance is tighter and the veto is automatic.
4. Cost and latency as gates. A better answer that adds 30% to cost per session or 2 seconds to p95 time to first token is a different decision, taken with different people.

import random

def paired_bootstrap(old, new, n_boot=10_000, seed=7):
    """old, new: 0/1 scores on the SAME items, in the same order."""
    assert len(old) == len(new)
    diffs = [n - o for o, n in zip(old, new)]
    rng = random.Random(seed)
    k = len(diffs)
    means = sorted(sum(rng.choices(diffs, k=k)) / k for _ in range(n_boot))
    return sum(diffs) / k, means[int(0.025 * n_boot)], means[int(0.975 * n_boot)]

# 500 items: 400 right under both, 55 wrong under both,
# 30 the new prompt fixes, 15 the new prompt breaks
old = [1]*400 + [0]*55 + [0]*30 + [1]*15
new = [1]*400 + [0]*55 + [1]*30 + [0]*15
print(paired_bootstrap(old, new))   # about +0.030, interval about [+0.004, +0.056]

Note what the example shows. Of 500 items, 45 flipped. The new prompt gained 3 points, and the paired interval excludes zero by a small margin. Two unpaired intervals at this size would each be about ±3 points wide and would overlap, so you would wrongly call it a tie. For wiring it into CI, the eval runs on every pull request that touches a prompt, model name or retrieval setting. It posts the delta, the interval, the worst slice and the cost and latency change as a comment, and a failed gate blocks the merge.

One case flips all of this. If the change is a pure cost cut, such as a cheaper model, the question is not "is it better" but "is it not worse by more than I can accept". That is a non-inferiority test: ship if the lower bound of the interval stays above a negative margin that you set in advance, say −1 point.

Worked answer: a prompt change at Harbourline

The change tells the help-centre assistant to answer in at most three sentences and always name the policy section. The aim is lower output tokens and fewer complaints about long answers. The decision set has 500 items: 380 sampled from the last 90 days of chat logs across eight slices, 70 synthetic rare-case items, and 50 red-team items. Two people labelled 100 of them with kappa agreement acceptable, and the judge model matched human labels on 91 of those 100.

The overall gain is real and the cost falls. The UK cancellation slice drops by three items out of 40. That is within chance at this size, but UK cancellations carry a notice-period rule, and the three-sentence limit could have cut it off. So the next step is to read the three flipped transcripts. If the answers omit the notice period, I block and change the prompt to allow a longer answer for that topic. If the judge was wrong, I fix the rubric and re-run. If it was noise, I ship with that slice on a watch list in the first week of traffic.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. A teammate runs the old and new prompts on a 50-item set and gets 88% against 84%. They want to ship. What do you ask before agreeing?

   Show answer

   Whether the difference exceeds noise. On 50 items a single pass rate carries an interval of about ±10 points, and a 2-item gap is well inside that. Ask for the paired difference with an interval, and for the number of items that flipped in each direction. See How many items and From score to decision.

2. Two intervals, one for each version, overlap by a wide margin. Why can the paired interval on the difference still exclude zero?

   Show answer

   Both versions find the same items easy and the same items hard, so that variation cancels in the per-item difference. Only the items that flip contribute to the uncertainty. Two separate intervals each include the large item-to-item variation and ignore the correlation. See From score to decision.

3. Your judge model scores answers from the same model family higher than human labellers do. Which bias is this, and what two changes reduce its effect?

   Show answer

   Self-enhancement bias. Grade with a model from a different family than the one that wrote the answers, and calibrate against human labels so you know the size of the gap. A length-controlled rubric helps with the related verbosity bias. See Graders and the judge model.

4. An eval set was built entirely from a uniform sample of last month's chat logs, with 600 items. Name two things it will fail to tell you about a new release.

   Show answer

   It under-represents rare, expensive questions, such as flood exclusions or notice periods, so a regression there hides behind the common cases. It also contains no adversarial input, so injection and refusal behaviour go unmeasured. Stratify by slice and add synthetic and red-team items. See Where the items come from.

5. A cheaper model is proposed for the help-centre assistant. Overall it scores 0.5 points lower, with an interval of [−2.4, +1.4]. Cost per session falls 40%. How do you decide, and what would make you decide the other way?

   Show answer

   This is non-inferiority. Ship if the lower bound stays above the margin you set in advance, for example −1 point. Here the lower bound is −2.4, so the data allows a worse result than the margin and you either gather more items or accept a wider margin knowingly. A 40% saving makes a wider margin tempting, but if the deductible slice is where the loss sits, the answer is no regardless. See From score to decision.

Takeaway

• An eval score is an estimate with an interval, and a decision needs the interval, not the point value.
• Items should be sampled from production by slice, with synthetic items for rare cases and hand-written red-team items, and each item stores its context, label and source.
• Fifty items is a smoke test, and a few hundred to a thousand is a decision set, because the interval on a pass rate narrows with the square root of the size.
• Judge models have position, verbosity and self-enhancement biases, so calibrate against human labels, per slice, and re-calibrate when the rubric or judge changes.
• Compare versions on the same items and analyse the per-item difference, which is much tighter than two separate scores.
• The ship rule is written before the run: interval on the difference, per-slice tolerances with a stricter veto on high-stakes slices, and cost and latency gates.

A final answer is only one thing to grade when the system is an agent. → Evaluating an Agent Beyond Its Final Answer

Further reading

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023. The source for position, verbosity and self-enhancement bias and for judge-to-human agreement.
• Adding Error Bars to Evals — Evan Miller, 2024. Formulas for analysing eval data, comparing two models and planning how many items you need.
• Create strong empirical evaluations — Anthropic docs. Success criteria, task-specific eval design and the advice to grade with a different model.
• Demystifying evals for AI agents — Anthropic Engineering, January 2026. Starting small from real failures and reading transcripts to check your graders.