Building the Eval Set

Where items come from

Harbourline's help-centre set has about 400 labelled items. Where they came from decides what the set can say. Four sources cover almost everything, and a good set uses all of them because each fails differently.

Production logs are the base. They are the only source that reflects what customers actually ask, in their phrasing, with their typos and missing context. The cost is that logs are dominated by easy, common questions, and that they contain personal data (see PII and consent). Items drawn from logs also arrive without labels; someone has to decide what the right answer was.

Hard cases from support tickets and escalations are the second source. Every conversation that ended with a human agent, a complaint or a refund is a case where the system, or the process around it, failed once. They are rare in the logs and valuable in the set. Tickets also carry a ready-made label, since a human wrote the eventual answer, though that answer needs checking against the policy wording before it becomes ground truth.

Synthetic items cover what logs do not contain: rare, dangerous or not-yet-launched cases. Harbourline needs exclusion questions for a product it launches next quarter, a question phrased as a prompt-injection attempt, a claimant who gives a policy number in the wrong format. A model can draft many such items quickly from policy documents and seed examples. Filter the drafts before they count. The Hugging Face RAG-evaluation cookbook has a second model score each generated question for groundedness in the source chunk, usefulness to the intended reader and whether it stands alone without the chunk, and keeps only questions that score at least 4 of 5 on all three; about half the drafts fail, which is a fair planning figure. The risks are real, though. Model-written questions are more fluent, more uniform and more answerable than customer questions; they tend to share the writer's blind spots, so a system built on a related model finds them easier than real traffic; and a synthetic item whose reference answer was also written by a model carries any error the model made. Use synthetic items to widen coverage, label them as such in a source field, and report results with and without them. Never let them make up the bulk of a set you will quote as a quality estimate.

Red-team items come from people trying to break the system: adversarial phrasing, requests to bypass confirmation, attempts to extract another customer's data. They belong to the permit decision more than the ship decision. They are few and are not a sample of anything, so they are reported separately as a pass/fail list, never folded into a percentage.

Whatever the source, keep the set balanced on each behaviour it tests. For every item where the assistant must refuse, abstain or ask for confirmation, include an item where it must not. A set with only the first kind gets the prompt tuned until the assistant refuses everything, and the set calls that a pass.

Sampling by slice, not at random

A simple random sample of 400 sessions from the logs mirrors production, which is the problem. The help-centre traffic is dominated by procedures and limits-and-deductibles questions. Exclusion questions are a minority of sessions but a large share of the harm when wrong, and identifier lookups (policy numbers, claim references) are rarer still. A random 400 might hold only a couple of dozen of either, too few to say anything about them. Stratified sampling fixes the allocation: decide how many items each slice needs to be measured, then sample that many from each slice's population.

The cost is that the raw pass rate of the set no longer equals production's pass rate. A set with 110 exclusion items out of 400 over-represents exclusions relative to the traffic, so a naive average over-weights them. Record a weight on each item, the slice's production population divided by its items in the set, and use it whenever you want a number that speaks for production traffic. Report per-slice results as they are, unweighted, because that is where decisions are made (chapter 05).

This is the code Harbourline's team uses to build the sample. It runs as written with the standard library alone; the log here is synthetic, so the slice mix is an illustration, not Harbourline's real traffic.

import random
from collections import defaultdict

def stratified_sample(rows, slice_key, quotas, seed=7):
    """rows: list of dicts. quotas: {slice: n_items}.
    Returns items with a weight = slice_population / slice_sample, so
    aggregates can be re-weighted to the production mix."""
    rng = random.Random(seed)
    by_slice = defaultdict(list)
    for r in rows:
        by_slice[r[slice_key]].append(r)
    out = []
    for name, n in quotas.items():
        pool = by_slice[name]
        take = min(n, len(pool))
        for r in rng.sample(pool, take):
            out.append({**r, "weight": len(pool) / take})
    return out

def cohens_kappa(a, b):
    n = len(a)
    labels = set(a) | set(b)
    po = sum(x == y for x, y in zip(a, b)) / n
    pe = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (po - pe) / (1 - pe)

if __name__ == "__main__":
    mix = {"limits_deductibles": 0.34, "procedures": 0.30, "definitions": 0.16,
           "exclusions": 0.14, "identifiers": 0.06}
    rng = random.Random(1)
    log = [{"session_id": i, "slice": rng.choices(list(mix), list(mix.values()))[0]}
           for i in range(40_000)]
    quotas = {"limits_deductibles": 90, "procedures": 70, "definitions": 50,
              "exclusions": 110, "identifiers": 80}
    items = stratified_sample(log, "slice", quotas)
    print(len(items), "items")
    for s in quotas:
        k = [i for i in items if i["slice"] == s]
        print(f"{s:20s} n={len(k):3d} weight={k[0]['weight']:.0f}")

Run on the synthetic log, it returns 400 items. Exclusions get 110 items at a weight of about 52; identifiers get 80 at a weight of about 30; procedures get 70 at about 173. An identifier item therefore stands for far fewer production sessions than a procedures item, and the weights make that explicit. Slices come from a classifier or from rules over the question; if the classifier is itself an LLM, audit its slice labels on a small sample, because a misassigned item silently fills the wrong slice.

The allocation need not be proportional to harm, but it should be chosen on purpose. A rule of thumb: give each slice enough items to bound its error rate to a width you can act on (see How big is big enough), then add more to the slices where an error is costly.

Labelling and agreement

An item without a label is a question. The label is what the grader will compare against, and every mistake in it becomes a mistake in the metric, in either direction. For a help-centre item the label might be the correct deductible, the policy version it applies to, and the document that supports it. For a claims trajectory it is the reference tool sequence. Writing these down needs a guideline: the definition of correct for each slice, how to treat a partly right answer, what to do when the policy is ambiguous, and when an item should be discarded as unanswerable.

Have at least two annotators label a subset, and measure how often they agree. Raw agreement is misleading, because two people agree by chance more often when one label dominates. Cohen's kappa corrects for that: it is (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is the agreement expected from each rater's label frequencies alone. Suppose two annotators judge 100 system answers as ok or bad. Both say ok on 85. A says ok and B bad on 5; the reverse on 5; both say bad on 5. Observed agreement is 90%. Each rater says ok 90% of the time, so chance agreement is 0.9 × 0.9 + 0.1 × 0.1 = 0.82, and kappa is (0.90 − 0.82) / 0.18 = 0.44. That is moderate agreement on the commonly quoted scale, and the 90% headline hides it. Krippendorff's alpha extends the same idea to more than two raters, to missing labels, and to ordinal or numeric scales; use it when raters see different subsets.

Treat the commonly quoted bands (below 0.2 slight, 0.41 to 0.6 moderate, 0.61 to 0.8 substantial, above 0.8 almost perfect) as a convenience, not a standard. Landis and Koch offered them from judgment, and kappa also moves with how common each label is, so compare it across slices with that in mind.

What low agreement means for the metric is the part teams skip. If two careful people disagree on a slice, the slice has no single right answer under the current guideline, and no grader can be scored above the humans' own agreement against that label. Either the guideline is vague and needs sharpening, or the question is ambiguous and the item should be removed, or the system should abstain on it. Disagreement on exclusion questions is often informative: the clause really is ambiguous, and the right behaviour is to hand off to a human. Resolve disagreements by discussion between the annotators and a domain owner, record both labels, and keep the adjudicated one as ground truth.

Model pre-labelling saves time. A strong model drafts the label and rationale, and a person confirms or corrects. The risk is anchoring: people accept a fluent suggestion, so the human audit adds less than it looks. Counter it by auditing a random sample of the model's labels blind, with annotators writing their own answer before seeing the draft, and by tracking how often they overturn it. If a slice's overturn rate is high, stop pre-labelling there. If the labelling model is the same one you later use as a judge, a shared blind spot would appear in both places and never show as disagreement (chapter 04 returns to this).

How big is big enough

A pass rate measured on n items has sampling error. For a true pass rate p, the 95% interval half-width is roughly 1.96 × √(p(1 − p) / n). At p = 0.85 that is 9.9 points for 50 items, 7.8 points for 80 and 3.5 points for 400. So a 50-item set is a smoke test: it will notice a system that is broken and nothing finer. A change that moves the true rate by four points is invisible in the noise. None of this argues against starting small. Anthropic's guidance for agent evals is to begin with 20 to 50 items drawn from real failures and grow from there; the question here is what a set of a given size can decide, not when to start one.

Two points make this less grim and then worse. First, comparing two versions on the same items is more precise than these single-set intervals suggest, because item difficulty cancels in a paired comparison. Chapter 05 works the numbers for how many items a given difference needs. Second, the interval applies per slice, not just to the whole set. A 400-item set split five ways leaves under 100 items for any slice, and its per-slice intervals are two to three times wider than the overall one. The exclusions slice, where the cost of error is highest, can be given more than its share. Put the largest allocations where the decision is most expensive to get wrong.

When a set is too small to answer, you have three options: collect more items, accept a wider interval and set the threshold accordingly, or change the question. A permit decision that must show an error rate below 1% needs about 300 error-free items just to bound it at 1% (the "rule of three": zero failures in n gives an upper limit near 3/n), and more if you want to support a stricter claim. That sample size is rarely something labelling effort alone can reach for rare categories, which is where synthetic and red-team items earn their place.

PII and consent

Production logs at an insurer contain names, addresses, policy numbers, claim details and sometimes health information. Copying them into an eval set spreads that data into repositories, labelling tools, vendor APIs when you send items to a hosted judge, and the prompts of future engineers. Three practices cover most of the risk. Redact or replace personal data before items enter the set, and keep the replacement consistent so that "the policy number in the question matches the one in the document" still holds. Check that the terms under which customers' data was collected permit this use, and ask Harbourline's privacy team rather than inferring it; the answer differs between the US and the UK, and by data type. Restrict access to the unredacted source, and log who exported what.

Redaction has a cost. Replacing a real policy number with a fabricated one can change what the retrieval system does with it, and removing a name can change how an intent classifier reads the question. Run a sample of redacted items through the system against their originals in a controlled environment and compare. Where the redaction changes behaviour, prefer synthetic rewriting of the whole item over find-and-replace, and mark the item as rewritten.

Held-out and development sets

If you tune a prompt until the eval set passes, the set stops being an estimate of anything. This is ordinary overfitting, and it works with only a few dozen tuning cycles on a few hundred items. The remedy is the usual one: split the set. A development set is open to everyone; engineers read its failures, fix them and rerun. A held-out set is touched only for decisions, ideally by a person or pipeline different from the one tuning the prompt, and its item text stays out of prompts, repositories the model can read, and any hosted tool that may retain data. A reasonable split for a set of 400 is roughly 100 to 150 development items and the rest held out, but check that the held-out part still has enough items in each slice to be useful (How big is big enough).

The held-out set degrades through use. Each time you look at its failures and respond, a little information leaks back. Keep a count of how many decisions each held-out item has informed. When the count is high, or the product has changed, retire items and draw fresh ones from the logs. In contrast, if your set is small and your team is two people, a split may leave too few items for either half; then keep one set, rotate fresh production items in monthly as a rolling check, and say plainly that the number is optimistic.

Refreshing and versioning

A set is a snapshot of a product at a moment. Harbourline adds a flood-cover rider, changes the claims form, retires a plan. Items whose correct answer changed are now wrong labels, and the set under-represents the new product. A quarterly review does three things: re-verify labels against current policy documents; add items from the last quarter's production failures and new products; and retire items that no longer describe the product. Record a reason for each change.

Version the set like code. Store it in a repository as files, with an id, slice, source (log, ticket, synthetic, red-team), the label and its annotators, the guideline version, and the date of the last verification. Tag every release of the set, and record the set version and the grader version in every eval report. Without that, "accuracy went from 84% to 87%" may mean the system improved or may mean the set changed. When a set changes, rerun the previous system version on the new set so comparisons stay paired; never compare a number from the old set with a number from the new one.

✓ Check yourself

Answer from memory before opening each one.

1. A colleague draws 400 sessions uniformly at random from the help-centre logs and reports 91% correct. The exclusions slice is 14% of traffic. What is wrong with using this set to decide whether to ship an exclusions-focused prompt change?

   Show answer

   The set holds about 56 exclusion items, which gives a 95% interval of about 7.5 points either side at a 91% pass rate, too wide to see a change of a few points. Stratify so the slice that the change targets has enough items, and weight items if you need a production-level number. See Sampling by slice.

2. Two annotators agree on 94% of 200 labels, and the reviewer proposes calling the labelling reliable. Name what you need to check before agreeing, and one pattern in the data that would make 94% look much worse.

   Show answer

   Compute kappa or alpha, not raw agreement. If one label (say "correct") is 95% of cases, chance alone yields about 90% agreement and kappa can be low even at 94%. Check agreement per slice too. See Labelling and agreement.

3. Kappa on the exclusions slice is 0.35 while procedures reaches 0.85. The team proposes a stronger judge model to "fix" the exclusions metric. Why will this not work, and what do you do?

   Show answer

   Humans disagree on exclusions, so there is no stable label for a judge to match; a better judge only reproduces one annotator's view more faithfully. Sharpen the guideline, adjudicate the disputed items, drop truly ambiguous ones, or treat "defer to a human" as the correct behaviour for them. See Labelling and agreement.

4. Your team has a 400-item set and two engineers who tune prompts daily against it. After a month, the pass rate has risen from 82% to 93%, but complaints have not fallen. List the likely causes in order of what you would check first.

   Show answer

   First, overfitting: engineers have fitted the prompt to visible items, so check pass rate on a held-out or freshly drawn set. Second, a set that no longer matches traffic (new products, wording shifts), so compare slice mix and recent failures. Third, labels or a grader that miss the failures customers complain about. See Held-out and development sets and Refreshing and versioning.

5. You must show that the claims agent's rate of unconfirmed writes is under 1% before launch, and you have only 100 labelled trajectories. What can you honestly claim, and what are your options?

   Show answer

   Even with zero failures in 100, the upper 95% bound is about 3%, so you cannot support "under 1%". Collect around 300 or more error-free trajectories, generate synthetic and red-team trajectories aimed at that failure, or enforce the property with a deterministic gate instead of measuring it statistically. See How big is big enough.

Takeaway

• Use all four sources: logs for realism, tickets for hard cases, synthetic items for coverage, red-team items for attacks. Label the source of every item and report results with and without synthetic ones.
• Sample by slice with chosen quotas and record a weight on each item, so per-slice results stay unweighted and any production-level number is re-weighted.
• Raw agreement overstates label quality; kappa or alpha corrects for chance. Low agreement on a slice means the slice has no stable ground truth, which a better grader cannot repair.
• Fifty items is a smoke test. A single pass rate near 85% needs about 400 items for a 3.5-point half-width, and each slice needs its own count.
• Strip or replace PII before items enter the set, and confirm with the privacy owner that the use is permitted.
• Split development from held-out items, count how often held-out items inform decisions, and retire them when overused.
• Version the set like code, and compare versions of the system only on the same version of the set.

With items and labels in hand, you need something to score outputs against them. → Graders

Further reading

• Define success criteria and build evaluations — Anthropic documentation. Good for the advice to mirror the real task distribution, include edge cases such as ambiguous inputs, and favour more automatically graded items over fewer hand-graded ones.
• Cohen's kappa — reference article. Good for the formula, the commonly quoted interpretation bands and the criticism of them.
• Investigating Data Contamination in Modern Benchmarks for Large Language Models — Deng et al., 2023. Good for why held-out items must stay out of training and prompt text.
• Hugging Face Cookbook: RAG Evaluation — Aymeric Roucher. A worked synthetic eval set with critique filters, a 1 to 5 correctness judge with a per-level rubric, and sweeps over chunk size, embeddings and reranking.
• Demystifying evals for AI agents — Anthropic, January 2026. Start with 20 to 50 items from real failures, write items two experts would grade the same way, and balance each behaviour with items where it must not fire.

For the same material framed for an interview, see AI Engineer Interview Prep 07; for the retrieval side of labelling, Production RAG 06.