The Ship Decision

The decision this eval serves

A Harbourline engineer rewrites the help-centre assistant's system prompt to explain exclusions better. Someone has to say yes or no to deploying it, and the answer has to be the same whether the engineer, a reviewer or an auditor asks. The earlier chapters gave you the pieces: a labelled set (chapter 02), graders (chapter 03) and a judge you have checked against people (chapter 04). This chapter turns two sets of per-item scores into a decision you can defend.

The mechanism is statistical, not procedural. Each item gets a score under the old system and under the new one. The question is whether the average difference is real or the product of which items happened to be in the set, and whether the average hides a slice where the change did damage. Three things have to be fixed before you look at results: the items, the comparison method and the thresholds. Any of them chosen afterwards can be bent toward the answer you wanted.

Compare on the same items, item by item

Two systems scored on 400 items give you two accuracies, 80% and 84%. The tempting test treats them as two independent samples of 400 and asks whether 80 and 84 are far apart. They are not independent. Both systems answered the same questions, so the easy items are easy for both and the hard ones hard for both. That shared difficulty is most of the variance in either total, and an unpaired test counts it as noise.

A paired comparison removes it. Look at each item and put it in one of four cells: right under both, wrong under both, fixed by the change, broken by the change. Only the last two carry information. If 36 items are fixed and 20 broken, the net gain is 16 items, or 4 points, and the other 344 items say nothing about the difference. The same table gives the win/loss/tie view that reviewers read quickly: 36 wins, 20 losses, 344 ties.

The size of the gain from pairing depends on how much the systems agree. With these counts the unpaired 95% interval on the 4-point difference is about ±5.3 points, which includes zero. The paired interval is about ±3.6 points, which does not. The data is the same; only the model of it changed. Pairing helps more the more the two systems agree, and a prompt tweak usually changes few answers, so for the changes you test most often it is large.

A bootstrap interval on the difference

The paired difference per item is +1, 0 or -1. The bootstrap resamples those 400 values with replacement, takes the mean each time, and repeats thousands of times. The middle 95% of the resampled means is the interval. It assumes the items are an independent sample of the traffic you care about, and nothing else about the shape of the data, which is why it suits scores that are bounded and lumpy. Miller's Adding Error Bars to Evals (2024) treats model evaluations as experiments in the same way and gives formulas for comparing two models and for planning an evaluation.

import math, random

def paired_report(old, new, n_boot=10_000, seed=7):
    """old, new: lists of 0/1 per item, same order. Returns delta and 95% bootstrap CI."""
    assert len(old) == len(new)
    n = len(old)
    diffs = [b - a for a, b in zip(old, new)]
    delta = sum(diffs) / n
    rng = random.Random(seed)
    boots = sorted(sum(rng.choices(diffs, k=n)) / n for _ in range(n_boot))
    lo, hi = boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot)]
    return delta, (lo, hi), diffs.count(1), diffs.count(-1)

def n_needed(delta, discordant, z_alpha=1.96, z_power=0.84):
    """Items for a paired test; discordant = share of items where old and new differ."""
    return math.ceil((z_alpha + z_power) ** 2 * discordant / delta ** 2)

On the 400-item example this prints a delta of +0.040 with an interval of roughly +0.003 to +0.077, from 36 wins and 20 losses. The lower end sits barely above zero. Read that as "probably better, by an amount we cannot pin down", not "4 points better". If the threshold you set in advance was "lower bound above zero", this change passes by a hair. If it was "at least 2 points", it does not.

Two cautions. Items drawn from the same session or the same policy are not independent, so if your set has 5 questions per policy, resample policies, not items, or the interval comes out too narrow. And if the grader is a model judge, the judge's own noise is extra variance; run it with a fixed seed and temperature 0, or average a few runs, and treat a judge change as a change to the instrument that invalidates old comparisons.

How many items a difference needs

A set that is too small cannot tell you anything, and the failure is silent: you see a delta and an interval so wide that neither verdict is justified. Sample size follows from the smallest difference you would act on, and from how often the two systems disagree. For a paired test at 5% significance and 80% power, the approximate count is (1.96 + 0.84)² × d / δ², where d is the share of discordant items and δ the difference. That is what n_needed computes.

Halving the difference quadruples the items. Harbourline's 400-item help-centre set can reliably see a 5 to 6 point gain on the whole set, and cannot see 2 points at all. That is a fact about the set, and it should be written in the eval's README so that nobody reads "no significant change" as "no change". It is also why chapter 02 treats 50 items as a smoke test: it detects only catastrophic breakage.

Slices are blockers even when the total improves

The Harbourline prompt change scores +6 points on exclusions: 80 items, 5 net items fixed. It scores -4 points on identifiers (policy and claim numbers, which the assistant must copy exactly): 75 items, 2 items fixed and 5 broken. Suppose the other slices barely move, so the aggregate over 400 items is a small gain with an interval that spans zero. A reader of the aggregate alone sees a wash. A reader of the slices sees a trade, and the trade is not symmetric: a mangled policy number sends a customer to the wrong record, and a better exclusions paragraph is a nicety.

Now the statistics get awkward. Seven discordant items on identifiers split 2 to 5, which an exact sign test puts at p = 0.45. You can neither call the regression real nor call it noise. The way out is to decide the rule in advance and to make it asymmetric. For slices where an error is costly, the gate is not "significantly worse" but "shown not to be worse than a margin": the lower bound of the slice's one-sided interval must stay above, say, -3 points. Failing to show safety counts as a block.

That rule has a price, and it is worth stating plainly. With about 10% of items differing, showing non-inferiority within 5 points needs roughly 340 items in the slice, and within 3 points roughly 950. A 75-item identifier slice cannot pass either. The consequence is not to relax the gate. It is to grow the slice, by sampling more identifier-heavy production traffic (see chapter 02), until the gate can be passed by a good change. A cheap interim measure is a programmatic check on the whole production log that no output alters an identifier string found in the retrieved text.

The rule flips when the slice is small by nature. If a slice is a rare, dangerous case, say 12 claims-denial explanations, no interval will be tight. There you read every item, by hand, on every change. That is a review step, not a statistic, and it is cheaper than it sounds for a dozen items.

Gates that are not about quality

Accuracy is one gate of several. Latency, cost and safety are also pass or fail, and they are measured on the same run, so there is no reason to evaluate them separately.

• Latency p95. Record time to first token and time to full answer for each item, and gate on the 95th percentile, not the mean. Retrieval and a longer prompt both move the tail first. Gate on a ratio to the current system (for example, no more than 15% worse) so that a slow day on the provider does not read as a regression.
• Cost per item. Input and output tokens times the price per token, summed over every call the item made. Prices change, so store the token counts and compute the cost at report time with the price table in force then. A prompt that adds 600 tokens of examples to every call at 13,000 sessions a day is a cost decision, and the report should say so.
• Safety checks. Programmatic and binary: no output leaks another customer's data, no refusal-bypass strings, no coverage promise on the forbidden list. These are zero-tolerance gates, with no interval. One failure blocks.

The thresholds belong in a file under version control, set by the person who owns the risk, before the run. A threshold chosen after seeing the numbers measures the numbers, not the system.

Wiring it into CI

The decision set runs on every change to a prompt, a model version, a retrieval setting or a tool definition. It does not run on every commit: a full run costs money and minutes, so the fast deterministic checks from chapter 01 gate the commit and the decision set gates the merge. The job runs both systems on the set, computes the paired report and the slice table, evaluates the gates, writes the report as a build artefact, and exits non-zero on a block.

decision_set: helpcentre-v7        # pinned version; changing it resets the baseline
baseline: prod                     # the system currently serving traffic
overall:
  delta_lower_bound_min: 0.0       # 95% paired interval must not include a loss
blocking_slices:
  limits_deductibles: {noninferiority_margin: 0.03}
  identifiers:        {noninferiority_margin: 0.03}
  exclusions:         {noninferiority_margin: 0.05}
latency_p95_ratio_max: 1.15
cost_per_item_ratio_max: 1.10
safety_failures_max: 0

Three details decide whether this survives contact with a team. Keep the baseline run cached against the set version, so the old system is not paid for twice. Save every item's outputs and scores in the report, so a reviewer can open the 20 losses. And log who overrode a block and why; an override is allowed, an invisible one is not. This is also where the traceability a regulator asks for comes from: a dated report per release, the set version and the thresholds in force.

When the result is "no difference"

An interval that spans zero has two meanings that look alike: the change does nothing, or the set was too small to see what it does. Check the interval's width against the smallest difference you care about. If the interval runs from -1 to +2 and you only care about 4 points, you have learned that the change is not worth 4 points, and that is a real answer. If it runs from -4 to +8, you have learned nothing.

Then decide on what you do know. A change that is no worse on quality and cheaper or faster ships on those grounds; a change that is no better and adds risk or cost does not. When the cost of waiting is low, collect more items on the slices that matter, or run the change as a canary on live traffic (chapter 08). Do not re-run until the interval happens to clear zero. Each re-run is another draw, and the chance that one of them clears it by luck grows with every try. Fix the number of runs and the rule beforehand.

For how this decision is explained under questioning, see the interview-prep treatment in Eval Set & the Ship Decision. Retrieval-specific gates are in Measuring Retrieval.

✓ Check yourself

Answer from memory before opening each one.

1. Two prompts score 82% and 85% on the same 300 items, and a colleague runs an unpaired two-proportion test and finds no significant difference. You rerun it paired and find one. What did the paired test use that the first ignored, and what would you check before trusting it?

   Show answer

   It used the fact that both prompts answered the same items, so shared item difficulty cancels and only the items where the systems differ carry variance. Before trusting it, check how many items differ and whether items are clustered (several per policy), since clustering makes the paired interval too narrow. See Paired comparison.

2. Your set has 500 items, and about one in eight answers changes under a typical prompt edit. A product owner asks you to confirm a 2-point improvement. Can you, and what do you tell them?

   Show answer

   No. Detecting 2 points at that disagreement rate needs roughly 2,000 to 2,700 items, so 500 items give an interval several points wide. Say the set can see about 5-point changes, and offer either more items or a live test. See How many items.

3. A change lifts the overall score by 2 points and drops the limits-and-deductibles slice by 3 points on 60 items, with an interval including zero. A teammate argues it is within noise and should ship. Where is the argument wrong?

   Show answer

   "Not significantly worse" is the wrong test for a costly slice. An interval that includes zero also includes a large loss, and 60 items cannot rule one out. The gate should demand evidence the slice is no worse than a margin, and a set too small to supply it should be grown. See Slices as blockers.

4. Thresholds were written down after a first run showed the new prompt at +1.2 points, and set at "at least +1". The author says the numbers merely recorded the team's expectation. Why does this not count as a pre-set gate, and what would?

   Show answer

   A threshold set after seeing the result is fitted to the result and will pass whatever the data show. A real gate is written by the risk owner and committed before the run, tied to a set version. See Gates and CI wiring.

5. A change is neither better nor worse, with a paired interval of -1.5 to +1.8 points, and cuts cost per item by 20%. Another is the same on quality but 30% slower at p95. What do you do with each, and what would change your mind about the first?

   Show answer

   Ship the first: the interval excludes any loss larger than about 1.5 points, which is within your margin, and cost improves. Block or rework the second, since quality did not pay for the latency. The first would be reconsidered if a blocking slice lost ground, because the aggregate can hide that. See When nothing differs.

Takeaway

• Compare systems item by item on the same set; the discordant items hold all the information, and a delta without its wins and losses is incomplete.
• Report a paired bootstrap interval, resampling clusters where items share a source, and read its width against the smallest difference you would act on.
• Detectable difference scales with 1/√n: halving it costs four times the items, so a 400-item set sees 5 to 6 points, not 2.
• Slices that carry harm are blockers, tested for non-inferiority against a margin set in advance; a slice too small to pass should be grown, not waved through.
• Latency p95, cost per item and zero-tolerance safety checks are gates on the same run, with thresholds committed before the run.
• "No difference" is a result only when the interval is narrow; otherwise it means the set cannot answer, and re-running until it clears zero is not a remedy.

Quality on a single answer is the easy case; a system that calls tools has to be judged on what it did, not only on what it said. → Evaluating Agents

Further reading

• Adding Error Bars to Evals — Evan Miller, 2024. Statistical treatment of LLM evals as experiments: comparing two models and planning an evaluation.
• τ-bench — Yao et al., 2024. Why repeated trials matter when the same item can pass once and fail the next time.
• AI Agents That Matter — Kapoor et al., 2024. Argues for reporting cost with accuracy and for holdout sets; the basis for the cost gate here.