Graders

Overview

Every eval ends in a grader, and the grader sets the ceiling on what the eval can tell you. A grader that cannot see the policy version cannot catch a stale deductible. A grader that sees only the final text cannot catch a claims agent that skipped the coverage check. Before choosing a metric, ask what evidence the grader has in front of it and whether the failure you care about leaves a trace in that evidence.

Harbourline's help-centre assistant answers about 13,000 sessions a day, and the eval set from chapter 02 holds roughly 400 labelled items. A wrong deductible is a complaint. A wrong coverage statement is a regulatory problem. Those two costs justify different graders, and this chapter builds the set of graders that covers both without paying for human review on every item.

What each grader can see

Four kinds of grader cover almost every case, and they differ less in quality than in field of view. Each one reads a different slice of the run.

The cost column spans orders of magnitude: microseconds for a regex, a model call for a rubric grader, minutes of an expert's time for a human. That spread is why the design question is never "which grader is best". It is "what is the cheapest grader that can see this failure". Anthropic's own guidance for building evals makes the same trade: prefer automated grading, and accept more items with a somewhat noisier grader over fewer, hand-graded ones (Further reading).

Deterministic graders: match and programmatic checks

Exact match works when the answer is a closed set: an intent label, a routing queue, a yes/no coverage verdict. It stops working the moment the answer is a sentence. Normalised match loosens it by lowercasing, stripping punctuation and whitespace, and canonicalising numbers and dates, so "$1,000", "1000 dollars" and "£1,000" collapse to comparable forms. Normalisation is where silent bugs live: if your normaliser maps "£1,000" and "$1,000" to the same number, you have just erased the difference between a UK and a US policy. Keep the currency and the jurisdiction in the comparison key.

Programmatic checks go further than string comparison. They read structure, not prose. For Harbourline the useful ones are:

• Citations resolve. Every cited clause id exists in the current policy corpus, and the cited clause is one the gold label accepts for this question.
• Numbers within tolerance. A limit or deductible must match exactly; a prorated premium may match within a cent.
• Forbidden content. No claim that a loss is covered, no promise of payout, no policy number or other customer's data in the answer.
• Forbidden actions. No call to a write tool without a confirmation turn before it (see chapter 06).
• Schema. The claims-intake agent's output parses, and every required field is present.
• Instruction following. The fixed instructions in the system prompt that a program can verify: the answer is in the language of the question, under the length ceiling, carries the required disclaimer on coverage questions, and uses no markdown where the channel cannot render it.

These checks are cheap, fast and unambiguous, so run them on every item and every change. Their blind spot is the one named in the table: an answer can satisfy every rule and still be wrong, because the rules encode only what you thought to check. The mechanism to hold in mind is that a programmatic grader scores evidence of correctness, not correctness. Where the evidence is tight (an amount that must equal a gold number), the proxy is nearly the thing itself. Where it is loose (an answer that "mentions the exclusion"), it is not.

A guardrail is a grader that runs in production, and it needs the same measurement as any other. Harbourline's output filter blocks coverage promises and other customers' data before a response is shown. Treat it as a classifier with its own labelled set of violating and clean outputs: how many violations it catches, how many clean answers it blocks, and whether the correction it applies (a rewrite, a refusal, a hand-off) leaves the customer with something usable. A filter that is never measured is a filter you are trusting. The eval set should carry both the violation it must catch and the near-miss it must let through.

Here is the deductible gate. It extracts the money amounts from the answer, requires exactly one, and checks it against the gold amount, the policy version and the citation.

import re
from dataclasses import dataclass

MONEY = re.compile(r"[$£]\s?(\d[\d,]*(?:\.\d+)?)")

@dataclass
class Item:
    gold_amount: float          # deductible in the gold policy version
    gold_version: str           # e.g. "HO-3 2025-07"
    gold_citation: str          # id of the policy clause that states it

def gate(answer: str, cited_ids: list[str], item: Item, corpus_ids: set[str]) -> dict:
    amounts = {float(m.replace(",", "")) for m in MONEY.findall(answer)}
    return {
        "number": item.gold_amount in amounts and len(amounts) == 1,
        "version": item.gold_version in answer,
        "citation_resolves": bool(cited_ids) and all(c in corpus_ids for c in cited_ids),
        "citation_correct": item.gold_citation in cited_ids,
    }

item = Item(1000.0, "HO-3 2025-07", "HO3-2025-07:s4.2")
corpus = {"HO3-2025-07:s4.2", "HO3-2024-01:s4.2"}
print(gate("Your wind/hail deductible is $1,000 (HO-3 2025-07).", ["HO3-2025-07:s4.2"], item, corpus))
print(gate("Your deductible is $500 or $1,000.", ["HO3-2024-01:s4.2"], item, corpus))

The second call shows the point of requiring a single amount. "$500 or $1,000" contains the right number, and a naive "gold number appears in the answer" check would pass it. A customer reading that sentence has no answer.

Writing a rubric two people score the same way

When the property you care about needs reading comprehension (is the answer faithful to the clause, does it say what the customer should do next), you need a rubric. A model grader and a human reviewer use the same one, which is the first reason to write it carefully: if two people cannot apply it consistently, a model will not apply it consistently either, and you will not be able to tell which of them is wrong.

Four habits separate a usable rubric from a vague one.

1. Binary criteria, not scales. "Does the answer state the deductible that the cited clause states? yes or no" is something two reviewers can agree on. "Rate accuracy from 1 to 5" is not: the 3-versus-4 boundary is each reviewer's private taste. A five-point scale also hides disagreement, because averages of noisy scores look smooth.
2. One property per criterion. "Accurate and helpful and concise" lets a reviewer fail the answer for any of three reasons and lets you read the failure as any of three. Split it, and each criterion points at a fix.
3. Anchor examples. For each criterion, give a passing and a failing example, with the borderline ones that came up in calibration. Reviewers rarely disagree about the clear cases. They disagree on the edge, and the edge is only learned from examples.
4. State what to ignore. Tone, length and formatting should not move a correctness verdict. Say so in the rubric.

Test the rubric before you rely on it. Have two reviewers score the same 40 to 50 items independently, then compute agreement with a chance correction such as Cohen's kappa (the statistic is defined, with a worked example, in chapter 04). Raw percentage agreement flatters rubrics on skewed data. When 90% of answers pass, two reviewers who guess "pass" nine times in ten agree on 82% of items by chance alone, so a raw 90% agreement is a kappa of about 0.44, which is moderate at best. Low kappa after the first round almost always means the rubric, not the reviewers. Read the disagreements, find the criterion they cluster on, and add the example that would have settled it.

Ask the grader for its reasoning before the verdict, and keep the reasoning in the stored report. A grader that writes "the answer states $500; the cited clause states $1,000" gives you an item you can audit in ten seconds. A bare "fail" gives you nothing to audit.

Partial credit and how it gets abused

Partial credit is tempting because an answer that gets four of five things right feels better than one that gets one of five. The problem is that a weighted sum treats failures as exchangeable, and in an insurance product they are not. An answer with the right deductible, the right procedure, a polite tone and a wrong exclusion scores 0.8 on a four-criterion sum. The customer who relies on it loses a claim.

Three rules keep partial credit honest.

• Gate before you sum. Criteria that carry regulatory or financial harm are pass/fail gates. An answer that fails a gate scores zero regardless of the rest. Partial credit applies only to quality criteria that do not carry that cost, such as completeness of next steps.
• Report the criteria, not just the total. A drop from 0.86 to 0.83 in a sum could be the exclusions slice collapsing and tone improving. The per-criterion table is the result; the sum is a headline.
• Fix the weights before you see the results. Weights chosen after a run tend to favour the variant you already liked. Record them with the rubric version.

The case where partial credit earns its place is long-form output with many independent facts, such as a claims summary with a dozen fields. There, per-field accuracy is the right metric and a single pass/fail on the whole summary hides how close the output is. The exception still holds: a field that drives a decision, such as the coverage determination, remains a gate.

Grading the process, not only the output

An output grader sees the answer. A process grader sees how the system got there: which passages were retrieved, which tools were called, with what arguments and in what order. The two disagree more often than teams expect. A help-centre answer can be right because the model remembered a policy from training data, and break when the policy changes. A claims agent can reach the right routing by a path that skips a required check. Output-only grading rates both as passes.

Process grading costs more to set up, because you must define what a good process is and capture it in a trace. Use it where the process carries the risk. For the help-centre assistant, a cheap process check is "the cited clause was in the retrieved set" (the answer was grounded, not recalled). For the claims agent and the customer-service agent, the process is the product, and chapter 06 treats it in full, along with the interview-prep view in evaluating agents beyond the answer. For retrieval specifically, the mechanics are in measuring retrieval.

Combining graders: gate, then judge, then audit

Layer the graders from cheapest and most certain to dearest and most subjective. Each layer sees only what survived the one before, and each is justified by what the previous layer cannot see.

The gate does three jobs. It saves money, since a clearly failed item needs no model call. It removes the cases where a model grader is weakest, which is exact numbers and identifiers (chapter 04 explains why). And it makes the model grader's job smaller and more consistent, because every item it sees already has the right number and a resolving citation.

The human audit is what makes the cheaper graders trustworthy. Send the humans three kinds of item: everything the gate failed, to confirm the gate is not rejecting correct answers; every item where the model grader and a programmatic check disagree; and a random sample of the passes, to estimate how many wrong answers slip through. A 10% sample of 400 items is 40 reviews per run, which is an afternoon rather than a project. If the audit finds the gate rejecting good answers, fix the normaliser. If it finds bad answers passing, add the rule that would have caught them, and add the item to the set.

✓ Check yourself

Answer from memory before opening each one.

1. A teammate proposes scoring the deductible slice by checking that the gold amount appears somewhere in the answer. Name two answers that would pass this check and still harm a customer.

   Show answer

   An answer that lists several amounts ("$500 or $1,000") passes, and so does an answer that states the right amount for the wrong policy version or jurisdiction. The check scores presence, not a claim. Extract the stated amount, require exactly one, and compare version and jurisdiction as well. See deterministic graders.

2. Two reviewers score 100 answers with a pass/fail rubric. They agree on 90, and each marks 90 answers as passes. Their manager calls the rubric solid. What is the chance-corrected agreement, and what do you do next?

   Show answer

   Chance agreement is 0.9 × 0.9 + 0.1 × 0.1 = 0.82, so kappa is (0.90 − 0.82) / (1 − 0.82), about 0.44. The raw 90% looks strong only because most answers pass. Read the 10 disagreements, find the criterion they cluster on, and add anchor examples before you trust the rubric. See rubrics.

3. A customer-service answer scores 0.85 on a weighted rubric: correct procedure, friendly tone, clear next steps, and a wrong statement that a flood loss is covered. Why is the score misleading, and how do you restructure it?

   Show answer

   A weighted sum lets three good qualities offset one failure that carries regulatory cost. Turn the coverage statement into a gate: failing it zeroes the item. Keep partial credit for quality criteria that carry no such cost, and report per-criterion results. See partial credit.

4. Your gate rejects 6% of answers that a human audit says are correct. Where do you look first, and why does fixing it matter more than fixing the model grader?

   Show answer

   Look at the normaliser: formats such as "1,000.00", "one thousand" or a currency symbol the parser misses. A false fail in the first layer removes the item from every later layer and depresses the headline score on every run, so the error is systematic and invisible downstream. See combining graders.

5. Where would you not add a model grader at all in the help-centre pipeline, and what do you use instead?

   Show answer

   On exact figures, identifiers and anything with a resolvable source, such as a deductible, a form number or a clause citation. A programmatic check is exact, free and repeatable, while a model grader can pass a plausible wrong number. Use the model grader for properties that need reading, such as faithfulness to the clause. See combining graders.

Takeaway

• A grader scores only what it can see, so choose it by whether the failure you fear leaves a trace in its inputs.
• Programmatic checks are the default for numbers, identifiers, citations, schemas and forbidden actions, because they are exact and free, but they score evidence of correctness rather than correctness.
• "Contains the gold string" passes hedged answers; require one extracted claim and compare version and jurisdiction too.
• A rubric works when two people apply it the same way: binary criteria, one property each, anchor examples, and a chance-corrected agreement check before use.
• Gate the criteria that carry regulatory or financial harm and sum only the rest; fix weights before the run, and report per criterion.
• Layer graders from cheap and certain to dear and subjective, and spend human time on gate fails, disagreements and a random sample of passes.

The model grader in the middle of that stack has its own biases and needs its own calibration. → LLM-as-Judge

Further reading

• Define success criteria and build evaluations — Anthropic docs, accessed 2026. Eval design principles, including preferring automated grading at volume and using a different model to grade than the one that generated the output.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023. The evidence base for what a model grader can and cannot do; chapter 04 uses it in detail.
• G-Eval — Liu et al., 2023. Chain-of-thought evaluation steps plus form-filling scoring, and its noted bias toward model-written text.