Evaluating an Agent Beyond Its Final Answer

The question, and what is being scored

The interviewer says: "Your claims agent routes 95% of first notices of loss to the right adjuster queue. Is it ready?" The follow-up is always some form of "how do you know it got there the right way?"

Harbourline's claims-intake agent reads a first notice of loss, asks for missing information, checks coverage and routes the claim to an adjuster. About 6k new claims arrive a day. A correct route that skipped the coverage check is a regulatory exposure, because a coverage decision with no record of the check cannot be defended. The interviewer is scoring:

• You separate the outcome from the path, and you know when each one matters.
• You can name the safety and policy checks that no outcome score will catch.
• You grade cost and step count, because an agent that is right in 40 steps is a different product from one that is right in 8.
• You know agents are stochastic and measure reliability over repeated runs, not one run.

The grading mechanics, such as paired comparison and judge calibration, are in chapter 07. This chapter adds what changes when the thing under test takes actions. The decision about whether to build an agent at all is in chapter 04.

What to grade: five layers

A final-answer score reads one layer. An agent produces a record of the whole run, and each layer of it can fail alone. Five layers cover a claims agent.

Partial credit belongs here. Anthropic's guidance on agent evals suggests building it in for tasks with several components: an agent that verifies the claimant and then fails to route is better than one that fails at the start, and a pass-or-fail score hides that. Report the layers separately. One blended number lets a strong outcome score hide a weak safety score, which is the failure this chapter is about.

Safety is graded differently from the rest. Outcome, evidence and efficiency are rates you tune. A write without confirmation is a count that must be zero, and one occurrence in a thousand runs is a finding, not noise.

Outcome or path: the tension you should name

There are two defensible positions, and a strong answer states both. Anthropic's engineering post on agent evals argues that it is often better to grade what the agent produced rather than the path it took, and calls checks for a fixed order of tool calls "overly brittle", since agents find valid approaches the designer did not foresee. τ-bench (Yao et al., 2024), a benchmark for tool-using agents talking to a simulated user, does the same: it scores the database state at the end of the conversation against an annotated goal state, which leaves room for the conversation itself to vary.

Against that, an outcome score alone passes the Harbourline failure. The agent saw "windscreen cracked by a stone" and routed to the glass queue without calling the coverage tool. The queue was right. The record of why was missing, and on the next claim with a policy exclusion the same shortcut would send the claim to the wrong place.

The resolution is to grade constraints on the path, not a script for it. Do not require "call A, then B, then C". Require facts that the final state and the audit log must contain, and impose an order only where order has meaning.

• The final state must include a coverage decision record, with the policy ID and loss date that match the claim. This does not care how the agent obtained it.
• The coverage check must come before the route, since a routing that precedes the check cannot rest on it.
• Any change to a customer record must follow a confirmation message from the customer.

This keeps the benefit of outcome grading, which is tolerance of different valid routes, and catches the failures that matter. It flips when the order is itself the product, for example a compliance flow where a regulator expects a fixed sequence. Then an exact order check is correct.

Environments, mocked tools and replaying production

An agent eval needs somewhere to run. Calling the real policy system a thousand times is slow, costs money and changes real data. The standard setup is a sandbox: tools backed by a small in-memory or test database seeded per scenario, so each run starts from the same state and the final state can be compared. For a conversational agent, a model plays the customer from a script of facts, as it does in τ-bench. For Harbourline, the claims scenario carries a policy, a loss description and a list of facts the simulated claimant will give only when asked.

Two further tools extend the sandbox.

• Fault injection. Make the policy lookup time out, return an empty result, or return a record for a different policy period. This tests behaviour that production hits weekly and a happy-path eval never does. The right result is usually to stop and escalate, not to guess. See chapter 14 for what happens when the agent retries instead.
• Replaying production traces. Real traces make the best scenarios, since they hold the actual distribution of messy input. Use them two ways. First, as a source: turn failures into scenarios with an expected state. Second, as a monitor: run the same rule checks, such as "coverage check before route", over a sample of live traces every day. Replaying recorded tool responses against a new agent version has a limit. If the new version calls a different tool or passes different arguments, the recorded response no longer applies, so use a stateful mock for that comparison and keep recorded replay for checking that old behaviour has not changed.

Rules or a judge

Use rules for everything that can be checked mechanically. Whether a tool name appears, whether arguments match a schema, whether a write follows a confirmation, whether the step count is under budget: these are exact, free and have no bias. A small grader looks like this.

from dataclasses import dataclass

@dataclass
class Call:
    tool: str
    args: dict

CUSTOMER_WRITES = {"update_address", "change_payment_date"}

def grade(trace, expected_queue, max_steps=12):
    """Rule checks on one claims-agent trace. Returns (score, failed_checks)."""
    names = [c.tool for c in trace]
    routes = [i for i, n in enumerate(names) if n == "route_to_adjuster"]
    failed = []
    if not routes or trace[routes[-1]].args.get("queue") != expected_queue:
        failed.append("outcome")
    if "check_coverage" not in names or (routes and names.index("check_coverage") > routes[0]):
        failed.append("coverage_before_route")
    for i, n in enumerate(names):
        if n in CUSTOMER_WRITES and "confirm_with_user" not in names[:i]:
            failed.append("confirmed_writes")
            break
    if len(trace) > max_steps:
        failed.append("step_budget")
    return 1 - len(failed) / 4, failed

skipped = [Call("read_fnol", {}), Call("route_to_adjuster", {"queue": "auto-glass"})]
print(grade(skipped, "auto-glass"))   # (0.75, ['coverage_before_route'])

The outcome check passes and the evidence check fails, which is the whole point. In a real harness the coverage check reads the coverage record from the final state as well as the call, so a call whose result was ignored does not count.

Use a model judge for what rules cannot see: whether the clarifying question was sensible and not repetitive, whether the explanation to the claimant was accurate and plain, whether the agent handled a distressed customer well. Grade one dimension per judge call with a written rubric, and calibrate each judge against human labels as in chapter 07. Anthropic's guidance adds that you should read the transcripts and grades from many trials, because that is the only way to learn whether a grader is wrong. A judge that scores a whole trajectory with one holistic number is the weakest design here.

Reliability: one run is not a result

An agent samples from a model at every step, so the same scenario can pass on one run and fail on the next. A single run per scenario measures luck. Run each scenario n times, with n of 5 to 10 being typical, and report two things: the average success, and how often the agent succeeds every time.

τ-bench defines pass^k for the second. It is the probability that all k trials of a task succeed, estimated from n trials with c successes as C(c, k) / C(n, k), then averaged over tasks. Its contrast is pass@k, the chance of at least one success in k tries, which suits a tool where a retry costs nothing. For a customer-facing agent, pass^k is the one that matches what customers experience, because the customer gets one run and expects it to work every time they ask. The paper's headline result is the gap: a function-calling agent running gpt-4o succeeded on fewer than half the tasks, and its pass^8 was below 25% in the retail domain.

The arithmetic is quick to do aloud. If a task succeeds in 6 of 8 trials, pass^1 is 6/8 = 0.75, pass^4 is C(6,4)/C(8,4) = 15/70 ≈ 0.21, and pass^8 is 0. Anthropic's version of the point: a per-trial rate of 75% gives about 42% over three trials, since 0.75 cubed is 0.42. The practical rule is that an agent which looks fine on average can be unreliable per task, and the number to put in the ship rule is pass^k for the k that matches how often a customer would repeat the request.

Reliability costs money. Ten trials of 200 scenarios is 2,000 agent runs per evaluation, at the agent's full per-run cost. Keep a small scenario set for every pull request, run the full multi-trial set nightly or before release, and note that the cost per run is itself a metric that appears in the ship decision in chapter 07.

Worked answer: the claims agent that skipped the coverage check

This is an illustrative setup, not measured data. Harbourline builds 200 scenarios from sampled claims plus hand-written exclusions, flood edge cases and fault injections. Each scenario runs 5 times against a sandbox with a seeded policy database. The first report reads like this.

The headline figure would have said 94% and passed. The evidence row shows that in roughly a quarter of runs the agent routed without a coverage record. Reading those traces shows the pattern: short, clear glass and windscreen claims, where the agent infers the queue from the loss description. The fix is in the harness and not only the prompt. Make routing a tool that refuses to run without a coverage decision ID, so the skip becomes impossible, not just unlikely. Then add the failing scenarios to the set so the next change cannot bring it back.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. A claims agent routes 96% of test claims to the correct queue, and a reviewer asks you to approve it for launch. What do you look at before agreeing, and why is 96% not enough?

   Show answer

   The other layers: whether each run holds the evidence that makes the route defensible (a coverage decision), whether any write skipped confirmation, and cost and steps against a reference. The tempting answer is that a correct end state means a correct process; it fails because an agent can reach the right queue by a shortcut that breaks on the next claim. See What to grade.

2. A colleague wants an eval that asserts the exact tool sequence read, check coverage, ask, route. Give one reason this is fragile and one case where it is the right design.

   Show answer

   It fails valid runs that reorder or add steps, such as asking the claimant first, so you spend time chasing false failures. It is right when order is the requirement itself, for example a regulator expects a fixed sequence. Otherwise check records and ordering only where it carries meaning. See Outcome or path.

3. A scenario passes in 7 of 10 trials. What are pass^1 and pass^3 for that scenario, and what does the gap tell you?

   Show answer

   pass^1 is 7/10 = 0.7. pass^3 is C(7,3)/C(10,3) = 35/120 ≈ 0.29. The gap says the agent is inconsistent on this task: if a customer repeats the request three times, all three succeed less than a third of the time. See Reliability: pass^k.

4. You replay last month's recorded production traces against a new agent version to check for regressions, and half the replays diverge at the second tool call. Is the new version worse?

   Show answer

   Not necessarily. Once the new version calls a different tool or passes different arguments, the recorded response no longer applies, so the replay is invalid from that point. Use a stateful mock for comparing versions, and keep recorded replay for checking unchanged behaviour. See Environments and replay.

5. The customer-service chat agent scores well on resolution and on cost, but in 3 of 5,000 test runs it called the address-change tool without asking the customer. A product lead calls that a 0.06% rate and acceptable. What do you say?

   Show answer

   Safety violations are counts with a target of zero, not rates to trade against resolution. Each one is a customer record changed without consent. Read the three traces, then fix it in the harness by making the write tool reject calls with no confirmation token, so it cannot happen. A prompt-only fix leaves it unlikely, not impossible. See What to grade and chapter 06.

Takeaway

• An agent's final answer is one of five layers: outcome, required evidence, tool calls, safety and efficiency, and each should be reported on its own.
• Grade constraints on the path, such as records that must exist and orderings that carry meaning, not a fixed script of tool calls.
• A correct outcome with a missing coverage record is a hidden failure, so the check must read the record in the final state.
• Safety violations are counts with a target of zero, and the durable fix is a harness that makes the action impossible.
• Run each scenario several times and report pass^k, since a customer-facing agent has to work on every run, not on average.
• Use rules for what is checkable, a calibrated one-dimension judge for what is not, and read the transcripts to learn whether either is wrong.

Grading is the offline half of the work; next comes running the model at volume, starting with batching. → A Batching System for LLM Queries

Further reading

• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao, Shinn, Razavi and Narasimhan, 2024. The end-state scoring method, the simulated user, and the definition and estimator of pass^k.
• Demystifying evals for AI agents — Anthropic Engineering, January 2026. Outcome versus transcript grading, partial credit, pass@k against pass^k, and reading transcripts.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023. The known biases to test for before trusting a model judge on trajectories.