Evaluation & Quality

Two layers of eval — and where you live

The agent ecosystem evaluates at two layers, and your role lives mostly in the second one:

The boundary is fuzzy at "agent end-to-end success as observed at the infra tier" — see below. Mostly, your evals are about the system staying up and fast under realistic conditions.

SLOs — what the service promises

A senior SWE on this team should be able to write SLOs in their sleep. The typical bundle:

• Availability — 99.9% (about 43 min/month down) or 99.95% (~22 min/month).
• Success rate — % of requests returning a valid response (excludes user-caused 4xx). E.g., ≥99.9%.
• Latency — p50, p95, p99, p999. For inference gateway: e.g., p99 first-token-latency < 500ms; p99 total < 30s for completions.
• Throughput — sustained QPS the system handles before SLO violation.
• Saturation — utilization headroom (CPU, memory, queue depth, KV cache).

For streaming endpoints, latency SLOs split:

• Time to first token (TTFT) — how fast the first byte streams to the client. Dominated by queueing + prefill.
• Inter-token latency (ITL) — pause between tokens. Dominated by decode step.
• Total completion time — for fixed-length workloads.

Load testing

You should be able to drive the service with realistic load and watch where it breaks. Tools:

What to measure:

• Open-loop vs closed-loop. Open-loop holds the request rate constant regardless of latency — this is what real users do. Closed-loop matches concurrency. Always include open-loop for tail-latency truth.
• Latency at saturation, not at empty. Bench at 50%, 75%, 90% of expected peak.
• Prompt-length distributions matter for LLM gateways. Long-context requests behave differently from short ones.

Chaos engineering

Inject failures in pre-production (and very carefully in production) to verify the system degrades gracefully.

• Kill a backend mid-traffic. Do circuit breakers open? Does routing failover?
• Inject latency on the wire. Linux tc netem, Toxiproxy. Tail latency under wire jitter.
• Drop packets. Same tools.
• Saturate Redis. Watch the gateway degrade — does it use rate-limit decisions as soft hints, or hard-fail?
• Postgres failover. Does the orchestrator resume runs after the DB rotates?
• Fill memory. Run out of KV cache on a backend. Does the gateway route around it?

The principle: failures you've already practiced are not 2am pages. They're routine.

Integration tests against real LLM endpoints

Most code can be unit tested with mocked LLM calls. Some can't:

• Streaming response handling — SSE keepalive, partial-message reassembly, mid-stream cancellation. These break against mocks but pass; test against a real endpoint in CI.
• Backend protocol compliance — vLLM, Triton, and Anthropic's API all have subtle differences in error shapes, token formats, retry semantics. Tests that mock all three lie.
• Latency-sensitive contracts — does your timeout still produce a clean error path when the upstream is slow but not failing?

The pattern: a small dedicated test backend (local vLLM with a tiny model, or a recorded-replay HTTP server), run as part of nightly integration. PR tests stay on mocks for speed.

Fuzz testing protocol parsers

Anywhere you parse external input — JSON tool args, model-emitted JSON, SSE frames, protobuf payloads — is a fuzz target. cargo-fuzz (libFuzzer) and afl.rs are the standard tools.

// fuzz_targets/parse_tool_args.rs
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    if let Ok(s) = std::str::from_utf8(data) {
        let _ = my_service::parse_tool_args(s);
        // The parse must not panic on any input. That's the contract.
    }
});

What you get: confidence that malformed model output doesn't panic the service. Critical, because LLMs emit malformed JSON for fun.

Agent end-to-end success rates (as observed from infra)

The Agent Systems team owns "is this agent good?" You own "did the system give the agent a fair chance to be good?" The infra-tier metrics:

• Runs that completed vs runs that hit step/budget/deadline limits.
• Tool call success rate per tool. A tool that's at 95% success is healthier than one at 60%.
• Median and tail step count per run. Tail run with 200 steps is suspicious — probably stuck.
• Cost per successful run. Trend over weeks. Regressions are a signal.
• Approval rate on human-in-the-loop gates. Low approval = the agent is suggesting bad actions = something upstream regressed.
• Cancellation rate by client. High cancellation = users giving up = latency or quality problem.

You feed all this to dashboards. The Agent Systems team uses it to decide whether to investigate a regression.

Perf regression suites

Every release should be benchmarked before deploy. The minimum suite:

• Microbenchmarks (criterion) for hot functions. Catches algorithmic regressions in PR review.
• Load test (k6 / ghz) in CI against a representative workload. Compare p99 and throughput against last release.
• Allocation count benchmarks (dhat-rs or iai) on a synthetic request. Catches "this PR added a per-request allocation."

// benches/parse.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn bench_parse(c: &mut Criterion) {
    let input = include_str!("../fixtures/sample_tool_args.json");
    c.bench_function("parse_tool_args", |b| {
        b.iter(|| my_service::parse_tool_args(black_box(input)))
    });
}

criterion_group!(benches, bench_parse);
criterion_main!(benches);

Error budgets

SLOs define a budget. 99.9% availability = 43 minutes/month down. You spend that budget on real outages, planned maintenance, risky deploys. When the budget is burned, deploys freeze.

• Burn-rate alerts — alert when budget is burning N× faster than nominal over a short window. Earlier detection than "the budget is gone."
• Multi-window burn rate — alert on both fast-burn (over 1h) and slow-burn (over 24h) to catch both incidents and slow degradations.
• Budget freezes — when the monthly budget is gone, no new feature deploys. Forces investment in reliability.

✓ Check yourself

Answer from memory before opening each one.

1. A completion endpoint meets its p99 total-latency target, yet users call it slow. Which latency measures explain the gap, and what dominates each?

   Show answer

   Time to first token and inter-token latency. The first is dominated by queueing plus prefill and the second by the decode step. A total-time target hides a slow first token or a stuttering stream, which are what the user feels. See SLOs.

2. A load test holds 2,000 concurrent clients and shows a healthy p99, but production times out at peak. What was wrong with the test design?

   Show answer

   A closed-loop test matches concurrency, so when the service slows the clients send fewer requests and the test eases its own load. Real users keep arriving, so use an open-loop test at a constant rate, and run it at 50, 75 and 90 percent of expected peak with realistic prompt lengths. See Load testing.

3. All your SSE tests pass against a mock, and production breaks on mid-stream cancellation. Why did the mock miss it, and how do you arrange CI?

   Show answer

   A mock cannot reproduce real keepalive behaviour, partial-frame reassembly or the differences in error shape between vLLM, Triton and a hosted API. Keep PR tests on mocks for speed and run a nightly integration suite against a small real backend or a recorded-replay server. See Integration tests against real LLM endpoints.

4. Your service has a 99.9% monthly availability SLO and has used most of its budget by day ten. A risky deploy is proposed. How do you decide?

   Show answer

   The budget is about 43 minutes a month, so you are burning far faster than nominal. Without budget remaining you hold the risky change and put the time into reliability; with budget you ship and watch it closely. The decision comes from the number, not from opinion. See Error budgets.

5. Why is the parser for model-emitted JSON a fuzz target, and what is the contract the fuzzer checks?

   Show answer

   Models emit malformed JSON, and a panic on bad input takes down a request path in production. The contract is that the parser never panics on any byte sequence; it returns an error. Fuzzing finds inputs that a hand-written test would not. See Fuzz testing protocol parsers.

6. After a deploy, tool-call success is steady but the approval rate on human gates falls sharply and median steps per run rise. What does an infra owner conclude, and who gets the dashboard?

   Show answer

   A low approval rate means the agent is suggesting actions humans reject, and more steps suggest runs are wandering. The infra tier reports it as a signal that something upstream regressed, such as a prompt or model change, and the Agent Systems team decides whether to investigate. See Agent end-to-end success rates.

Takeaway

• Eval has two layers: agent quality belongs to Agent Systems, and service quality, meaning SLOs, latency and cost, belongs to you.
• For streaming endpoints set separate SLOs for time to first token and inter-token latency, since total time hides what users feel.
• Load test open-loop at a constant rate, near saturation, with realistic prompt lengths; a closed-loop test hides tail latency.
• Practise failures in chaos tests, such as killing a backend, adding latency and failing over Postgres, so they are routine instead of pages at 2am.
• Mocks lie about streaming and protocol differences, so run nightly integration tests against real backends and fuzz every parser of external input.
• Benchmark every release against the last one, and use error budgets and burn-rate alerts to turn "should we ship?" into a numeric decision.

Evals tell you the system is degrading; the next chapter covers how it fails and how it recovers. → Error handling & failure modes

Further reading

• Service Level Objectives — Google SRE Book, 2016. SLIs, SLOs and how to choose targets.
• Implementing SLOs — Google SRE Workbook, 2018. Error budgets and a policy for acting when they run out.
• Alerting on SLOs — Google SRE Workbook, 2018. Burn-rate and multi-window alerts.
• Open and closed models — Grafana k6 documentation. Why the arrival model changes what a load test measures.
• Rust Fuzz Book — Rust Fuzzing Authority. Setting up cargo-fuzz targets for a parser.
• Toxiproxy — Shopify. A proxy for injecting latency and failures in tests.