Evaluation & Quality

Two layers of eval — and where you live

The agent ecosystem evaluates at two layers, and your role lives mostly in the second one:

The boundary is fuzzy at "agent end-to-end success as observed at the infra tier" — see below. Mostly, your evals are about the system staying up and fast under realistic conditions.

SLOs — what the service promises

A senior SWE on this team should be able to write SLOs in their sleep. The typical bundle:

• Availability — 99.9% (3.5 min/month down) or 99.95% (~22 min/month).
• Success rate — % of requests returning a valid response (excludes user-caused 4xx). E.g., ≥99.9%.
• Latency — p50, p95, p99, p999. For inference gateway: e.g., p99 first-token-latency < 500ms; p99 total < 30s for completions.
• Throughput — sustained QPS the system handles before SLO violation.
• Saturation — utilization headroom (CPU, memory, queue depth, KV cache).

For streaming endpoints, latency SLOs split:

• Time to first token (TTFT) — how fast the first byte streams to the client. Dominated by queueing + prefill.
• Inter-token latency (ITL) — pause between tokens. Dominated by decode step.
• Total completion time — for fixed-length workloads.

Load testing

You should be able to drive the service with realistic load and watch where it breaks. Tools:

What to measure:

• Open-loop vs closed-loop. Open-loop holds the request rate constant regardless of latency — this is what real users do. Closed-loop matches concurrency. Always include open-loop for tail-latency truth.
• Latency at saturation, not at empty. Bench at 50%, 75%, 90% of expected peak.
• Prompt-length distributions matter for LLM gateways. Long-context requests behave differently from short ones.

Chaos engineering

Inject failures in pre-production (and very carefully in production) to verify the system degrades gracefully.

• Kill a backend mid-traffic. Do circuit breakers open? Does routing failover?
• Inject latency on the wire. Linux tc netem, Toxiproxy. Tail latency under wire jitter.
• Drop packets. Same tools.
• Saturate Redis. Watch the gateway degrade — does it use rate-limit decisions as soft hints, or hard-fail?
• Postgres failover. Does the orchestrator resume runs after the DB rotates?
• Fill memory. Run out of KV cache on a backend. Does the gateway route around it?

The principle: failures you've already practiced are not 2am pages. They're routine.

Integration tests against real LLM endpoints

Most code can be unit tested with mocked LLM calls. Some can't:

• Streaming response handling — SSE keepalive, partial-message reassembly, mid-stream cancellation. These break against mocks but pass; test against a real endpoint in CI.
• Backend protocol compliance — vLLM, Triton, and Anthropic's API all have subtle differences in error shapes, token formats, retry semantics. Tests that mock all three lie.
• Latency-sensitive contracts — does your timeout still produce a clean error path when the upstream is slow but not failing?

The pattern: a small dedicated test backend (local vLLM with a tiny model, or a recorded-replay HTTP server), run as part of nightly integration. PR tests stay on mocks for speed.

Fuzz testing protocol parsers

Anywhere you parse external input — JSON tool args, model-emitted JSON, SSE frames, protobuf payloads — is a fuzz target. cargo-fuzz (libFuzzer) and afl.rs are the standard tools.

// fuzz_targets/parse_tool_args.rs
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    if let Ok(s) = std::str::from_utf8(data) {
        let _ = my_service::parse_tool_args(s);
        // The parse must not panic on any input. That's the contract.
    }
});

What you get: confidence that malformed model output doesn't panic the service. Critical, because LLMs emit malformed JSON for fun.

Agent end-to-end success rates (as observed from infra)

The Agent Systems team owns "is this agent good?" You own "did the system give the agent a fair chance to be good?" The infra-tier metrics:

• Runs that completed vs runs that hit step/budget/deadline limits.
• Tool call success rate per tool. A tool that's at 95% success is healthier than one at 60%.
• Median and tail step count per run. Tail run with 200 steps is suspicious — probably stuck.
• Cost per successful run. Trend over weeks. Regressions are a signal.
• Approval rate on human-in-the-loop gates. Low approval = the agent is suggesting bad actions = something upstream regressed.
• Cancellation rate by client. High cancellation = users giving up = latency or quality problem.

You feed all this to dashboards. The Agent Systems team uses it to decide whether to investigate a regression.

Perf regression suites

Every release should be benchmarked before deploy. The minimum suite:

• Microbenchmarks (criterion) for hot functions. Catches algorithmic regressions in PR review.
• Load test (k6 / ghz) in CI against a representative workload. Compare p99 and throughput against last release.
• Allocation count benchmarks (dhat-rs or iai) on a synthetic request. Catches "this PR added a per-request allocation."

// benches/parse.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn bench_parse(c: &mut Criterion) {
    let input = include_str!("../fixtures/sample_tool_args.json");
    c.bench_function("parse_tool_args", |b| {
        b.iter(|| my_service::parse_tool_args(black_box(input)))
    });
}

criterion_group!(benches, bench_parse);
criterion_main!(benches);

Error budgets

SLOs define a budget. 99.9% availability = 43 minutes/month down. You spend that budget on real outages, planned maintenance, risky deploys. When the budget is burned, deploys freeze.

• Burn-rate alerts — alert when budget is burning N× faster than nominal over a short window. Earlier detection than "the budget is gone."
• Multi-window burn rate — alert on both fast-burn (over 1h) and slow-burn (over 24h) to catch both incidents and slow degradations.
• Budget freezes — when the monthly budget is gone, no new feature deploys. Forces investment in reliability.