Deep Dive

Evals for LLM Systems

How to know whether an LLM system works, before and after it ships. Eval sets that predict production, graders you can trust, LLM-as-judge with its biases measured, ship-or-hold decisions with real statistics, and the harder cases: agents, RAG, model upgrades, and the moment offline and online numbers disagree.

A number you can defendeval setby slicethe systemgradersrules · judge · humanship? choose? permit?paired · per slice · with an intervaljudge biases:position · verbosityself-preference→ check against peoplecost, latency andsafety gates tooproduction samplefailures become new itemsoffline decides · online checks the decision
If you only have an hour

Read 00 for what an eval is for, then 02 (the set) and 05 (the decision). If you judge with a model, add 04 before you trust another score.

Study pathsPick the plan that fits the time you have
A week of evenings

In order, two a night. Build a 100-item set for a system you own after 02 and re-run it after every later chapter.

One afternoon

00, 02, 04, 05, 08. Read the Takeaway of the rest.

Before an interview

00, 02, 04, 05, 06 here; then AI Engineer Interview Prep 07–08 for the interview framing.

Section A · Orient
00 Start Here What an eval is for, the decisions it serves, the vocabulary, and the running example.
Section B · Foundations
01 What an Eval Is For Offline and online, capability and regression, the decision each one serves, why benchmark scores do not transfer, and the eval stack a team actually needs. 02 Building the Eval Set Sampling from production by slice, synthetic items for rare and dangerous cases, labelling and inter-annotator agreement, how big is big enough, PII, and keeping the set from going stale. 03 Graders Exact match, programmatic checks, rubric-based model grading and human review: what each can and cannot see, how to write a rubric that two people score the same way, and partial credit. 04 LLM-as-Judge Position, verbosity and self-preference bias with the evidence, calibrating a judge against human labels, agreement statistics, pairwise versus absolute scoring, and when a judge is the wrong tool.
Section C · Decisions
05 The Ship Decision Paired comparison on the same items, confidence intervals and how many items a given difference needs, per-slice regressions, cost and latency as gates, and wiring it into CI so the decision is made the same way every time. 06 Evaluating Agents Trajectories, tool-call correctness, step and cost budgets, partial credit, safety checks, mocked environments, replaying production traces, and reliability across repeated trials. 07 Evaluating RAG Separating retrieval from generation, context relevance, faithfulness and answer relevance, citation checks, abstention rate, and the labelled set shared with the retrieval team. 08 Model and Prompt Changes Regression suites for prompt edits and model upgrades, pinning and canaries, shadow traffic, what changes silently between versions, and the rollback you rehearse before you need it.
Section D · In production
09 Online Signals and Drift Production metrics that stand in for quality, feedback loops and their biases, drift in inputs and in the model, and what to do when offline and online disagree.