Section D · In production

Online Signals and Drift

Production metrics that stand in for quality, feedback loops and their biases, drift in inputs and in the model, and what to do when offline and online disagree.

Production proxies for quality

Once a change is live, the offline set stops being the only evidence. Production gives you far more volume than any labelled set, but none of it is labelled with correctness. What you get instead are proxies: behaviour that tends to move with quality. Harbourline's help-centre assistant sees about 13,000 sessions a day, the claims-intake agent about 6,000 claims. Each proxy answers a different question, and each is wrong in a known way.

Signal	What it tends to indicate	How it misleads
Thumbs up or down	Customer's satisfaction with one answer	Rare, self-selected, and rewards pleasant over correct
Escalation to a human	The assistant could not or would not resolve it	Rises when the assistant correctly declines; falls when it wrongly refuses to hand off
Re-ask or rephrase	The first answer missed	Also fires when the customer is exploring; silent when they give up
Edit of a drafted message (agent assist)	Staff corrected the output	Staff edit for style and habit, and stop editing under time pressure
Task completion	The customer reached the end state	Needs a defined end state; a completed wrong change counts as success
Containment (no human needed)	Cost saving	Rewards deflection, including customers who left without being helped
CSAT survey	Overall satisfaction with the session	Response bias, and it scores the whole journey including human rescue

Pick signals by the harm you are guarding against. For the customer-service agent, the dangerous failure is an unconfirmed or wrong write, and no satisfaction proxy sees it: a customer whose address was changed correctly and one whose address was changed wrongly both say thanks. Those need direct outcome checks such as a confirmation-step record, a reversal within seven days, or a complaint that cites the change. Use soft proxies for the help-centre assistant and hard outcome records for the write path.

Who clicks, and who does not

Every explicit feedback signal is a sample of the people willing to give it. Thumbs ratings typically come from a small share of sessions, and the share is not random: it over-represents people who are annoyed and people who are delighted, and under-represents the large middle who got a fine answer and left. A rating rate that falls is as likely to mean a changed interface as changed quality. A satisfaction survey sent after a handoff measures the human agent as much as the assistant.

The practical rule is to never compare a feedback rate across periods without checking the response rate and the mix of who responds. Report the denominator next to every proxy. When a proxy is the only evidence for a decision, estimate the bias directly: take a random sample of sessions, send them for expert review regardless of whether the customer clicked, and compare the reviewed quality of clickers and non-clickers. If the two differ by more than the effect you are tracking, the proxy cannot see that effect.

Sampling production for human and judge review

The way to get unbiased quality numbers from production is to sample and grade, not to wait for signals. Two samples serve different purposes, and the second one is the one teams skip.

A uniform random sample, a few hundred sessions a week, graded by humans against the same rubric as the offline set. This gives an unbiased quality estimate with an interval, and it is the baseline against which every proxy is calibrated.
A targeted sample of sessions that look risky: escalations, re-asks, low judge scores, low-confidence retrievals, citation-check failures, and every write action on the customer-service agent. This is where failures are dense, and it feeds the loop back into the eval set.

An LLM judge can grade far more sessions than humans can, and is the right tool for the targeted stream and for trend lines. Calibrate it against the human random sample on the same rubric, and report agreement with a chance-corrected statistic as covered in chapter 04, rather than raw percent agreement. Re-check the calibration whenever the judge model or the production distribution changes; a judge that agreed with humans in spring may not agree on storm-season claims.

Drift in inputs and in the model

Production quality moves for two reasons that look alike on a dashboard and need opposite responses. Inputs change, or the system changes.

Input drift. The questions customers ask shift: a new product launches, a regulation changes wording, a storm season sends flood and wind queries from a small share to a large one. The model is the same, but the mix of easy and hard questions is not. Track the distribution of topic, slice, language and policy tier week by week. A common summary is the population stability index over category shares; as a rule of thumb from credit-risk practice rather than a standard, below 0.1 is stable and above 0.25 deserves investigation. Treat those cut-offs as a starting alert level and tune them on your own history.

System drift. A hosted model can change under a stable name, a retrieval index is rebuilt, a corpus document is replaced, a downstream service changes a response format. Chen, Zaharia and Zou (2023) showed behaviour shifting between releases of the same hosted models, which is why pinning matters and why even a pinned system needs a monitor: your own index and tools drift too. The strongest defence is a fixed canary set, a few dozen representative items with known good answers, run against production on a schedule. If that set's score moves while inputs and code did not, something in the system moved.

A fixed probe separates the two kinds of drift

Production metrics move when inputs change and when the system changes. A scheduled run of a frozen item set moves only when the system does. Having both lets you subtract one from the other, which is the only reliable way to tell a hard quarter from a worse model.

Alerts and dashboards

A dashboard is for looking; an alert is a decision made in advance about when someone should be woken. Alert on the few signals tied to harm, with a threshold, a window and a named owner, and keep everything else on a dashboard. For Harbourline that looks like this.

Page-worthy: citation-resolution rate below a floor; any write action without a confirmation record; judge-flagged coverage statements above a limit on the claims-intake agent; an error or timeout rate step.
Ticket-worthy: escalation or re-ask rate moving beyond a control band for several days; topic-mix PSI over the investigation level; the frozen probe set dropping by more than its run-to-run noise.
Dashboard only: thumbs rate, CSAT, containment, average length, cost per session.

Slice every dashboard by the same dimensions as the eval set: topic slice, product, jurisdiction, policy tier, model version, prompt version and canary arm. A blended rate is the same trap as an aggregate accuracy, and the aggregate is exactly where a mix shift hides. At 13,000 sessions a day nearly any difference is statistically significant, so rely on effect size and a control band built from your own weekly variation, not on a p-value alone.

When offline and online disagree

Harbourline had a quarter where CSAT held flat while escalations climbed from 9.1% to 11.0% of sessions, and the offline suite had not moved. Both views looked healthy and unhealthy at once. The useful move is to work through the explanations in a fixed order, cheapest and most common first.

Did anything ship? Check model id, prompt, tool schema, index snapshot and corpus changes against the date the metric moved. Most "mysteries" end here.
Did the measurement change? A new escalation button, a survey change, a logging bug. Compare the metric's definition and its denominator across the break.
Did the inputs change? Compare topic and slice distributions, then compare the metric within each slice. This is the step that usually settles it.
Did the system change without a deploy? Run the frozen probe set now and compare with its history.
Is the offline set stale? Sample production, grade it, and compare with the offline score. If production quality is lower than offline, the set no longer represents what customers ask.

Step 3 is a standardisation: apply last quarter's within-topic rates to this quarter's topic mix and see how much of the change the mix explains.

mix_shift.py

import math

share_q2 = [0.34, 0.22, 0.28, 0.14, 0.02]   # limits, exclusions, procedures, definitions, storm
share_q3 = [0.27, 0.21, 0.27, 0.12, 0.13]
rate_q2 = [0.070, 0.140, 0.080, 0.050, 0.350]  # escalation rate within each topic
rate_q3 = [0.066, 0.130, 0.075, 0.047, 0.300]

def overall(shares, rates):
    return sum(s * r for s, r in zip(shares, rates))

def psi(expected, actual, eps=1e-6):
    return sum((a - e) * math.log((a + eps) / (e + eps)) for e, a in zip(expected, actual))

print("Q2 observed          %.1f%%" % (100 * overall(share_q2, rate_q2)))
print("Q3 observed          %.1f%%" % (100 * overall(share_q3, rate_q3)))
print("Q3 mix, Q2 rates     %.1f%%" % (100 * overall(share_q3, rate_q2)))
print("PSI of topic mix     %.2f" % psi(share_q2, share_q3))

It prints 9.1%, 11.0%, 12.1% and a PSI of 0.23 (illustrative inputs). Storm questions grew from 2% to 13% of sessions, and they escalate at about a third of the time because they involve claims the assistant should hand off. The mix alone would have lifted escalations to 12.1%; the observed 11.0% means the assistant escalated slightly less within every topic. Nothing was broken. CSAT held for the same reason: handed-off customers were served by people. The decision that follows is not a rollback but a new slice. Add storm and flood questions to the eval set, and set the escalation alert against a per-topic baseline, not the blended rate.

The advice flips when step 3 explains none of the change: if within-topic rates rose in every slice, the system moved, and steps 1 and 4 are where to dig.

Closing the loop

An eval set is a snapshot of what you knew when you built it. Production is the only source of what you did not know, so build the path from one to the other and keep it routine. When a reviewer, a judge or a customer complaint identifies a failure, the sequence is the same each time:

Reproduce it: capture the question, the retrieved chunks, tool calls and the model, prompt and index versions.
Label it: write the correct answer or reference trajectory, and tag the slice and cost.
Remove personal data before it enters the set; policy numbers, names and addresses never belong in a labelled item.
Add it to the set through the same pull request review as any other item, with a link to the incident.
Confirm it fails on the current system, fix the system, and confirm it passes. Then it is a regression test for good.

Set a rate for it: for instance, every incident reviewed produces an eval item within a week, and the targeted sample from the previous section adds a fixed number each week. Watch two things as the set grows. Items drawn from failures make the set harder than production, so keep the uniform-sample estimate as the honest quality number and treat the failure-derived items as a regression suite. And re-balance slices, since the set can quietly become all storm questions. The mechanics of building and stratifying the set are in chapter 02. The ship-decision framing for the interview version is in the interview guide's agent evaluation chapter, and the trajectory side of the agents' loop is in chapter 06.

✓ Check yourself

Answer from memory before opening each one.

Thumbs-up rate on the help-centre assistant falls from 4.1% to 3.2% of sessions, the week after a redesign moved the thumbs buttons below the fold. Has quality dropped? What would you check?

Show answer
You cannot tell. The change is confounded with how easy it is to respond. Compare the share of sessions that gave feedback and the mix of those who did, and read a uniform random sample graded by humans for an unbiased quality figure. See who clicks.
The customer-service agent changes addresses, and customers rate those sessions highly. Why is that not evidence the write path is safe, and what do you track instead?

Show answer
A correct change and a wrong one both get thanked, since the customer cannot see the error yet. Track the outcome directly: confirmation records for every write, reversals within a window, and complaints that cite the change. See production proxies.
Escalations rise two points while the offline suite is unchanged and CSAT is flat. List the first three things you check, in order, and say what each rules out.

Show answer
First, whether anything shipped or any index or corpus changed, which rules out a self-inflicted regression. Second, whether the metric's definition or denominator changed, which rules out a measurement artefact. Third, the topic mix and the metric within each topic, which separates input drift from system change. See when offline and online disagree.
A frozen set of 40 items scored 0.88 for six weeks and now scores 0.80, while production topic mix is stable. What does that suggest, and what caveat applies to a 40-item set?

Show answer
Because inputs to the probe are fixed, a move points to the system: model, index, tool or corpus. But 40 items means a wide interval, so compare against the run-to-run noise you measured over the six weeks before acting. See drift.
The team wants to set the escalation alert at the blended rate used last quarter. Argue for or against.

Show answer
Against: a blended rate moves with the topic mix, so seasonal storm traffic will trigger it without any change in quality, and a real regression in one slice can hide behind a stable aggregate. Alert per slice against its own baseline and band. The case for blended is a very small team that can only maintain one alert, and then it should be paired with a mix-shift alert. See alerts.

Takeaway

Production signals are proxies, each biased in a known way; choose them by the harm you guard against and use direct outcome records for the write path.
Explicit feedback comes from a self-selected minority, so report the response rate beside every rate and calibrate against a graded random sample.
Sample production two ways: uniformly for an unbiased quality estimate, and targeted at risky sessions for failure density.
Input drift and system drift look alike on a dashboard; a frozen probe set and a per-slice breakdown separate them.
When offline and online disagree, check what shipped, what the metric measures, and the input mix within slices, in that order, before blaming the model.
Every production failure becomes a labelled eval item through a reviewed, routine path, and the uniform sample stays the honest quality number.

That completes the guide's path from what an eval is for to keeping it honest in production. → All chapters