Section C · Decisions

Model and Prompt Changes

Regression suites for prompt edits and model upgrades, pinning and canaries, shadow traffic, what changes silently between versions, and the rollback you rehearse before you need it.

What changes silently

A model swap looks like a one-line config change, and the headline benchmark usually improves. What you ship is not the benchmark. It is the behaviour your prompt, parser and downstream rules were tuned against, and a new version changes that behaviour in places nobody announces. Harbourline's help-centre assistant is the example: suppose its model is deprecated and the vendor names a successor. The questions that decide the migration are not "is the successor smarter" but "what did it change that our system depends on".

The usual silent changes, in rough order of how often they bite:

Tokenisation. A new tokenizer changes token counts for the same text, which changes cost, latency and how much fits in a context window. Anthropic's models overview states that 1M tokens is about 555k words on the tokenizer introduced with Claude Opus 4.7, against about 750k words for earlier models (as of November 2026). That is roughly a third more tokens for the same policy wording. A prompt that stuffed five chunks to a token budget now fits fewer, and the bill moves with it.
Refusal and hedging behaviour. A model that now adds "please consult your policy documents" to every deductible answer can fail a correctness grader and raise escalations while being technically right.
Formatting. Markdown, list style, how citations are rendered, whether JSON arrives wrapped in a code fence. Parsers fail here before graders do.
Tool-call style. How eagerly the model calls tools, how it orders calls, how it phrases arguments. For the claims-intake and customer-service agents this is the largest risk; chapter 06 covers the trajectory checks that catch it.
Latency and length. Longer answers and more reasoning per turn change time to first token and total cost per session.
Request parameters. Anthropic's deprecations page lists temperature, top_p and top_k as removed in API version 2026-10-21, with a 400 error when sent. Code that pinned determinism through sampling settings can fail loudly or quietly lose its determinism, depending on the model and API version. Check the page for the exact rules.

Providers also change a model that keeps its name. Chen, Zaharia and Zou (2023) re-ran the same prompts against two dated releases of the same hosted models and found large shifts: GPT-4's accuracy on a prime-versus-composite task fell from 84% in March 2023 to 51% in June, formatting errors in generated code rose, and willingness to answer sensitive questions fell. The lesson is not that any one vendor drifts. It is that "same model name" is a hypothesis you test, not a fact you assume.

Pinning, and what it buys

Pin the exact model identifier in configuration, never a moving alias. The reason is causal: when production quality moves, you need to know whether you changed something. Anthropic states that every Claude API model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on; for older models an alias is a convenience pointer that resolves to a dated ID. OpenAI's deprecations page shows an alias and its dated snapshot retired together. So pinning gives you stable behaviour for a while, and it does not extend the model's life.

Pinning also has limits you should not oversell to a reviewer. It fixes the weights you call. It does not fix the platform in front of them: a cloud partner platform may set its own retirement schedule, the API version may change accepted parameters, and a hosted system prompt or safety layer may change independently. Record in every eval run the model id, API version, prompt hash, tool schema hash, retrieval index snapshot and sampling settings. When two runs disagree, that record is what lets you say why.

The regression suite per change

Every change to a model, prompt, tool schema or retriever runs the same suite, and the suite is the eval set from chapter 02 plus the checks from earlier chapters. For the help-centre assistant that means the roughly 400 labelled items scored per slice with the graders of chapter 03, the citation checks from chapter 07, and format, length and latency assertions. For the agents it means the roughly 150 claims trajectories against reference tool sequences.

The comparison must be paired. Both versions answer the same items, and the unit of analysis is the item, not the pass rate. A change can improve the aggregate while breaking items that used to pass, and the broken items are what a reviewer and a customer care about. Count the discordant pairs: items the old version passed and the new fails (regressions), and items the new passes that the old failed (fixes). An exact McNemar test asks whether the two directions are equally likely.

paired_change.py

import math

def exact_mcnemar(b, c):
    """b: old passes, new fails. c: new passes, old fails.
    Two-sided exact binomial test under H0: both are equally likely (p = 0.5)."""
    n, k = b + c, min(b, c)
    p = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * p)

old_pass = 330                     # of 400 items: 82.5%
b, c = 17, 21                      # regressions, fixes
print("old", old_pass / 400, "new", (old_pass - b + c) / 400)
print("discordant pairs", b + c, "p =", round(exact_mcnemar(b, c), 3))

This prints 82.5% to 83.5% on 38 discordant pairs with p of about 0.63. Four things follow. The new version is not shown to be better. It is also not shown to be worse. The 17 regressions still deserve reading item by item, because a handful on the exclusions slice matter more than 21 fixes on definitions. And 400 items cannot resolve a one-point difference at all, which is the arithmetic behind the stopping rules in chapter 05. Set the bar before the run: for instance, no slice may lose more than a stated number of items, and zero regressions are allowed on the items tagged high-cost.

Run each item more than once when outputs are stochastic, and report the spread. If sampling controls are not available for the new model, run-to-run variance is something you measure rather than set to zero. A difference smaller than your own repeat noise is not a result.

Shadow and canary traffic

The offline suite tells you about the questions you thought of. Real traffic contains the ones you did not, so the second gate is production, in two stages.

live request ──▶ current model ──▶ answer shown to customer │ └─────▶ candidate model ──▶ answer logged, NOT shown (SHADOW) │ └─▶ online checks: citations resolve, format valid, length, latency, judge score vs current answer then: 1–5% of sessions get the candidate's answer (CANARY) compare escalation, re-ask, thumbs, task completion by arm

Shadow sends a copy of each request to the candidate and discards the answer after scoring it. It costs tokens but exposes no customer. It is where you run the cheap, deterministic online checks plus an LLM judge that compares the two answers pairwise, with the position-bias and verbosity controls from chapter 04. Shadow cannot measure what the customer does next, because the customer never saw the candidate's answer.

Canary routes a small random share of live sessions to the candidate, so outcome signals exist. Size it from the effect you need to detect. If escalation runs near 10% and a 2-point rise is the smallest harm you would act on, a two-arm comparison at 5% significance and 80% power needs roughly 3,500 sessions per arm. At 13,000 sessions a day, a 5% canary yields about 650 candidate sessions a day, so about five and a half days. Assign by customer, not by message, so one conversation does not mix versions, and exclude sessions where the customer has seen both.

The advice flips for the agents. A shadow run of the customer-service agent must stub the write tools: executing an address or payment change twice, once for each model, is real harm. Run shadow with read tools live and write tools recorded but not executed, and compare the proposed write against the production agent's. Canary for write-capable agents should start with confirmation-required actions only, which is what Harbourline already enforces.

Prompt diffs reviewed like code

A prompt edit is a behaviour change with no compiler. Treat it as code: keep the prompt in a file in the repository, change it in a pull request, and let continuous integration run the regression suite on the branch. The reviewer needs two diffs, not one. The first is the text change. The second is the output change: the list of items whose graded result or answer text differs between the base and the branch, with before and after side by side. The output diff is where reviewers find that a "clarify tone" edit quietly dropped the jurisdiction caveat.

Keep edits small and separable. A pull request that rewrites the system prompt, adds two examples and bumps the model gives you no way to attribute a movement. If you must change model and prompt together, because the new model needs different instructions, run the matrix: old model with old prompt, old model with new prompt, new model with old prompt, new model with new prompt. The four cells tell you which change did what, and they cost four suite runs, not forty.

Rollback you have rehearsed

A rollback plan that has never run is a hope. Make the model id, the prompt version and the tool schema version configuration values that a flag flips, not code that needs a deploy. Keep the previous version deployable and keep the previous model reachable for the whole canary period, which matters because a retired model cannot be rolled back to. Define the trigger in advance in measurable terms: a named online metric crossing a named threshold for a named window, for example citation-resolution rate below a floor for 30 minutes.

Then rehearse it. Once a quarter, flip the staging flag back and forth under load and record the time to revert and whether in-flight sessions, caches and the logging pipeline behaved. The rehearsal finds the real problems: a cache keyed without the model id that serves the old answers to the new model's sessions, a conversation store whose state format changed, a dashboard that stops tagging the version. Rolling back an agent mid-conversation is the hard case, since half the tool calls were planned by one model. Decide whether sessions finish on the version that started them.

Deprecation timelines

You do not always choose the date. Vendor policies as of November 2026:

Vendor	Stated notice	How you find out
Anthropic	At least 60 days before retirement for publicly released models	Email to customers with active deployments, and the model deprecations page
OpenAI	At least 6 months for generally available models, 3 months for specialised variants, as little as 2 weeks for preview models, unless safety or compliance requires faster	Email to active users, the deprecations page, and a `shutdown_date` field on the models endpoint

Anthropic's page shows what 60 days looks like in practice: one Claude model was marked deprecated on 30 September 2026 with retirement on 30 November 2026. That is the whole window for building the comparison set, running the suite, canarying and finishing the change. Partner clouds set their own schedules, so a model can live longer on one platform than another. Work backwards from the retirement date: the suite must already exist before the notice arrives, because two months is not enough time to build an eval set and also migrate. Poll the lifecycle data your vendor exposes and alert on a retirement date inside your longest migration time. If the notice is long, spend the slack on a second candidate, not on waiting.

Migration checklist

For the help-centre assistant moving to a new model version, in order:

Record the baseline: model id, API version, prompt hash, index snapshot, and the current suite results per slice.
Re-count tokens on a sample of real prompts. Update context budgets, chunk counts and the cost model.
Run the offline suite twice on each side. Compare paired, per slice, and read every regression on the high-cost slices.
Check structured outputs, citation format and tool-call schemas with deterministic assertions.
Fix the prompt in a separate pull request, with its own output diff, and re-run the four-cell matrix if both changed.
Shadow for a few days; review the pairwise judge results and the disagreements, sampling disagreements for human review.
Canary at a small share sized to the effect you care about; watch escalation, re-ask rate and completion by arm.
Rehearse the rollback with the flag, then ramp. Keep the old version reachable until the retirement date is far away or you no longer need it.
Write the new baseline, and add every canary failure to the eval set (see chapter 09).

✓ Check yourself

Answer from memory before opening each one.

The candidate model scores 83.5% on the 400-item help-centre set against 82.5% for the current one. A colleague wants to ship. What do you compute before answering, and why does the aggregate not settle it?

Show answer
Count items that regressed and items that were fixed, and test the discordant pairs. Here 17 versus 21 gives p of about 0.63, so no difference is shown, and the aggregate hides which items broke. Read the regressions on the high-cost slices. See the regression suite.
After a model upgrade, correctness graders pass the same answers as before, yet context-window errors and the monthly bill both rise. What changed, and what should have caught it?

Show answer
Most likely tokenisation: the same text now costs more tokens, so fewer chunks fit and spend rises. A token-count check on real prompts and a cost-per-session assertion in the suite would have caught it. Quality graders do not see it. See what changes silently.
You want to shadow-test the customer-service agent's new version on live traffic. What must be true of the write tools, and what do you compare instead of outcomes?

Show answer
Write tools must be recorded, not executed, or the change happens twice. You compare the candidate's proposed tool calls and arguments with the production agent's, since the customer never sees the candidate's reply and so no outcome exists. See shadow and canary.
A change bumps the model and rewrites the system prompt in one pull request, and escalations rise. How do you attribute the rise?

Show answer
Run all four combinations of old and new model with old and new prompt on the suite, and on a canary if the offline suite does not reproduce it. The cells show whether the model, the prompt or their interaction moved the metric. See prompt diffs.
The vendor gives 60 days' notice that your model retires, and you have no regression suite. Do you spend the first month building the suite or evaluating candidate models? Argue for one.

Show answer
Build a small suite first, since a candidate cannot be judged without it, and seed it from logged production questions and the highest-cost slices. Start candidate runs as soon as 100 items exist, in parallel with growing it. The deciding factor is the cost of error: with write tools or coverage answers, a thin suite is not a basis to ship, and you should ask the vendor or platform about extended availability. See deprecation timelines.

Takeaway

A model or prompt change moves tokenisation, refusal behaviour, formatting, tool-call style, latency and parameters that your graders may not see; test for each, not only for answer accuracy.
Pin exact model ids and record the full run configuration; pinning stabilises behaviour but does not extend a model's life.
Compare versions item by item and test the discordant pairs; a one-point aggregate move on 400 items is noise, and regressions on high-cost slices decide.
Shadow traffic checks the candidate without exposing customers; canary traffic measures outcomes and needs thousands of sessions per arm to see a two-point change.
Prompts live in the repository, change by pull request and are reviewed as output diffs; change model and prompt separately or run the four-cell matrix.
Rollback is a flag, a written trigger and a rehearsal; vendor notice periods range from weeks to months, so the suite must exist before the notice does.

Once a change is live, the question becomes how you notice quality moving when nobody changed anything. → Online Signals and Drift