Latency and Cost at Scale

The question, and what is being scored

The weak answer is a list: caching, a smaller model, shorter prompts. The strong answer is a procedure. You measure first, you rank the levers by payoff on this workload, you quantify each, and you say which ones you refuse to touch yet. Interviewers score four things:

• Measurement before action. You name the metrics and the percentiles, and you know that time to first token and total time are different problems.
• Mechanism per lever. You can say why prefix caching cuts both cost and time to first token, why output tokens dominate total time, and why speculative decoding helps at low load.
• Ordering by payoff. The cheap, safe, large wins go first. You know what is last and why.
• Correctness risk. You see that some savings, semantic caching above all, buy speed by risking a wrong answer, and you price that risk by surface.

The running example is Harbourline's web chat: about 40,000 sessions a day, a customer-service agent with tools for policy lookup, payment status and address changes. The agent design itself is in the customer-service chapter. This chapter asks what it costs to run and how fast it feels.

Measure first: what, at which percentile

A model call has two phases with different physics. In prefill, the engine processes the whole prompt in parallel, and the cost grows with prompt length. In decode, it generates one token at a time, and the cost grows with output length. That gives the identity you should be able to write on a whiteboard:

Prompt length moves the first term. Output length moves the second, and for chat answers it is usually the larger one. Output tokens also cost more: on Sonnet 5.5 an output token is five times the price of an input token ($10 against $2 per million, as of November 2026). So shorter outputs improve both numbers, and a shorter prompt mainly improves time to first token and the input bill.

Record these per surface, per model and per tool, not as one global average:

• p50, p95 and p99 of time to first token and of total time. The average hides the users who complain. Sessions have several turns, so the tail compounds: with six turns and a p95 that is slow one time in twenty, about 26% of sessions (1 − 0.95⁶) contain at least one slow turn.
• Inter-token time, for streamed answers. It tells you whether the stream looks smooth.
• Cost per session, computed from the usage fields each response returns: uncached input, cache-written input, cache-read input and output tokens, each at its own price.
• Cache hit ratio: cache-read tokens divided by all input tokens. A drop after a release is a bug report.

PRICES = {  # USD per million tokens, as of November 2026
    "claude-sonnet-5-5": {"in": 2.00, "cache_write_5m": 2.50, "cache_read": 0.10, "out": 10.00},
    "claude-haiku-5-5":  {"in": 0.10, "cache_write_5m": 0.125, "cache_read": 0.01, "out": 0.50},
}

def call_cost(model: str, usage: dict) -> float:
    """Cost of one call from the `usage` object on a Messages API response."""
    p = PRICES[model]
    return (
        usage.get("input_tokens", 0) * p["in"]
        + usage.get("cache_creation_input_tokens", 0) * p["cache_write_5m"]
        + usage.get("cache_read_input_tokens", 0) * p["cache_read"]
        + usage.get("output_tokens", 0) * p["out"]
    ) / 1_000_000

# One chat turn: 4,000-token stable prefix, 1,500 variable tokens, 250 output tokens.
cold = {"input_tokens": 5500, "output_tokens": 250}
warm = {"input_tokens": 1500, "cache_read_input_tokens": 4000, "output_tokens": 250}
print(f"cold {call_cost('claude-sonnet-5-5', cold):.5f}  warm {call_cost('claude-sonnet-5-5', warm):.5f}")
# cold 0.01350  warm 0.00590

Anthropic's own advice on latency is worth repeating in an interview: first get a prompt that performs well without constraints, then optimise. Trimming too early hides what top quality looks like, and you cannot judge a cheaper variant without that reference.

The levers, in order of payoff

The order below is for Harbourline's chat workload. The ranking is workload-specific: a job with short prompts and long outputs would reorder it. The percentages come from the budget table later in this chapter, built on stated assumptions.

Batching belongs on the list for the non-interactive half of the company, and it is the one lever with a price tag attached: half price for work that can wait. That is the subject of the previous chapter and is not repeated here.

Prefix caching: the biggest lever and the easiest to break

The model's key and value tensors for a prompt prefix depend only on that prefix. If the provider has already computed them for an identical prefix, it can skip that prefill work. Anthropic's prompt caching exposes this: you mark a breakpoint, the first call writes a cache entry, later calls with the exact same prefix read it. A read of Sonnet 5.5 costs 0.05 times the base input price, and a five-minute write costs 1.25 times (as of November 2026). Every use refreshes the five-minute lifetime for free. One-hour entries cost twice the base price to write.

The break-even is fast. A write costs 0.25 base extra and each read saves 0.95 base, so a single reuse pays for the write. For Harbourline's chat, the stable prefix is the system prompt, policy-handling rules and tool definitions: about 4,000 tokens, identical on every turn of every session. At roughly 0.5 sessions a second the entry never goes cold.

import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=400,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT_AND_RULES,            # stable: identical on every request
        "cache_control": {"type": "ephemeral"},     # breakpoint: cache everything up to here
    }],
    messages=conversation,                          # variable: history and new user turn
)
u = resp.usage
hit_ratio = u.cache_read_input_tokens / max(
    1, u.cache_read_input_tokens + u.cache_creation_input_tokens + u.input_tokens)

Caching is a prefix match over the order tools, then system, then messages. A change at one level invalidates that level and everything after it. That makes the failures predictable, and they are all the same bug: something that varies sits in front of something that does not.

• A timestamp, a request ID or the customer's name at the top of the system prompt. Move it into the user turn.
• Tool definitions reordered, or serialised with unstable key order, on each deploy. Anthropic's docs call out non-deterministic key ordering in tool inputs as a cache-breaker.
• A prefix shorter than the minimum cacheable length (512 tokens for Sonnet 5.5 and Haiku 5.5, 4,096 for Haiku 4.5). The call succeeds and caches nothing, so only the usage fields tell you.

Routing, output discipline and shorter prompts

Model routing. Most turns in a customer-service chat are easy: "has my payment gone through", "confirm the new address". A small model handles them well, and Haiku 5.5 costs a twentieth of Sonnet 5.5 per input token ($0.10 against $2). The router can be rules (intent from the UI button the customer pressed), a small classifier, or the small model itself deciding to escalate. The mechanism that matters is the asymmetry of errors: sending a hard case to the small model costs quality, sending an easy one to the large model costs money. Harbourline's cost of a wrong answer differs by surface, so the router is conservative where it hurts. Anything touching a coverage decision goes to the larger model whatever the router thinks. Check routing quality with the same eval set you use for ship decisions, per route. See the eval-set chapter.

Output discipline. Decode time is linear in output length, so the cheapest speed-up for total time is a shorter answer. Ask for it in the prompt in sentences or paragraphs; Anthropic notes that word-count limits work less well than sentence limits because models count tokens, not words. Use max_tokens as a guard rail, not a style control: hitting it truncates mid-sentence. On models that expose an effort setting, lower effort also shortens hidden reasoning, which counts as output.

Shorter prompts. Once the prefix is cached, the stable part of the prompt costs 5% of its size, so trimming it is nearly worthless. What remains worth trimming is the variable part: tool results. A policy-lookup tool that returns a 3,000-token JSON document when the model needs the deductible and the effective date is the usual offender. Have the tool return what the agent needs. The token-budget discipline is covered in the claims-agent chapter.

Streaming, parallel tool calls and speculative decoding

Streaming does not make the answer arrive sooner. It makes the first words arrive sooner, which is what a person perceives, so time to first token becomes the number that matters. It has a cost: anything that must inspect the whole answer before showing it, such as a grounding check or a policy-wording validator, cannot sit behind a stream. Harbourline streams help-centre answers and does not stream anything that states a coverage position. Those are buffered, checked, then shown, and the extra seconds are accepted.

Parallel tool calls. A model can request several tools in one turn. For a customer who asks "did my payment go through and what's my renewal date", policy lookup and payment status are independent. Run them concurrently and you save a full model round trip compared with calling them one after another. The API leaves execution order to you, which is the catch: reads can run in parallel, but an address change that must follow a verification step cannot. Make side-effecting tools sequential and independent reads concurrent.

Speculative decoding applies only when you run the model yourself. A small draft model proposes several tokens, the large model checks all of them in one forward pass, and a rejection-sampling step keeps the longest correct prefix. Leviathan, Kalman and Matias (2022) show the output distribution is unchanged, with a 2 to 3 times speed-up on T5-XXL. The vLLM documentation positions it for medium-to-low request rates where decode is memory-bound, supports draft models, EAGLE, multi-token prediction, n-gram and suffix methods, and says real gains depend on model, traffic and hardware. My reading of the mechanism: at high concurrency continuous batching already keeps the GPU busy with useful work, so extra verification compute competes with it and the gain shrinks. Test it against your own traffic. The PagedAttention deep dive has the scheduler view. For a hosted API it is not a lever you hold.

Regional endpoints shorten the network leg and may be required for data residency, but providers can price in-region processing differently and the saving is tens of milliseconds, not seconds. Choose them for compliance first. Confirm the current terms with your provider before putting a number in a design.

Semantic caching: fast, cheap and wrong in the worst place

A semantic cache embeds the incoming question, finds a prior question within a similarity threshold, and returns the stored answer without calling the model. The latency is a vector lookup and the cost is close to zero. The mechanism of the risk is that embeddings measure topical closeness, not equivalence of meaning for this customer. "What is my deductible for water damage?" from two policyholders embeds almost identically and has two different correct answers. "Can I cancel my policy?" and "Can I cancel my claim?" are one word apart.

Where Harbourline would use it: public, non-personalised help-centre questions ("how do I upload photos of the damage") with an entry lifetime tied to the document version, and a threshold chosen from a labelled set of paired questions, not by feel. Where it never would: anything personalised or touching coverage, where a wrong deductible is a complaint and a wrong coverage position is a regulatory issue. For personalised surfaces, an exact-match cache keyed on the full normalised request including policy ID and policy version is safe, and rarely hits.

The worked answer: a budget table for 40,000 sessions a day

State the assumptions out loud, since the interviewer will test them. Each session has six model turns. Each turn sends 4,000 stable-prefix tokens plus 1,500 variable tokens (history and tool results) and returns 250 output tokens. That is 33,000 input tokens and 1,500 output tokens per session, or 1.32 billion input and 60 million output tokens a day. Prices are Sonnet 5.5 and Haiku 5.5 list prices as of November 2026.

The shape is the answer: two levers remove nearly four fifths of the bill, and the last one is worth a few percent. Each assumption is where the interviewer will push. The caching row is an upper bound: every point of hit rate lost costs about $23 a day (9.6 million tokens at the difference between write and read price). The routing row depends on the 60% share, which is an eval result, not a guess. A missed routing decision costs quality; a missed cache costs money.

For latency, write the budget the same way and mark it as a target you propose, not a benchmark: p95 time to first token under 1.5 seconds and p95 total under 8 seconds on the web chat; tighter for voice, which has its own chapter. Then attribute: retrieval and tool calls typically take a fixed slice, prefill takes a slice that caching shrinks, and decode takes the remainder in proportion to output tokens. Measure the split before choosing the lever.

What not to do first

• Fine-tune or distil a smaller model. It works, but costs weeks and a retraining loop, while routing to an existing small model costs a day.
• Self-host to save money. At this volume you buy GPUs, on-call and an inference stack to save a bill that caching and routing already cut by four fifths. It flips at sustained high volume or under a residency rule.
• Add a semantic cache. It is the lever with a correctness cost, and it is worth the least once prefix caching is on.
• Cut max_tokens to shorten answers. It truncates; it does not summarise.
• Optimise without a per-surface measurement. You will fix the average and miss the surface that complains.

Interview probes

Answer aloud before opening each one.

✓ Check yourself

Answer from memory before opening each one.

1. A change halves the average prompt length of a chat feature, but users report that long answers still feel slow. Why did the change not fix that?

   Show answer

   Prompt length drives prefill, and so time to first token. Long answers are dominated by decode, which scales with output tokens. A shorter prompt cannot speed that up. See measure first.

2. You enable prefix caching and the usage fields show zero cache reads and zero cache writes. Name two causes unrelated to a code bug in your caching call.

   Show answer

   The prefix is shorter than the model's minimum cacheable length, so the request is processed uncached with no error. Or the cache is invalidated by content before the breakpoint changing on each call, which writes a new entry each time. Zero writes points to the first cause. See prefix caching.

3. A customer asks, "Is flood damage covered on my policy?" A semantic cache returns a stored answer to a nearly identical question from a different customer. Describe the failure and the smallest design change that prevents it.

   Show answer

   Similar wording does not imply the same coverage, and the stored answer was grounded on the other customer's policy. The cache skipped retrieval and validation. Keying the cache on policy ID, jurisdiction and policy version prevents cross-customer hits, but for coverage questions I would not use a semantic cache at all. See semantic caching.

4. Traffic doubles and you consider a more aggressive batch size on a self-hosted server, plus speculative decoding. Which of the two is more likely to help at the new load, and why?

   Show answer

   Batching, as higher concurrency fills the batch and raises throughput per GPU. Speculative decoding targets memory-bound, lower-load decode, where spare compute can verify drafts; at high load that spare compute is gone. The inter-token latency cost of the larger batch still needs a check. See speculative decoding and continuous batching.

5. The budget shows caching and routing removing most of the cost. Finance asks why you did not start by fine-tuning a small model. Give the deciding factors, and the condition under which you would reconsider.

   Show answer

   Fine-tuning needs labelled data, training runs, evaluation and a maintenance loop, while the cheaper levers need days and no new model. I reconsider when the remaining bill is still large after those levers, the task is narrow and stable, and an eval shows a tuned small model matches the larger one. See what not to do first.

Takeaway

• Time to first token depends on the prompt and prefill; total time depends mostly on output tokens, so you must measure both, per surface, at p95.
• Prefix caching is the largest and cheapest lever, and it is only as good as your discipline about keeping the stable prefix first and identical.
• Routing easy turns to a small model removes most of the remaining cost, provided an eval set, not intuition, decides which turns qualify.
• Streaming improves perceived speed, not total time, and cannot sit in front of validators that need the whole answer.
• Semantic caching trades correctness for speed and does not belong on personalised or coverage surfaces.
• At 40,000 sessions a day, caching plus routing plus output discipline cut a list-price bill of $3,240 a day to about $569 before any model is trained or hosted.

Cost and latency change once the input is no longer a short text turn: a 2 GB video does not fit any of the prompts above. → A Large-File Uploader for a Multimodal Chat

Further reading

• Anthropic: Prompt caching — pricing multipliers, minimum lengths, breakpoints and what invalidates a cache. Good for the cache-bug diagnosis.
• Anthropic: Reducing latency — model choice, prompt and output length, streaming, and the advice to optimise after quality.
• Anthropic: Parallel tool use — execution semantics and when to run tools concurrently.
• Anthropic: Pricing — list and cache prices behind the budget table. Check the date before quoting.
• Fast Inference from Transformers via Speculative Decoding — Leviathan, Kalman and Matias, 2022. The method and its exactness argument.
• vLLM: Speculative decoding — supported methods, configuration keys, and the load regime it targets.