A Batching System for LLM Queries

The question, and what is being scored

The sentence is deliberately ambiguous, and the first thing a strong candidate does is say so. Two different systems answer it. One is application-level: you hold work in a queue and send it to a model in bulk, because nobody is waiting. The other is inside the inference server: the engine groups many in-flight requests into one GPU forward pass, because the GPU is wasted otherwise. A good answer covers both, says which one the interviewer's workload needs, and does not let one leak into the other.

Interviewers score four things:

• Segmentation. You split traffic by how long the caller can wait before choosing any mechanism.
• Mechanism at both layers. You can explain why provider batches are cheap and slow, and why continuous batching raises throughput and costs latency.
• Failure handling. Partial failure, expiry, retries and idempotency are designed in, not bolted on.
• Arithmetic and SLOs. You size the cost for a stated volume and give separate latency targets for interactive and batch work, including what happens under overload.

The running example is Harbourline's nightly job that classifies and summarises the day's claims: about 6,000 new claims a day, to be ready before adjusters start work. The same company also serves about 40,000 chat sessions a day, which is the contrast case.

Two layers, one rule: batching trades latency for efficiency

Every form of batching spends waiting time to buy efficiency. What differs is who waits and what is bought.

The rule that keeps the design honest: never put a batch window in front of a person. A provider batch can take up to 24 hours. A window you add to a chat path, even 200 ms, lands directly on time to first token. Interactive traffic gets the server layer only, where the "window" is a single scheduler step. Deferred traffic gets the application layer.

For Harbourline that gives three classes. Chat and the claims-intake agent are interactive: a person or a phone call is waiting. The adjuster's "re-summarise this file" button is near-interactive: a few seconds is fine, minutes is not. The nightly classification and summary is deferred: the deadline is 06:00, not "now". Only the third class goes to a batch API.

Provider batch APIs: what the contract actually is

Both major providers sell the same deal. You trade a completion guarantee measured in hours for half the price and a separate rate-limit pool. The details matter when you design the retry logic, so know them. All figures below are from the vendors' documentation as of November 2026.

Three consequences follow from that contract.

Cost is per request, not per batch. Anthropic does not bill errored, canceled or expired requests. Retrying them is therefore free of double-charging, but the retry costs you wall-clock time. Order is not preserved, so a deterministic custom_id is the only join key you have. Anthropic keeps results downloadable for 29 days and limits it to 1 to 64 characters of letters, digits, hyphens and underscores, which is enough for claim-8841207-v3-20261104. Prompt caching is best-effort inside a batch. Requests run concurrently and in any order, so a cache write by one request may not exist yet when another runs. The Anthropic docs suggest the one-hour cache duration for batches that share a long prefix.

Two smaller limits catch people out. A batch cannot stream, and Anthropic notes that processing can slow with demand and your own volume, which shows up as more requests expiring. Treat the 24-hour figure as a ceiling that your own deadline must beat, not as a service level.

Server-side: what continuous batching buys and what it costs

If you self-host, the question moves inside the engine. Start from the bottleneck: during decode, each generated token requires reading the model's weights from GPU memory once. If you decode one sequence at a time, that read is amortised over one token. If you decode 64 sequences together, the same read serves 64 tokens. That is the whole throughput argument for batching, and it is why a lightly loaded GPU is an expensive GPU.

The older approach, static batching, collected N requests, padded them, and returned all N when the longest finished. With variable output lengths, 19 finished sequences wait idly for the 20th. Orca (Yu et al., OSDI 2022) fixed this with iteration-level scheduling: the scheduler decides the batch membership at every forward pass, so a finished sequence leaves and a waiting one joins immediately. vLLM combined that with PagedAttention (Kwon et al., SOSP 2023), which makes joining and leaving cheap because KV-cache memory is allocated in blocks; the authors report 2 to 4 times the throughput of the prior systems at the same latency. The PagedAttention deep dive walks through the scheduler loop step by step. This chapter only needs the consequences.

What it buys. No window and no padding. A new request starts at the next iteration, so under light load its time to first token is close to the unbatched case, and under heavy load throughput climbs without the caller doing anything.

What it costs. Each iteration now does the work of every running sequence, so the time between tokens rises with batch size. Prefill is the sharper problem: a long prompt processed in one go stalls everyone's decode. vLLM's V1 engine turns on chunked prefill by default, schedules pending decodes first, and fills the rest of the per-step token budget with prefill chunks. The knob is max_num_batched_tokens. Per the vLLM docs, smaller values such as 2048 give better inter-token latency, and larger values give better time to first token, with throughput best above 8192. That is the latency-throughput dial in one parameter. When the KV cache fills, vLLM preempts and recomputes requests, which shows up as latency spikes. The cure is more cache memory or a lower max_num_seqs, not more patience.

Harbourline's chat traffic averages about 0.5 sessions a second. A single replica does not need a deep batch for that, so the right setting favours inter-token latency over peak throughput. The inference gateway guide covers the layer above the engine: coalescing identical in-flight prompts into one upstream call, and when not to.

The worked answer: Harbourline's nightly claims job

Say the structure aloud, in this order: segment the traffic, give each class its path, state SLOs, size the cost, then walk the failures.

SLOs. Interactive traffic gets latency targets: a p95 time to first token and a p95 total time per surface. These are budgets you set; the next chapter shows how to measure them. Batch traffic gets a completion deadline and a success ratio instead: "at least 99.5% of claims summarised by 06:00, the rest visibly marked". Mixing the two in one SLO hides the real problem, because a batch job that finishes in 50 minutes tells you nothing about chat.

Back-pressure. On the interactive path, put a bounded queue in the gateway and reject early when it fills, with a 429 and a Retry-After, rather than let the engine's waiting queue grow until time to first token is ten seconds. Shed the lowest class first. On the batch path, back-pressure is the scheduler itself: it submits one batch of 6,000 requests, not 6,000 calls, and it never submits a second night's batch while the first is unreconciled.

Cost. Assume, and say you are assuming, 12,000 input tokens and 600 output tokens per claim. That is 72 million input and 3.6 million output tokens a day. Using standard prices as of November 2026:

The batch discount saves $90 a day on the larger model. Picking the smaller model saves more, but that is a quality decision, and it belongs to the eval set, not to the cost sheet: see the eval-set chapter. A sensible shape is a cascade: the small model classifies every claim, the larger one summarises only those routed as complex. The 12,000-token prompt probably carries a shared instruction block; if 3,000 tokens are identical across claims, a one-hour cache write could save about $17 a day at best, because batch cache hits are best-effort. Worth turning on, not worth designing around.

Partial failure and idempotency

A batch of 6,000 does not succeed or fail. It produces 6,000 individual outcomes, and the design question is what happens to each.

• succeeded: write the result keyed by custom_id. Applying twice must be harmless, so write with an upsert, not an insert.
• errored with an invalid-request error: the payload is bad. Retrying the same request fails again. Log it, count it, and fix the input (a 300-page document over the limit, a malformed attachment).
• errored with a server error: transient. Put the claim back in the pending state for the next submission.
• expired or canceled: never sent to the model, not billed. Resubmit, or fall back to the synchronous path if the deadline is close.

Idempotency comes from the ledger, not the provider. Derive custom_id from the claim ID, the prompt version and the run date, so the same logical job always has the same key. Record the batch ID in the ledger the moment the API returns it, and on restart look the ledger up before submitting. If the process dies between "API accepted the batch" and "ledger updated", you will have an orphan batch; a reconciliation step that lists recent batches in the workspace and matches them to ledger rows closes that gap. I would not assume the create call de-duplicates for you.

import time
import anthropic

client = anthropic.Anthropic()
CUTOFF = 4 * 3600          # seconds after submission; our deadline, not the provider's
PROMPT_VERSION = "v3"

def custom_id(claim_id: str, run_date: str) -> str:
    return f"claim-{claim_id}-{PROMPT_VERSION}-{run_date}"   # <= 64 chars, [A-Za-z0-9_-]

def submit(claims: list[dict], run_date: str, ledger) -> str:
    requests = [
        {
            "custom_id": custom_id(c["id"], run_date),
            "params": {
                "model": "claude-sonnet-5-5",
                "max_tokens": 800,
                "system": c["system_prompt"],
                "messages": [{"role": "user", "content": c["claim_text"]}],
            },
        }
        for c in claims
    ]
    batch = client.messages.batches.create(requests=requests)
    ledger.record_batch(batch.id, [r["custom_id"] for r in requests])  # before anything else
    return batch.id

def reconcile(batch_id: str, started: float, ledger) -> None:
    while time.time() - started < CUTOFF:
        if client.messages.batches.retrieve(batch_id).processing_status == "ended":
            break
        time.sleep(60)
    else:
        client.messages.batches.cancel(batch_id)       # leftovers go to the sync path
    for entry in client.messages.batches.results(batch_id):
        kind = entry.result.type
        if kind == "succeeded":
            ledger.upsert_summary(entry.custom_id, entry.result.message)
        elif kind == "errored":
            ledger.mark_error(entry.custom_id, entry.result.error)  # split by error type
        else:                                           # expired or canceled
            ledger.mark_pending(entry.custom_id)

The sketch leaves out pagination of the claim list, the sync fallback and metrics. In the interview, name those omissions rather than pretend they are not there.

Interview probes

Expect these after your first pass. Answer aloud before opening each one.

✓ Check yourself

Answer from memory before opening each one.

1. A colleague says, "batching always adds latency, so we should avoid it on the chat path." Where is the claim right, and where is it wrong?

   Show answer

   It is right for application-level windows and provider batches, which add waiting time directly. It is wrong for server-side continuous batching, where a new request joins at the next iteration, so under light load the added wait is about one scheduler step. The cost shows up under load, as slower inter-token time. See two layers and continuous batching.

2. Results from a 6,000-request batch come back with the second request first. Your code zips results against the input list. What breaks, and what is the fix?

   Show answer

   Summaries get written against the wrong claims, which is silent and serious in insurance. Providers return results in any order, so the only safe join is a deterministic custom_id. See provider batch APIs.

3. A vLLM replica serves chat well until adjusters upload 300-page PDFs for summarising, and then chat users see long pauses mid-answer. Explain the mechanism and name the setting you would adjust.

   Show answer

   A long prefill occupies the iteration, and every running decode waits for it. Chunked prefill splits the prompt and schedules decodes first, and max_num_batched_tokens sets how much prefill fits per step; a smaller value protects inter-token latency at the cost of slower first tokens for the long prompt. Better still, move the PDF work to a separate pool. See continuous batching.

4. The nightly job's cost is $90 a day on the larger model. Finance asks you to cut it in half. List the options in the order you would try them, and say what each risks.

   Show answer

   First, trim the input: 12,000 tokens per claim is probably carrying text the model does not need, and this risks losing facts. Second, route simple claims to the smaller model, which risks quality on edge cases and needs the eval set to decide. Third, cache the shared instruction block, which saves little and is best-effort in a batch. Moving to self-hosting is last, since it adds fixed cost. See the worked answer.

5. The tempting design is one retry loop: any non-success result goes back into the next batch. Why is that wrong for errored results?

   Show answer

   An invalid-request error is deterministic: the same payload fails every night and the loop never ends. Only transient server errors, expired and canceled requests are safe to retry. Invalid requests need a fix and an alert. See partial failure.

Takeaway

• "Batching" means two systems: application-level queues and provider batch APIs trade hours of waiting for half the price, while server-side continuous batching trades a little inter-token latency for GPU utilisation.
• Segment traffic by how long the caller can wait before choosing a mechanism, and never put a batch window in front of a person.
• A provider batch yields thousands of individual outcomes in any order, so the design needs a deterministic custom_id, a ledger, upserts and a separate path for each failure type.
• The provider's 24-hour expiry is a ceiling; your own deadline sets the real cut-off, with a synchronous fallback for what is unfinished.
• In an engine like vLLM the batch-size knobs set a latency-throughput dial, so size them to the p95 inter-token target and watch preemptions, not to peak throughput.
• At 6,000 claims a day the API beats self-hosting; the case flips on data residency, sustained volume, or a model you must fine-tune.

Batching is one lever for cost and latency, and not the first one to pull. → Latency and Cost at Scale

Further reading

• Anthropic: Batch processing — limits, pricing, result types and the 29-day retention. Good for the retry design.
• OpenAI: Batch API guide — file format, limits, the separate error file. Good for comparing contracts.
• Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., OSDI 2022. The origin of iteration-level scheduling.
• Efficient Memory Management for LLM Serving with PagedAttention — Kwon et al., SOSP 2023. Why paged KV memory makes continuous batching practical.
• vLLM: Optimization and tuning — chunked prefill, max_num_batched_tokens and preemption. Good for the latency-throughput dial.