Deep Dive — Inference Serving

Why LLM serving is hard (different from generic ML serving)

A classic ML service is request-in, prediction-out, fixed compute per request. LLM serving has three properties that change the engineering shape:

1. Two distinct phases per request — prefill (process the prompt, compute-bound, parallelizable) and decode (generate one token at a time, memory-bandwidth-bound, sequential).
2. Variable, unknown output length. A request might emit 5 tokens or 2,000. You can't pre-allocate exactly.
3. A large per-request state — the KV-cache — that lives on the GPU for the entire generation.

Together these make naive batching catastrophic. If you batch eight requests of unknown length, the batch waits for the slowest. GPU goes to 30% utilization. Throughput collapses.

KV-cache, the central object

Transformer attention at decode time uses K and V projections from all previous tokens. Recomputing them each step would be quadratic. So we cache them per layer per head per sequence in GPU memory. The KV-cache:

• Grows linearly with sequence length and batch size.
• Dominates memory at long contexts. For Llama-3 70B FP16, ~320 KB per token per sequence at TP=1. At 8k context, batch 16: ~40 GB of cache alone.
• Is freed when the sequence finishes.
• Can be shared across requests with the same prefix (prefix caching).

vLLM — paged attention & continuous batching

vLLM (Kwon et al., 2023) reshaped open-source LLM serving with two ideas working in concert.

Paged attention

Treat the KV-cache like virtual memory. Carve cache space into fixed-size blocks (e.g. 16 tokens each). A sequence's KV-cache is a list of block IDs, not a contiguous slab. Benefits:

• Near-zero fragmentation. Naive allocators reserve max-context per sequence; paged attention only allocates what's used.
• Copy-on-write sharing. Two requests with the same prefix point at the same blocks until they diverge.
• Predictable admission. The scheduler knows exactly how many free blocks remain.

Continuous batching

Also called iteration-level batching or in-flight batching. Instead of batching at request granularity, batch at token-step granularity. Each forward pass, the scheduler:

1. Looks at active sequences.
2. Adds new requests if there's capacity.
3. Removes finished sequences.
4. Runs one decode step on whoever is in the batch.

Effect: the GPU is never idle waiting for a slow request, and new requests don't wait in line for a "batch window" to fill.

# Typical vLLM server invocation
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --max-num-seqs 256 \
  --kv-cache-dtype fp8

The knobs you'll defend in an interview:

• --gpu-memory-utilization — what fraction of GPU memory vLLM may use for weights + cache + activations. Higher = more concurrency, lower = more safety margin.
• --max-num-seqs — concurrency cap. Too high and tail latency explodes; too low and you under-utilize.
• --enable-prefix-caching — huge win for system-prompt-heavy workloads (most enterprise apps).
• --kv-cache-dtype fp8 — halve cache memory, small quality cost; very common production choice.

Triton Inference Server

NVIDIA's general-purpose serving engine. Not LLM-specific. It excels when you have a portfolio of models, possibly mixed framework, and want one server to host them.

Concepts

• Model repository — filesystem layout of models with config.pbtxt per model.
• Backends — TensorRT, ONNX Runtime, PyTorch, Python, TensorRT-LLM, vLLM (yes, you can wrap vLLM inside Triton).
• Dynamic batching — Triton batches incoming requests up to a max batch size and a max queue delay.
• Model ensemble — chain models server-side (e.g. tokenizer → model → detokenizer).
• Model analyzer — sweeps batch sizes and instance counts to find the perf sweet spot.

# Triton model repository layout
models/
  llama3-70b/
    config.pbtxt
    1/
      model.engine        # TensorRT-LLM plan
  embedding-bge/
    config.pbtxt
    1/
      model.onnx
  ensemble-rag/
    config.pbtxt          # ties tokenizer + retriever + LLM
    1/

Use Triton when you need: multi-model multi-tenant on the same node, mixed frameworks, dynamic batching for non-LLM models, or you're already on the NVIDIA stack and want one unified control plane.

TensorRT & TensorRT-LLM

TensorRT is NVIDIA's inference compiler. You feed it a graph (ONNX or torch); it produces a plan file tuned for the target GPU. The wins:

• Kernel fusion — adjacent ops compile into single kernels, dropping launch overhead and HBM round-trips.
• Quantization — INT8/FP8 with calibration; small quality cost, big throughput win.
• Layer auto-tuning — for each layer the compiler benchmarks several kernel variants and picks the fastest for your GPU. Plan files are not portable across SM versions.

TensorRT-LLM is the LLM-specific extension: paged KV-cache, in-flight batching, FP8, AWQ/SmoothQuant, speculative decoding, tensor-parallel support. Usually deployed behind Triton with the TensorRT-LLM backend.

KServe

KServe is a Kubernetes-native serving abstraction. It adds CRDs (InferenceService) that hide the pod/service/HPA plumbing. You declare what to serve; KServe materializes the runtime.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-70b
spec:
  predictor:
    minReplicas: 2
    maxReplicas: 8
    scaleTarget: 50
    scaleMetric: concurrency
    model:
      modelFormat: { name: huggingface }
      runtime: kserve-huggingfaceserver
      storageUri: s3://kraken-models/llama3-70b/
      resources:
        limits:
          nvidia.com/gpu: 4

KServe gives you canary traffic splitting, autoscaling (including scale-to-zero with Knative), and a uniform "deploy a model" UX. Good ergonomics for an internal platform; the cost is another abstraction layer to debug.

Ray Serve

Ray Serve is the right choice when your inference is a graph of Python computation, not "model in, model out." Use cases:

• RAG: retriever → reranker → LLM, each at different replica counts.
• Agent loops with tool calls.
• Heterogeneous models that must compose with shared state.

Ray Serve handles the deployment-graph plumbing, autoscaling per node, and Python-native development. It runs on top of Ray, so you're carrying a cluster manager.

TorchServe

The PyTorch-native option. Simpler than Triton, less optimized for LLMs than vLLM, but a reasonable choice for traditional PyTorch models (vision, embeddings) when you want minimal extra stack. Less commonly chosen for greenfield LLM workloads in 2025-2026 unless you're already deep on PyTorch tooling.

Parallelism strategies

When a model doesn't fit on one GPU — or fits but you want lower latency — you split it. Three strategies, often combined:

Speculative decoding

Decode is sequential — one token per forward pass. Speculative decoding breaks that limit. Run a small "draft" model fast, propose K tokens, then have the big model verify them in one forward pass (cheap because verification is parallel). Accepted tokens are kept; first rejected token starts the next round.

• Typical speedup: 2-3× on aligned workloads.
• Quality is bit-exact to the target model (it's a sampling trick, not an approximation).
• Cost: extra GPU memory for the draft model; extra complexity.
• Variants: Medusa (multiple decode heads), EAGLE, Lookahead decoding, n-gram speculation (cheapest, often best).

How to pick a serving stack

An interview-grade framework. State the workload, then map.

Latency vs throughput, made explicit

These two pull in opposite directions and every serving knob you turn lands somewhere on the curve.

• Larger batches → higher throughput (tokens/sec/GPU), worse per-request latency (you wait behind others' compute).
• Smaller batches → better latency, lower throughput, lower utilization.
• Continuous batching moves you along this curve much more efficiently than static batching.
• Tensor parallelism reduces latency at fixed throughput by spreading the compute, but adds collective overhead — diminishing returns past 4-8 way intra-node.
• Speculative decoding improves latency without sacrificing throughput, at the cost of memory and complexity.
• Quantization (FP8, INT4) improves both — fewer bytes moved, more cache fits — at the cost of measured quality.

✓ Check yourself

Answer from memory before opening each one.

1. Eight requests of unknown output length are batched together with static batching. Why does GPU utilisation fall, and what does continuous batching change?

   Show answer

   A static batch runs until its slowest sequence finishes, so slots freed by short sequences sit idle and new requests wait for the next window. Continuous batching decides at every decode step: drop finished sequences, admit waiting requests, run one step. The GPU stays full and arrivals do not queue for a batch to fill. See why LLM serving is hard.

2. Llama-3 70B in FP16 needs about 320 KB of KV-cache per token. Roughly how much cache do 16 sequences at 8k context need, and what does that tell you about memory settings?

   Show answer

   About 40 GB, since 16 times 8,192 tokens is about 131,000 tokens. Cache, not weights, then limits concurrency at long contexts. Raising the memory fraction or shrinking the cache with FP8 buys concurrency, and a concurrency cap set too high turns into evictions and tail latency. See KV-cache.

3. A chat service shares one long system prompt across all requests and runs out of KV-cache under load. Which two vLLM options do you reach for, and what does each cost?

   Show answer

   Prefix caching lets requests with the same prefix point at the same blocks, so the prompt is stored and prefilled once. An FP8 KV-cache halves cache memory at a small quality cost that you must measure. Prefix caching helps only when prefixes really repeat, so check the hit rate. See vLLM.

4. A 13B model fits on 2 GPUs of an 8-GPU node and you need four times the throughput. Do you raise tensor parallelism to 8 or add replicas?

   Show answer

   Add replicas. Data parallelism scales close to linearly because replicas do not communicate. Tensor parallelism adds an all-reduce per layer and gives diminishing returns past 4 to 8 GPUs, so use it only when the model does not fit or one replica cannot meet latency. Across nodes prefer pipeline stages to tensor splits. See parallelism strategies.

5. Why can speculative decoding cut decode latency without changing what the target model would have produced, and when does it help little?

   Show answer

   The draft model proposes several tokens and the target verifies them in one forward pass, which is cheap because verification runs in parallel. Accepted tokens match what the target would sample, so quality holds. When the draft is poorly aligned, acceptance is low and the draft cost and extra memory are wasted. See speculative decoding.

6. The same cluster must serve interactive chat and nightly batch scoring. One configuration or two pools, and what decides it?

   Show answer

   Usually two pools, because the knobs pull in opposite directions. Chat caps concurrency to protect time to first token and p95, while batch raises concurrency and accepts slow first tokens for throughput. One shared pool is defensible only when load is small enough that headroom for chat costs little. See latency versus throughput.

Takeaway

• LLM serving is hard because each request has a compute-bound prefill, a memory-bound decode, an unknown output length and a large per-request KV-cache.
• KV-cache management is most of LLM serving: paged attention, prefix caching, FP8 cache and eviction are all answers to the same memory question.
• Continuous batching decides at every decode step, so the GPU stays busy and new requests do not wait for a batch window.
• Pick the stack from the workload: vLLM for one LLM behind an API, Triton for many models and backends, TensorRT-LLM for peak NVIDIA throughput, Ray Serve for Python pipelines, KServe for the deploy-a-model experience.
• Data parallel replicas are the first move when the model fits. Keep tensor parallelism inside a node over NVLink and use pipeline stages across nodes.
• A TensorRT plan file is bound to the GPU type that built it, so build per node pool.
• Latency and throughput trade directly, and interactive and batch workloads usually need separate pools.

Serving decides what each GPU does, and the scheduler decides who gets which GPU. → Deep Dive — GPU Scheduling & Orchestration

Further reading

• Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., SOSP 2023. The vLLM paper: block-based KV-cache, near-zero fragmentation and copy-on-write sharing.
• Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., OSDI 2022. Where iteration-level scheduling, the basis of continuous batching, was introduced.
• vLLM documentation — vLLM project. Engine arguments, prefix caching, quantised KV-cache and parallelism settings.
• Triton Inference Server user guide — NVIDIA docs. Model repository layout, backends, dynamic batching and ensembles.
• TensorRT-LLM documentation — NVIDIA. Compiled LLM engines, in-flight batching, quantisation and the Triton backend.
• Fast Inference from Transformers via Speculative Decoding — Leviathan et al., ICML 2023. The draft-and-verify method and the argument that output distribution is unchanged.
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Shoeybi et al., 2019. The tensor-parallel split of attention and MLP blocks and its communication cost.