Deep Dive — Inference Serving

Why LLM serving is hard (different from generic ML serving)

A classic ML service is request-in, prediction-out, fixed compute per request. LLM serving has three properties that change the engineering shape:

1. Two distinct phases per request — prefill (process the prompt, compute-bound, parallelizable) and decode (generate one token at a time, memory-bandwidth-bound, sequential).
2. Variable, unknown output length. A request might emit 5 tokens or 2,000. You can't pre-allocate exactly.
3. A large per-request state — the KV-cache — that lives on the GPU for the entire generation.

Together these make naive batching catastrophic. If you batch eight requests of unknown length, the batch waits for the slowest. GPU goes to 30% utilization. Throughput collapses.

KV-cache, the central object

Transformer attention at decode time uses K and V projections from all previous tokens. Recomputing them each step would be quadratic. So we cache them per layer per head per sequence in GPU memory. The KV-cache:

• Grows linearly with sequence length and batch size.
• Dominates memory at long contexts. For Llama-3 70B FP16, ~320 KB per token per sequence at TP=1. At 8k context, batch 16: ~40 GB of cache alone.
• Is freed when the sequence finishes.
• Can be shared across requests with the same prefix (prefix caching).

vLLM — paged attention & continuous batching

vLLM (Kwon et al., 2023) reshaped open-source LLM serving with two ideas working in concert.

Paged attention

Treat the KV-cache like virtual memory. Carve cache space into fixed-size blocks (e.g. 16 tokens each). A sequence's KV-cache is a list of block IDs, not a contiguous slab. Benefits:

• Near-zero fragmentation. Naive allocators reserve max-context per sequence; paged attention only allocates what's used.
• Copy-on-write sharing. Two requests with the same prefix point at the same blocks until they diverge.
• Predictable admission. The scheduler knows exactly how many free blocks remain.

Continuous batching

Also called iteration-level batching or in-flight batching. Instead of batching at request granularity, batch at token-step granularity. Each forward pass, the scheduler:

1. Looks at active sequences.
2. Adds new requests if there's capacity.
3. Removes finished sequences.
4. Runs one decode step on whoever is in the batch.

Effect: the GPU is never idle waiting for a slow request, and new requests don't wait in line for a "batch window" to fill.

# Typical vLLM server invocation
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --max-num-seqs 256 \
  --kv-cache-dtype fp8

The knobs you'll defend in an interview:

• --gpu-memory-utilization — what fraction of GPU memory vLLM may use for weights + cache + activations. Higher = more concurrency, lower = more safety margin.
• --max-num-seqs — concurrency cap. Too high and tail latency explodes; too low and you under-utilize.
• --enable-prefix-caching — huge win for system-prompt-heavy workloads (most enterprise apps).
• --kv-cache-dtype fp8 — halve cache memory, small quality cost; very common production choice.

Triton Inference Server

NVIDIA's general-purpose serving engine. Not LLM-specific. It excels when you have a portfolio of models, possibly mixed framework, and want one server to host them.

Concepts

• Model repository — filesystem layout of models with config.pbtxt per model.
• Backends — TensorRT, ONNX Runtime, PyTorch, Python, TensorRT-LLM, vLLM (yes, you can wrap vLLM inside Triton).
• Dynamic batching — Triton batches incoming requests up to a max batch size and a max queue delay.
• Model ensemble — chain models server-side (e.g. tokenizer → model → detokenizer).
• Model analyzer — sweeps batch sizes and instance counts to find the perf sweet spot.

# Triton model repository layout
models/
  llama3-70b/
    config.pbtxt
    1/
      model.engine        # TensorRT-LLM plan
  embedding-bge/
    config.pbtxt
    1/
      model.onnx
  ensemble-rag/
    config.pbtxt          # ties tokenizer + retriever + LLM
    1/

Use Triton when you need: multi-model multi-tenant on the same node, mixed frameworks, dynamic batching for non-LLM models, or you're already on the NVIDIA stack and want one unified control plane.

TensorRT & TensorRT-LLM

TensorRT is NVIDIA's inference compiler. You feed it a graph (ONNX or torch); it produces a plan file tuned for the target GPU. The wins:

• Kernel fusion — adjacent ops compile into single kernels, dropping launch overhead and HBM round-trips.
• Quantization — INT8/FP8 with calibration; small quality cost, big throughput win.
• Layer auto-tuning — for each layer the compiler benchmarks several kernel variants and picks the fastest for your GPU. Plan files are not portable across SM versions.

TensorRT-LLM is the LLM-specific extension: paged KV-cache, in-flight batching, FP8, AWQ/SmoothQuant, speculative decoding, tensor-parallel support. Usually deployed behind Triton with the TensorRT-LLM backend.

KServe

KServe is a Kubernetes-native serving abstraction. It adds CRDs (InferenceService) that hide the pod/service/HPA plumbing. You declare what to serve; KServe materializes the runtime.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-70b
spec:
  predictor:
    minReplicas: 2
    maxReplicas: 8
    scaleTarget: 50
    scaleMetric: concurrency
    model:
      modelFormat: { name: huggingface }
      runtime: kserve-huggingfaceserver
      storageUri: s3://kraken-models/llama3-70b/
      resources:
        limits:
          nvidia.com/gpu: 4

KServe gives you canary traffic splitting, autoscaling (including scale-to-zero with Knative), and a uniform "deploy a model" UX. Good ergonomics for an internal platform; the cost is another abstraction layer to debug.

Ray Serve

Ray Serve is the right choice when your inference is a graph of Python computation, not "model in, model out." Use cases:

• RAG: retriever → reranker → LLM, each at different replica counts.
• Agent loops with tool calls.
• Heterogeneous models that must compose with shared state.

Ray Serve handles the deployment-graph plumbing, autoscaling per node, and Python-native development. It runs on top of Ray, so you're carrying a cluster manager.

TorchServe

The PyTorch-native option. Simpler than Triton, less optimized for LLMs than vLLM, but a reasonable choice for traditional PyTorch models (vision, embeddings) when you want minimal extra stack. Less commonly chosen for greenfield LLM workloads in 2025-2026 unless you're already deep on PyTorch tooling.

Parallelism strategies

When a model doesn't fit on one GPU — or fits but you want lower latency — you split it. Three strategies, often combined:

Speculative decoding

Decode is sequential — one token per forward pass. Speculative decoding breaks that limit. Run a small "draft" model fast, propose K tokens, then have the big model verify them in one forward pass (cheap because verification is parallel). Accepted tokens are kept; first rejected token starts the next round.

• Typical speedup: 2-3× on aligned workloads.
• Quality is bit-exact to the target model (it's a sampling trick, not an approximation).
• Cost: extra GPU memory for the draft model; extra complexity.
• Variants: Medusa (multiple decode heads), EAGLE, Lookahead decoding, n-gram speculation (cheapest, often best).

How to pick a serving stack

An interview-grade framework. State the workload, then map.

Latency vs throughput, made explicit

These two pull in opposite directions and every serving knob you turn lands somewhere on the curve.

• Larger batches → higher throughput (tokens/sec/GPU), worse per-request latency (you wait behind others' compute).
• Smaller batches → better latency, lower throughput, lower utilization.
• Continuous batching moves you along this curve much more efficiently than static batching.
• Tensor parallelism reduces latency at fixed throughput by spreading the compute, but adds collective overhead — diminishing returns past 4-8 way intra-node.
• Speculative decoding improves latency without sacrificing throughput, at the cost of memory and complexity.
• Quantization (FP8, INT4) improves both — fewer bytes moved, more cache fits — at the cost of measured quality.