Applied Patterns

Pattern: multi-tenant inference cluster

The most-asked design question. Several teams want to serve LLMs from a shared GPU fleet. How do you build it?

The senior answer is layered. Don't jump to a YAML.

1. Identify the tiers. Online interactive (low p99, paying customer paths), online async (Slack bots, internal tools), batch (evals, offline scoring). Each gets a different SLO and different priority class.
2. Pool the hardware by accelerator type (H100 vs A100 vs L40), not by team. Tenants are namespaces, not node pools.
3. Quota by team with borrowing. Each team has a nominal quota; idle quota is borrowable up to a cohort cap.
4. Isolation strategy per tier. Online interactive: full-GPU or MIG. Batch: time-sliced or MPS, fine to mix.
5. Routing layer in front. An OpenAI-compatible gateway that routes by model name to the right backend pool. Caching here (prompt cache, response cache for deterministic queries).
6. Observability per tenant. Cost dashboard, queue depth, p99 latency, GPU utilization — broken out by namespace.
7. Failure mode handling. A team's runaway job can't starve the cluster. Per-namespace queue caps; per-request timeouts; per-deployment max replicas.

Pattern: online vs batch — they belong on different pools

Trying to run interactive latency-sensitive serving and large batch evaluation jobs on the same GPUs at the same time is a classic mistake. They want opposite things:

Keep them physically separated unless capacity pressure forces a mix — and even then, run batch as low-priority preemptible on the online pool's idle headroom, not as a co-resident.

Pattern: dynamic batching tuning

For non-LLM workloads (and pre-vLLM LLM workloads), dynamic batching means: hold incoming requests up to a deadline or batch size, then run them together.

Two knobs:

• max_batch_size — the most requests one forward pass handles. Bigger = more throughput, more memory, more tail latency.
• max_queue_delay_microseconds — how long the first arrival waits for company. Bigger = better batching efficiency under load, worse latency under no load.

# Triton config.pbtxt excerpt
dynamic_batching {
  preferred_batch_size: [4, 8, 16]
  max_queue_delay_microseconds: 2000
  preserve_ordering: false
}
max_batch_size: 32
instance_group [
  { count: 2, kind: KIND_GPU }
]

How to tune. Sweep with Triton's model-analyzer or a custom load generator. Look at the joint plot of p95 latency vs throughput as you vary max batch and delay. Pick the elbow.

Pattern: autoscaling on the right signal

K8s HPA on CPU utilization is wrong for GPU services — CPU is rarely the bottleneck, and GPU pods have CPU-light shapes. You need custom metrics. Three options, increasing in quality:

1. GPU utilization (SM-active%) — exposed via DCGM. Better than nothing but lags burst arrivals.
2. Queue depth — pending requests in front of each replica. Faster signal, but you need a server that exposes it (vLLM does).
3. Throughput vs SLO — observed p95 latency vs target. Most predictive but more wiring.

In practice: scale-out on queue depth or latency p95, scale-in on sustained low GPU utilization with a long cooldown. Asymmetric thresholds avoid flapping.

# KEDA scaling on Prometheus queue depth metric
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama-scaler
spec:
  scaleTargetRef:
    name: vllm-llama
  minReplicaCount: 2
  maxReplicaCount: 16
  cooldownPeriod: 600           # slow scale-in for warm models
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        query: |
          avg(vllm_num_requests_waiting{deployment="vllm-llama"})
        threshold: "8"

Pattern: spot & preemptible GPU strategy

On-demand H100s are expensive. Spot/preemptible/savings plans can cut 40-70%. The architecture must absorb interruption.

• Where spot works — batch training with checkpointing, batch inference, async evaluation, dev environments.
• Where spot doesn't — synchronous low-latency inference, anything with hard SLAs.
• The hedged strategy — a baseline of reserved capacity sized for the floor of demand; spot for the peaks; on-demand as the failover.
• Checkpointing for training — every N steps, async to durable storage; recover from latest.
• Graceful drain — preempt warning hooks (AWS gives 2 minutes), drain pods, save state, mark node unschedulable.
• Capacity-aware retry — when spot reclaims, try alternate AZs / instance families before giving up.

Cloud accelerator pricing models you should name:

• On-demand — pay-as-you-go.
• Spot / preemptible — interruptible, cheap.
• Capacity blocks (AWS) — pre-purchased GPU-hours in a future window, useful for training plans.
• Savings plans / committed use (AWS/GCP) — discount for committing usage over 1-3 years.
• Reserved instances — capacity reservation, often tied to a region/zone.

Pattern: model warm-up & lazy loading

Cold start for a 70B model means: pull the weights from S3, decompress, load to GPU, compile/CUDA-init, warm the KV-cache allocator, run a few dummy forwards to JIT-compile kernels. End to end can be minutes. Three patterns to manage:

• Bake weights into the image — fastest start, biggest image (50-200 GB). Works when you have a few models.
• Sidecar weight loader from local NVMe cache — image is slim, weights pulled once per node and shared across pod restarts. Best general-purpose pattern.
• Lazy / on-demand load — load model only when first request arrives. Saves GPU memory; adds cold latency to the first user.

Then warm-up: send synthetic requests at startup before declaring ready. The pod's K8s readiness probe should not return 200 until warm-up completes.

Pattern: A/B model deployments & canary rollouts

Two distinct concerns:

A/B between models

Route a fraction of traffic to a candidate model, measure quality and latency, compare to the champion. Tooling: a routing layer (KServe, Istio, Envoy) with weighted routes. Logging side-by-side responses for offline eval.

Canary for infrastructure changes

Driver upgrade, kernel upgrade, CUDA upgrade, vLLM version bump. Roll one node, watch:

• Inference latency (p50/p95/p99).
• Token throughput per GPU.
• Error rate.
• GPU temperature, power, ECC counts.
• Dmesg Xid drip.

If clean after 24 hours, expand to 10% of fleet. Then 50%. Then 100%. Automated rollback on regression.

Pattern: driver / kernel upgrade rollouts

NVIDIA driver upgrades are high blast radius. A bad driver can poison every pod on a node.

1. Pin and version-control driver. The GPU Operator's driver container is your friend — it lives outside the host package manager.
2. Drain the node (cordon, evict pods with PDBs respected).
3. Roll the driver. Reboot if required (UVM module sometimes wants it).
4. Smoke test with a known-good workload before re-admitting.
5. Watch DCGM & Xids for 24h before declaring the upgrade healthy.
6. Stagger across availability zones so a bad driver doesn't take out the whole pool.

Prototypes you can build this week

Real practice closes more gap than reading. Three 60-90 minute builds:

1. vLLM in a container. Run vLLM serving a 7B model on a single GPU (rent an H100 hour from RunPod/Lambda/Vast.ai for ~$3). Tune --max-num-seqs and --gpu-memory-utilization. Measure TTFT and throughput with vllm benchmark_serving. You'll have hands-on grounding for every chapter-04 question.
2. DCGM exporter + Grafana. Deploy DCGM exporter, scrape with Prometheus, build a Grafana board showing SM utilization, memory used, power, temperature. You'll never feel lost on chapter-12 questions again.
3. Kueue queue with two priorities. Spin up a tiny K8s (kind / minikube) with NVIDIA device plugin (or just CPU resources for the demo), install Kueue, configure two priority classes, submit jobs, watch preemption. Even on CPU, the model of the scheduler is the same.