Section B · Technical Core · Critical

Error Handling & Failure Modes

What goes wrong on a GPU fleet — CUDA OOM, NCCL hangs, NVLink failures, kernel panics, queue overflow — how you detect each, and the recovery patterns that keep the system going.

A taxonomy of GPU failures

Interviewers love debug questions because real production experience leaks through. The candidates who've owned a fleet reach for known buckets; the ones who haven't pattern-match on generic SRE answers. The taxonomy:

Layer	Typical failures
Hardware	HBM ECC, GPU off bus (Xid 79), thermal throttling, power glitch, NVLink degraded, NIC degraded
Driver / firmware	Xid storms, GSP RPC timeouts (Xid 119/120), driver/runtime mismatch, fabric manager crash
CUDA / app	OOM, illegal memory access (Xid 31), kernel assertion, deadlock
Collectives	NCCL hang at init, NCCL timeout mid-run, ring vs tree fallback, P2P disabled
Orchestration	Device plugin gone, pod stuck pending, gang-scheduling deadlock, evictions cascading
Network / fabric	Subnet manager flap, congestion, NIC port down, RDMA not negotiated
Storage	Slow weight pull, NFS hang, NVMe full, Lustre client stuck
Serving	Queue overflow, KV-cache exhaustion, model partially loaded, deadlock in batch

Knowing the buckets is half the answer.

CUDA out-of-memory

The most common and most varied. Diagnose by when it happens.

When	Likely cause	Fix
At model load	Weights don't fit at this precision / TP shape	Reduce precision, increase TP, smaller model
On first request	Activation memory + KV reservation too big	Reduce `max-model-len`, lower `gpu-memory-utilization`
Under load, after running fine	KV-cache pressure as concurrency rose	Cap `max-num-seqs`; consider FP8 KV-cache
Mid-training step N	Activation memory peaks; fragmentation; growing batch	Gradient checkpointing, memory snapshot, smaller microbatch
Random / intermittent	Fragmentation; allocator caching pathology	PyTorch `expandable_segments`, restart on memory pressure

# Triage: where did memory go?
nvidia-smi --query-gpu=memory.used,memory.free --format=csv
# Inside the process:
python -c "import torch; print(torch.cuda.memory_summary())"
# Time-series view (one second resolution):
dcgmi dmon -e 252,140  # FB_USED, FB_FREE

vLLM-specific

vLLM pre-allocates a fixed KV-cache pool at startup based on --gpu-memory-utilization. If you OOM at startup, lower that number. If you OOM mid-request you've configured something else wrong — vLLM should preempt to disk rather than crash if there's a swap config.

NCCL hangs & collective failures

NCCL is the collective communication library — all-reduce, all-gather, broadcast — across GPUs. Hangs are the worst class of failure because they don't error; they just stop.

Hang at init

Usually a network reachability or topology problem. Set NCCL_DEBUG=INFO and re-run; you'll see which rank is waiting for which peer.

Firewall blocking the NCCL ports.
NCCL_SOCKET_IFNAME picking a management NIC instead of the high-speed fabric.
IB link down (ibstat).
One rank's pod still pulling its image while others wait.

Hang mid-training

One rank crashed or stalled; the others wait for it in an all-reduce. NCCL 2.18+ has watchdog timers (NCCL_ASYNC_ERROR_HANDLING=1, TORCH_NCCL_BLOCKING_WAIT=1) that error out instead of hanging forever. Always set them in production.

Throughput cliff after a "successful" job

NCCL fell back from ring/IB to tree/TCP. The job runs but at 1/10 the bandwidth. Watch the NCCL INFO output for "via TCP" — that's the signal.

NVLink & fabric failures

NVLink errors show up as Xid 74. Sometimes one link of N is degraded and the GPU keeps running at reduced bandwidth — silently.

Detection — DCGM exposes per-link error counters. nvidia-smi nvlink -e shows link errors.
NVSwitch failures — fabric manager logs are first stop. journalctl -u nvidia-fabricmanager.
Action — drain the node, run diagnostics (NVIDIA Field Diagnostic), often RMA.

Inter-node fabric (IB/RoCE) failures look like a node "running slower than usual." Same drill: check link state, check switch port counters, check the subnet manager.

Driver / runtime mismatches

The most common new-cluster outage. Symptoms:

CUDA driver version is insufficient for CUDA runtime version on app startup.
Pods come up but nvidia-smi inside fails or shows fewer GPUs than expected.
Device plugin healthy but kubelet reports zero allocatable GPUs.

Causes:

Driver upgraded on host, kernel module not reloaded.
nvidia-container-toolkit mismatched with driver.
Container has newer CUDA than host driver supports.
UVM module not loaded after driver upgrade (needs reboot).

Fix: drain, reboot if needed, verify nvidia-smi on host, restart device plugin DaemonSet, smoke test.

Kernel panics & node death

NVIDIA drivers run privileged kernel modules. A bad interaction can panic the kernel. The node disappears from K8s.

K8s node controller marks NotReady after the kubelet heartbeat fails.
Pods on the node are eventually evicted (with grace; PDB-respecting workloads need attention).
The blast radius is everything that was running on the node. A training job with one rank on that node is dead unless it had checkpointing.

Resilience moves:

Checkpoint training to durable storage every N steps.
Replicas ≥ 2 for any online service; never colocate them on one node.
Pod anti-affinity by node, ideally by rack/AZ.
Aggressive node health gates (NPD — node-problem-detector — to auto-cordon on Xid bursts).

Queue overflow & backpressure

Inference servers have queues. Queues fill. When they fill, you either reject, slow down, or melt down. The right answer is reject loudly, fast, at the edge.

Hard concurrency limit at the gateway. Beyond it: 429 with Retry-After.
Per-tenant rate limits so one team can't starve others.
Queue time SLO — if a request has been queued > X ms, drop it (the user has already given up).
Load shedding on saturation — start dropping low-priority traffic before high-priority traffic suffers.
Circuit breakers — when a backend is melting, route around it.

Senior signal

"My instinct is to make failure modes boring: predictable 429s with a clear retry policy, rather than wandering p99 and gateway timeouts. Backpressure earlier is almost always better than 'just queue it.'"

Request timeouts

LLM requests can be long. Defaults of 30 seconds are wrong. Set timeouts deliberately:

Connect timeout — short (1-2s). Network problem, fail fast.
First-token timeout — based on TTFT SLO + slack (e.g. 5s).
Total request timeout — based on max output length × per-token budget. For a 2k-token output at 50 tok/s, that's 40s minimum; set 60s with margin.
Streaming heartbeat — if no tokens received in N seconds during streaming, abort. Detects stuck decoders.

Partial model loading

Distributed serving (TP > 1) means N processes each load a shard. If process 3 fails to load, processes 0/1/2 wait at the synchronization barrier and time out. Symptoms:

vLLM hangs at startup with no useful log.
One rank's process exits, others hang.
K8s pod readiness never goes true.

Fix patterns:

Aggressive startup probe timeouts; let K8s kill the pod and restart together.
NCCL watchdog on init.
All ranks in one pod (vLLM does this for intra-node TP) — the failure becomes a single process exit, not a hang.
Pre-fetch weights to local NVMe before pod start so loading is fast and deterministic.

Recovery patterns

You'll be asked "how do you recover" for several of the above. Have these patterns ready by name.

Restart with backoff

Crash → restart → exponential backoff if it keeps failing → eventually quarantine the node. The default for most things. Kubernetes restartPolicy: Always + CrashLoopBackOff handles 80% of cases.

Drain and replace

Suspect node: kubectl cordon, then kubectl drain with PDB respect. Replacement node comes online; ASG/MIG/MachineSet brings up new capacity. Use for Xid storms, driver suspicion, NIC errors.

Circuit breaker

Front of an unreliable backend, track error rate. Above threshold, fail fast for a cooldown window rather than piling on. Half-open probe traffic on recovery. Standard pattern; doubly important in front of GPU services because melt-down is worse than fail-fast.

Bulkhead

Isolate failure to a single tenant or replica pool. If team A's model is broken, only team A's queue overflows, not the cluster.

Graceful shutdown / drain on preemption

SIGTERM → stop accepting new requests → finish in-flight up to a budget → flush metrics → exit. Critical for spot/preemptible.

Checkpoint & resume (training)

Async snapshot every N steps to durable storage. On restart, load latest and continue. Sharded checkpoint formats (PyTorch DCP, DeepSpeed) so 1000-GPU jobs can checkpoint in seconds.

Idempotency for inference retries

If a request fails mid-stream, can the client retry safely? Three cases:

Pure inference, no side effects — retry is safe. Two responses with the same prompt are at most "different sampled outputs," and at a fixed seed/temperature they're identical.
Inference with tool calls — the tool may have side effects. Need idempotency keys; the agent layer (not infra) usually handles this.
Cached responses — same prompt → same cached output, naturally idempotent.

Operationally:

Accept a client-provided request ID at the gateway.
De-dupe on the request ID within a short window.
For streaming, an interrupted stream is hard to resume; document that the contract is "retry from scratch."
Don't bill twice for retried requests that failed before the first token.

Interview-grade closing

"My design assumption is that any individual GPU, node, or replica can die without warning. The system has to absorb that — retries with idempotency, graceful drains, circuit breakers in front of melting backends, and bounded queues so failure shows up as predictable 429s rather than wandering tails."