Core Fundamentals

Linux for GPU hosts

GPU nodes are Linux machines first. The unusual parts are NUMA topology, hugepages, kernel modules, and the device files. The fundamentals you must be fluent in:

• cgroups v2 & namespaces — how container isolation actually works. GPU access flows through bind-mounted device nodes (/dev/nvidia*) and a hook.
• NUMA topology — H100 nodes have multiple NUMA domains. PCIe lanes attach to specific sockets. Cross-NUMA traffic is slow. nvidia-smi topo -m shows the PCIe / NVLink topology; numactl --hardware shows the CPU side.
• Hugepages — RDMA and some CUDA paths benefit from hugepage backing. Misconfigured hugepages cause silent perf cliffs.
• Kernel modules — nvidia, nvidia_uvm, nvidia_drm, nvidia_peermem. A bad mismatch between the loaded module and userspace driver is a top-5 outage source.
• Systemd units — nvidia-persistenced keeps the driver warm; nvidia-fabricmanager sets up NVLink/NVSwitch.
• journalctl + dmesg — the first place driver and Xid errors land. You should reflexively check both during a GPU incident.

# The interview-grade Linux/GPU triage cheatsheet
nvidia-smi                          # are GPUs visible, healthy, temp/power sane?
nvidia-smi topo -m                  # NVLink / PCIe topology
nvidia-smi -q -d ECC,POWER,CLOCK    # detail per device
dmesg | grep -i -E 'nvidia|xid|nvrm' # driver-level errors
journalctl -u nvidia-fabricmanager  # NVSwitch fabric state
cat /proc/driver/nvidia/version     # loaded driver version vs userspace
lspci | grep -i nvidia              # PCIe enumeration sanity
numactl --hardware                  # NUMA layout
ibstat                              # InfiniBand link state (if applicable)

Networking and the fabric

GPU training is bottlenecked by interconnect bandwidth, not compute, more often than people expect. Three layers:

1. Intra-node — NVLink between GPUs on the same chassis (900 GB/s aggregate on H100 SXM). NVSwitch enables all-to-all at full bandwidth.
2. PCIe — host-to-GPU, NIC-to-GPU. Gen5 x16 ≈ 64 GB/s. GPUDirect RDMA lets a NIC DMA directly into GPU memory, bypassing the host.
3. Inter-node fabric — InfiniBand HDR/NDR (200/400 Gbps) or RoCE (RDMA over Converged Ethernet). NCCL uses this for cross-node all-reduce.

For inference, the fabric matters less unless you're doing tensor parallelism across nodes. For training, it's load-bearing.

Storage for model weights & checkpoints

A 70B-parameter model in FP16 is ~140 GB. A 405B model is ~810 GB. Sharded checkpoints during training generate terabytes per checkpoint. Three storage tiers:

• Object storage (S3, GCS, R2) — durable, cheap, slow first-byte. Source of truth for weights.
• Network filesystem (Lustre, Weka, FSx) — high parallel throughput, good for shared training scratch.
• Local NVMe — fastest, ephemeral. Used for staging weights into a serving pod, for KV-cache spill, for fast checkpoint write-then-flush.

A practical pattern for inference: bake or stage weights to local NVMe on pod start, then mmap them. Loading a 70B model over the network for every pod restart is a pager-class mistake.

Containers and the NVIDIA toolkit

Containers see GPUs because nvidia-container-toolkit injects device nodes and libraries via an OCI hook. The flow:

1. Container runtime (containerd, CRI-O, Docker) invokes the OCI hook.
2. Hook reads NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES.
3. Bind-mounts /dev/nvidia*, libcuda.so, and friends into the container namespace.
4. Container starts; nvidia-smi works inside.

The classic failure: the host driver is updated but the toolkit / device plugin doesn't restart cleanly, and new pods come up without GPUs. The drain-and-replace runbook lives in chapter 13.

Kubernetes basics for GPU workloads

K8s itself doesn't know about GPUs. The NVIDIA device plugin (a DaemonSet) advertises nvidia.com/gpu as an allocatable resource. A pod requesting nvidia.com/gpu: 2 gets two whole GPUs bound exclusively.

apiVersion: v1
kind: Pod
metadata:
  name: vllm-server
spec:
  nodeSelector:
    accelerator: nvidia-h100
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  priorityClassName: inference-online
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "meta-llama/Llama-3.1-70B-Instruct",
             "--tensor-parallel-size", "4"]
      resources:
        limits:
          nvidia.com/gpu: 4
          memory: "256Gi"
          cpu: "32"
      volumeMounts:
        - name: model-cache
          mountPath: /root/.cache/huggingface
        - name: shm
          mountPath: /dev/shm
  volumes:
    - name: model-cache
      hostPath: { path: /mnt/nvme/hf-cache }
    - name: shm
      emptyDir: { medium: Memory, sizeLimit: 64Gi }

Key primitives the role asks about:

• NodeSelector / nodeAffinity / taints / tolerations for accelerator placement.
• PriorityClass & preemption for fair scheduling between research and production.
• ResourceQuota & LimitRange per namespace for quota enforcement.
• PodDisruptionBudget so cluster operations don't murder live inference.
• HorizontalPodAutoscaler with custom metrics (queue depth, GPU util) — not just CPU.
• Gang scheduling controllers — Volcano, Kueue, Yunikorn — for "schedule all N pods or none."

Distributed runtimes

"Distributed" in this role means two different things, and you should be able to talk about both.

Training-side

• PyTorch DDP — data parallel, one process per GPU, NCCL all-reduce on gradients.
• FSDP — fully sharded data parallel, shards optimizer state + grads + params.
• DeepSpeed ZeRO 1/2/3 — analogous sharding stages.
• Megatron-LM — tensor + pipeline parallelism for very large models.
• Ray — task-level distributed compute, often used for orchestration around training.

Inference-side

• vLLM — server with paged attention and continuous batching.
• Triton Inference Server — multi-model multi-backend serving.
• Ray Serve — flexible multi-stage pipelines.
• TensorRT-LLM — compiled, often combined with Triton.

The depth is in chapter 04.

What "production debugging" means for GPU infra

For this role, "production debugging" almost always means one of these:

Interviewers love asking debug scenarios because they reveal whether you've actually held the pager. The strongest answer always starts: "First I'd look at what changed. Then I'd narrow the blast radius..."

The cost-perf-reliability triangle

Every nontrivial design call in this role lives inside this triangle. Pick any two; the third bends.

• Maximize throughput, hold reliability → cost goes up (more reserved capacity, less spot).
• Minimize cost, hold throughput → reliability gets shakier (spot preemption, packing, less headroom).
• Hold cost & reliability → throughput is fixed; you can't grow free.

✓ Check yourself

Answer from memory before opening each one.

1. After a host driver update, new pods start on a node but report no GPUs. Walk through the mechanism that should have given them GPUs, and where it broke.

   Show answer

   The container runtime calls the NVIDIA OCI hook, which reads the visible-devices settings and bind-mounts the device nodes and driver libraries into the container. After a driver change the toolkit or device plugin often does not restart cleanly, so the node stops advertising or injecting GPUs. Check the device plugin DaemonSet, the driver and toolkit versions and dmesg, then drain and replace the node. See containers and the NVIDIA toolkit.

2. An image built against a newer CUDA toolkit fails on a new cluster with "CUDA driver version is insufficient". Why, and which direction of mismatch is safe?

   Show answer

   The host driver supplies the CUDA runtime. A driver at or above the toolkit version works, and a driver older than the toolkit fails at load. Upgrade the driver or rebuild against an older toolkit. See driver versus CUDA toolkit.

3. Single-node training runs at expected speed, but across nodes all-reduce reaches half of the expected throughput. What are the first two suspects and how do you confirm?

   Show answer

   Suspect that NCCL picked TCP instead of InfiniBand or RoCE, or that GPUDirect RDMA is missing and transfers stage through host memory. Turn on NCCL debug output and read which transport each ring uses. A PCIe lane downgrade is the third suspect. See networking and the fabric.

4. Fifty inference pods restart after a node drain and each one pulls a 140 GB model from object storage. What goes wrong and what do you change?

   Show answer

   Every restart pays a long first-byte and transfer cost, and fifty at once saturates the network, so recovery takes far longer than the outage. Stage or bake weights onto local NVMe and mmap them from there, with object storage as the source of truth only. See storage for weights.

5. Research training jobs and production inference share one GPU pool. How do you keep production safe, and what does that cost?

   Show answer

   Use priority classes with preemption, quotas per namespace and disruption budgets on serving. The price is that preempted training loses work back to its last checkpoint. The alternative of separate pools gives clean isolation but leaves capacity idle, so choose by how bursty the research load is. See Kubernetes for GPU workloads.

6. Two jobs fail on different nodes, one with Xid 79 and one with Xid 31. How does that change who you look at first?

   Show answer

   Xid 79 means the GPU fell off the bus, which is often hardware, so cordon the node and start a repair path. Xid 31 is a memory-management fault, which is often the application, so look at the job first. Both are leads rather than verdicts, so confirm against dmesg and the Xid reference. See Linux for GPU hosts.

Takeaway

• GPU nodes are Linux machines first: NUMA layout, kernel modules, persistence and fabric services, and dmesg are the first stop in an incident.
• Interconnect, not compute, is often the training bottleneck, and the wrong NCCL transport or missing GPUDirect RDMA halves throughput.
• Model weights are large enough that staging them to local NVMe on pod start is a design requirement, not an optimisation.
• Containers see GPUs through the NVIDIA toolkit hook, and a driver update without a clean toolkit and plugin restart yields pods with no GPUs.
• Kubernetes does not know about GPUs until the device plugin advertises them, and fair sharing needs priority classes, quotas, disruption budgets and gang scheduling.
• Production debugging starts with what changed, then narrows the blast radius.

The two deep dives start with inference serving, where most of the tuning decisions live. → Deep Dive — Inference Serving

Further reading

• Schedule GPUs — Kubernetes docs. How device plugins expose GPUs as schedulable resources.
• NVIDIA Container Toolkit — NVIDIA docs. How containers get device nodes and driver libraries, and the install and runtime configuration.
• NCCL environment variables — NVIDIA docs. The settings behind transport selection, interface choice and debug output.
• CUDA Compatibility — NVIDIA docs. The rules for which toolkit versions run on which drivers.
• XID Errors — NVIDIA docs. The reference for Xid codes and their likely causes.