Core Fundamentals

Linux for GPU hosts

GPU nodes are Linux machines first. The unusual parts are NUMA topology, hugepages, kernel modules, and the device files. The fundamentals you must be fluent in:

• cgroups v2 & namespaces — how container isolation actually works. GPU access flows through bind-mounted device nodes (/dev/nvidia*) and a hook.
• NUMA topology — H100 nodes have multiple NUMA domains. PCIe lanes attach to specific sockets. Cross-NUMA traffic is slow. nvidia-smi topo -m shows the PCIe / NVLink topology; numactl --hardware shows the CPU side.
• Hugepages — RDMA and some CUDA paths benefit from hugepage backing. Misconfigured hugepages cause silent perf cliffs.
• Kernel modules — nvidia, nvidia_uvm, nvidia_drm, nvidia_peermem. A bad mismatch between the loaded module and userspace driver is a top-5 outage source.
• Systemd units — nvidia-persistenced keeps the driver warm; nvidia-fabricmanager sets up NVLink/NVSwitch.
• journalctl + dmesg — the first place driver and Xid errors land. You should reflexively check both during a GPU incident.

# The interview-grade Linux/GPU triage cheatsheet
nvidia-smi                          # are GPUs visible, healthy, temp/power sane?
nvidia-smi topo -m                  # NVLink / PCIe topology
nvidia-smi -q -d ECC,POWER,CLOCK    # detail per device
dmesg | grep -i -E 'nvidia|xid|nvrm' # driver-level errors
journalctl -u nvidia-fabricmanager  # NVSwitch fabric state
cat /proc/driver/nvidia/version     # loaded driver version vs userspace
lspci | grep -i nvidia              # PCIe enumeration sanity
numactl --hardware                  # NUMA layout
ibstat                              # InfiniBand link state (if applicable)

Networking and the fabric

GPU training is bottlenecked by interconnect bandwidth, not compute, more often than people expect. Three layers:

1. Intra-node — NVLink between GPUs on the same chassis (900 GB/s aggregate on H100 SXM). NVSwitch enables all-to-all at full bandwidth.
2. PCIe — host-to-GPU, NIC-to-GPU. Gen5 x16 ≈ 64 GB/s. GPUDirect RDMA lets a NIC DMA directly into GPU memory, bypassing the host.
3. Inter-node fabric — InfiniBand HDR/NDR (200/400 Gbps) or RoCE (RDMA over Converged Ethernet). NCCL uses this for cross-node all-reduce.

For inference, the fabric matters less unless you're doing tensor parallelism across nodes. For training, it's load-bearing.

Storage for model weights & checkpoints

A 70B-parameter model in FP16 is ~140 GB. A 405B model is ~810 GB. Sharded checkpoints during training generate terabytes per checkpoint. Three storage tiers:

• Object storage (S3, GCS, R2) — durable, cheap, slow first-byte. Source of truth for weights.
• Network filesystem (Lustre, Weka, FSx) — high parallel throughput, good for shared training scratch.
• Local NVMe — fastest, ephemeral. Used for staging weights into a serving pod, for KV-cache spill, for fast checkpoint write-then-flush.

A practical pattern for inference: bake or stage weights to local NVMe on pod start, then mmap them. Loading a 70B model over the network for every pod restart is a pager-class mistake.

Containers and the NVIDIA toolkit

Containers see GPUs because nvidia-container-toolkit injects device nodes and libraries via an OCI hook. The flow:

1. Container runtime (containerd, CRI-O, Docker) invokes the OCI hook.
2. Hook reads NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES.
3. Bind-mounts /dev/nvidia*, libcuda.so, and friends into the container namespace.
4. Container starts; nvidia-smi works inside.

The classic failure: the host driver is updated but the toolkit / device plugin doesn't restart cleanly, and new pods come up without GPUs. The drain-and-replace runbook lives in chapter 13.

Kubernetes basics for GPU workloads

K8s itself doesn't know about GPUs. The NVIDIA device plugin (a DaemonSet) advertises nvidia.com/gpu as an allocatable resource. A pod requesting nvidia.com/gpu: 2 gets two whole GPUs bound exclusively.

apiVersion: v1
kind: Pod
metadata:
  name: vllm-server
spec:
  nodeSelector:
    accelerator: nvidia-h100
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  priorityClassName: inference-online
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "meta-llama/Llama-3.1-70B-Instruct",
             "--tensor-parallel-size", "4"]
      resources:
        limits:
          nvidia.com/gpu: 4
          memory: "256Gi"
          cpu: "32"
      volumeMounts:
        - name: model-cache
          mountPath: /root/.cache/huggingface
        - name: shm
          mountPath: /dev/shm
  volumes:
    - name: model-cache
      hostPath: { path: /mnt/nvme/hf-cache }
    - name: shm
      emptyDir: { medium: Memory, sizeLimit: 64Gi }

Key primitives the role asks about:

• NodeSelector / nodeAffinity / taints / tolerations for accelerator placement.
• PriorityClass & preemption for fair scheduling between research and production.
• ResourceQuota & LimitRange per namespace for quota enforcement.
• PodDisruptionBudget so cluster operations don't murder live inference.
• HorizontalPodAutoscaler with custom metrics (queue depth, GPU util) — not just CPU.
• Gang scheduling controllers — Volcano, Kueue, Yunikorn — for "schedule all N pods or none."

Distributed runtimes

"Distributed" in this role means two different things, and you should be able to talk about both.

Training-side

• PyTorch DDP — data parallel, one process per GPU, NCCL all-reduce on gradients.
• FSDP — fully sharded data parallel, shards optimizer state + grads + params.
• DeepSpeed ZeRO 1/2/3 — analogous sharding stages.
• Megatron-LM — tensor + pipeline parallelism for very large models.
• Ray — task-level distributed compute, often used for orchestration around training.

Inference-side

• vLLM — server with paged attention and continuous batching.
• Triton Inference Server — multi-model multi-backend serving.
• Ray Serve — flexible multi-stage pipelines.
• TensorRT-LLM — compiled, often combined with Triton.

The depth is in chapter 04.

What "production debugging" means for GPU infra

For this role, "production debugging" almost always means one of these:

Interviewers love asking debug scenarios because they reveal whether you've actually held the pager. The strongest answer always starts: "First I'd look at what changed. Then I'd narrow the blast radius..."

The cost-perf-reliability triangle

Every nontrivial design call in this role lives inside this triangle. Pick any two; the third bends.

• Maximize throughput, hold reliability → cost goes up (more reserved capacity, less spot).
• Minimize cost, hold throughput → reliability gets shakier (spot preemption, packing, less headroom).
• Hold cost & reliability → throughput is fixed; you can't grow free.