Domain Context

GPU anatomy — the parts you should be able to name

Numeric precisions you'll discuss

Interconnect & fabric

• PCIe Gen4 x16 — ~32 GB/s. Common for A100 PCIe and lower-tier deployments.
• PCIe Gen5 x16 — ~64 GB/s. H100 PCIe form factor and modern hosts.
• NVLink-4 — used in H100 SXM, 900 GB/s aggregate per GPU.
• NVLink-5 — Blackwell era, 1.8 TB/s aggregate per GPU.
• NVSwitch — full-bandwidth switching between GPUs in a chassis.
• InfiniBand HDR — 200 Gbps. Older fabric standard.
• InfiniBand NDR — 400 Gbps per port. Current default for big training.
• RoCE (RDMA over Converged Ethernet) — RDMA semantics on Ethernet; 200/400/800 Gbps available.
• GPUDirect RDMA — NIC writes directly to GPU memory, bypassing the host. Required for IB at full speed.
• NIXL / NCCL — software layers on top.

Form factors

• SXM — module form factor, higher power envelope (700W+ for H100 SXM5), full NVLink. Found in HGX/DGX systems.
• PCIe — card form factor, lower power (350W typical for H100 PCIe), NVLink bridges between pairs only. More flexible for standard server chassis.
• SXM beats PCIe for big-model training and tensor-parallel serving because of full NVLink. For inference of smaller models or workloads with little inter-GPU traffic, PCIe is fine.

The NVIDIA accelerator line — what's on the menu

Non-NVIDIA accelerators worth recognizing

• Google TPU v5p / v5e / v6 (Trillium) — pod-based systolic-array architecture. JAX/PyTorch XLA. Strong for big training.
• AWS Trainium / Trainium2 — AWS's custom training chip. Neuron SDK. Price-competitive for AWS-native workloads.
• AWS Inferentia2 — inference-only sibling.
• Intel Gaudi 2 / Gaudi 3 — Habana labs. HPU. Competitive on certain LLM training workloads, with integrated networking.
• AMD MI300X / MI325X — large HBM (192-256 GB), ROCm software stack. Maturing; competitive for inference of big-context workloads.
• Apple Silicon / Cerebras / Groq / SambaNova — niche or specialized.

You won't be deployed on all of them, but say their names confidently when asked.

KV-cache, prefix caching, page tables

The plumbing language specific to LLM serving:

• KV-cache — per-layer K and V projections retained on GPU memory for an active sequence. The reason decode is fast.
• Paged attention — vLLM's idea of treating KV-cache as virtual memory: fixed-size blocks in HBM, with a block table (page table) per sequence mapping logical positions to physical blocks.
• Block size — typically 16 tokens; tunable. Smaller = less fragmentation, more table overhead.
• Prefix caching — when two requests share a prefix (e.g. the same system prompt), point both sequences' early block-table entries at the same physical blocks. Copy-on-write when they diverge.
• Chunked prefill — split a long prompt into chunks and process them across multiple iterations alongside ongoing decodes, so big prompts don't starve other requests.
• Swap to CPU / disk — vLLM can spill blocks to host memory when GPU memory is tight; trades latency for capacity.
• KV-cache quantization — store cache in FP8 / INT8 to double effective capacity at small quality cost.

Checkpoint formats

• safetensors — Hugging Face's safer alternative to .pt. No pickle deserialization; bounded memory mapping. The de facto serving format.
• pickle / .pt / .pth — original PyTorch format; opens arbitrary code execution surface. Avoid for untrusted inputs.
• GGUF — llama.cpp's quantized format. Common for CPU/edge inference, less so for cluster serving.
• TensorRT plan files — compiled engines, GPU-architecture-specific. Not portable.
• Sharded checkpoints — PyTorch DCP (Distributed Checkpoint), DeepSpeed shards, FSDP shards. One file per rank; metadata describes how to reassemble.
• Megatron checkpoint format — used by Megatron-LM with tensor + pipeline parallelism.

Cloud accelerator pricing tiers

You should be able to discuss these without specific numbers — the numbers change.

• On-demand — pay-per-second, highest unit cost, no commitment.
• Spot / preemptible — significant discount (50-80%), interruptible.
• Savings plans (AWS) / committed-use discounts (GCP) — commit to N hours per month for 1-3 years for a discount. Applies across instance types in the same family.
• Reserved instances — capacity reservation tied to a specific region/AZ/family. Stronger capacity guarantee than savings plans.
• Capacity blocks (AWS) — pre-purchase GPU-hours in a future contiguous window. Designed for training runs.
• Dedicated hosts — entire physical server; useful for licensing or strict isolation.
• On-prem amortization — bought hardware spread across its useful life (typically 3-4 years for GPUs). Add power, cooling, datacenter, ops loaded cost. Compare to cloud blended cost.

Quick glossary — the rest

• CUDA — NVIDIA's parallel compute platform and language.
• cuDNN — accelerated primitives for deep learning.
• NCCL — NVIDIA Collective Communications Library; all-reduce, broadcast, all-gather across GPUs.
• UCX / NVSHMEM — lower-level communication libraries.
• cuBLAS / cuBLASLt — matrix-multiply libraries.
• NVRTC — runtime CUDA compilation.
• DCGM — Data Center GPU Manager; telemetry, diagnostics, health.
• MIG — Multi-Instance GPU partitioning.
• MPS — Multi-Process Service for GPU sharing.
• UVM — Unified Virtual Memory between host and device.
• PTX / SASS — NVIDIA assembly languages (PTX virtual, SASS native).
• RDMA — Remote Direct Memory Access; bypass kernel for network transfers.
• RoCE — RDMA over Converged Ethernet.
• QSFP — connector form factor for high-speed networking.
• TFLOPS — Tera-FLOPS; vendor headline number, usually peak FP at a precision.
• Token — unit of LLM input/output; a piece of a word in the model's tokenizer.
• TTFT / TPOT / ITL — time-to-first-token, time-per-output-token, inter-token-latency.
• SGLang / TGI / LMDeploy — alternative LLM serving stacks worth knowing exist.

✓ Check yourself

Answer from memory before opening each one.

1. Two vendors quote CUDA core counts for competing GPUs. Why is that a poor basis for choosing hardware for a transformer workload, and what do you compare?

   Show answer

   A CUDA core is only an FP32 unit, and transformer work runs on tensor cores in the SMs. Compare SM count and clock, tensor throughput at the precision you will use, memory capacity and bandwidth, and interconnect. See GPU anatomy.

2. A team proposes serving a model in INT4 to halve cost, and training in FP16. What do you ask, and what is your default for each?

   Show answer

   For serving, ask what quality loss the task tolerates and how you measured it, since FP8 is the usual default on Hopper and newer and INT4 carries a real quality cost. For training, prefer BF16, which has a wider exponent than FP16 and trains more stably. See numeric precisions.

3. Why can an H200 beat an H100 on long-context inference when the compute is the same?

   Show answer

   Long-context decode is limited by memory: the KV-cache grows with context, and each token reads weights and cache. More HBM holds more sequences, and higher bandwidth raises tokens per second. Compute is not the constraint. See the accelerator line.

4. Two requests share a long system prompt, and the second diverges after it. What is physically shared, and what happens at the point of divergence?

   Show answer

   Both block tables point at the same physical KV blocks for the shared prefix. When the second request writes beyond it, it gets its own blocks, with copy-on-write for a partially shared block. Prefix caching saves prefill and memory only for the shared part. See KV-cache, prefix caching, page tables.

5. A model arrives as a pickle checkpoint from an outside team, and another as a TensorRT plan built on a different GPU. What is wrong with each?

   Show answer

   A pickle file can execute arbitrary code when loaded, so convert it in a sandbox or require safetensors, which holds only tensors. A TensorRT plan is compiled for a specific GPU architecture and will not run on another, so rebuild it on the target pool. See checkpoint formats.

6. A colleague says the fleet costs the on-demand hourly rate times the GPU count. What is missing from that figure?

   Show answer

   It ignores the blend: reserved capacity as a fixed cost over hours used, burst hours at on-demand rates, spot hours at the spot rate, and control-plane and observability overhead. On-prem adds amortisation, power, cooling and operations. A dollar-per-GPU-hour figure means little without that breakdown. See cloud accelerator pricing.

Takeaway

• CUDA core counts mislead. Compare SMs, tensor throughput, memory size and bandwidth, and interconnect.
• Default to BF16 for training and FP8 for inference on Hopper and newer, and judge anything lower on measured quality.
• Interconnect numbers set what scales: NVLink within a node, InfiniBand or RoCE between nodes, GPUDirect RDMA to skip the host.
• SXM carries full NVLink and the higher power, so it suits big-model training and tensor-parallel serving. PCIe suits smaller or less communication-heavy work.
• HBM size and bandwidth, not compute, often decide long-context inference, which is why H200-class parts matter.
• Use safetensors for sharing weights, treat pickle as unsafe, and rebuild TensorRT plans per GPU architecture.
• Real GPU cost is a blend of reserved, on-demand, spot and overhead, never a single hourly rate.

Next are twenty-eight practice questions that use this vocabulary under drill conditions. → Practice Interview Questions

Further reading

• FP8 Formats for Deep Learning — Micikevicius et al., 2022. The E4M3 and E5M2 formats and the evidence for training and inference in 8 bits.
• Transformer Engine user guide — NVIDIA docs. How FP8 is applied on Hopper and newer, with scaling.
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Dao et al., 2022. Why attention is memory-bound and how tiling in SM shared memory fixes it.
• Safetensors — Hugging Face docs. The tensor format that avoids code execution on load.
• vLLM documentation — vLLM project. Block size, prefix caching and chunked prefill settings.
• NVIDIA H100 Tensor Core GPU — NVIDIA. Vendor specifications for the H100 line, including memory and NVLink.