Domain Context
Vocabulary so you never sound lost. GPU internals, interconnects, precisions, the major accelerator lines, checkpoint formats, KV-cache plumbing, and cloud pricing tiers.
GPU anatomy — the parts you should be able to name
| Part | What it is |
|---|---|
| SM (Streaming Multiprocessor) | The execution unit. H100 has 132 SMs. Kernels are launched as grids of thread blocks; each block runs on one SM. |
| CUDA core | An FP32 ALU. Marketing number — "how many CUDA cores" is misleading; SM count and clock matter more. |
| Tensor core | Specialized matmul unit. Operates on small tiles (e.g. 16×16) in mixed precision. The reason GPUs are dominant for transformers. |
| Warp | Group of 32 threads executing in lockstep. SIMT execution model. |
| Register file | Per-SM fast storage. Allocation per thread limits warps in flight. |
| Shared memory / L1 | Per-SM scratchpad, programmer-managed. ~228 KB on H100. |
| L2 cache | Per-GPU, shared across SMs. 50-60 MB on H100. |
| HBM (High-Bandwidth Memory) | The big main memory. H100 SXM has 80 GB at ~3 TB/s; H200 has 141 GB at 4.8 TB/s. |
| NVLink | NVIDIA's GPU-GPU interconnect. H100 SXM has 18 NVLink-4 connections totaling 900 GB/s aggregate. |
| NVSwitch | Switch fabric inside multi-GPU nodes for all-to-all NVLink connectivity. |
| Transformer Engine | NVIDIA's mixed-precision (FP8/FP16/BF16) feature on Hopper+, integrated with cuDNN and frameworks. |
Numeric precisions you'll discuss
| Format | Bits | Typical use |
|---|---|---|
| FP64 | 64 | Scientific HPC; rarely used in deep learning |
| FP32 | 32 | Single precision; default in earlier ML |
| TF32 | 32 (19 effective) | NVIDIA Ampere+ "tensor-friendly FP32" — drop-in faster training |
| FP16 | 16 | Mixed-precision training (historic), inference baseline |
| BF16 | 16 | Wider exponent than FP16; preferred for training stability |
| FP8 (E4M3, E5M2) | 8 | Hopper+/Blackwell; serious throughput win, modest quality cost |
| INT8 | 8 | Quantized inference (SmoothQuant, GPTQ, AWQ) |
| INT4 | 4 | Aggressive inference quantization; meaningful quality cost |
| NVFP4 / FP4 | 4 | Blackwell-era; bleeding edge |
The interview answer
"Default to BF16 for training stability, FP8 for inference where the hardware supports it (Hopper+). Below FP8, evaluate the quality cost case-by-case; expect 1-4% degradation depending on the quantization technique and the model."
Interconnect & fabric
- PCIe Gen4 x16 — ~32 GB/s. Common for A100 PCIe and lower-tier deployments.
- PCIe Gen5 x16 — ~64 GB/s. H100 PCIe form factor and modern hosts.
- NVLink-4 — used in H100 SXM, 900 GB/s aggregate per GPU.
- NVLink-5 — Blackwell era, 1.8 TB/s aggregate per GPU.
- NVSwitch — full-bandwidth switching between GPUs in a chassis.
- InfiniBand HDR — 200 Gbps. Older fabric standard.
- InfiniBand NDR — 400 Gbps per port. Current default for big training.
- RoCE (RDMA over Converged Ethernet) — RDMA semantics on Ethernet; 200/400/800 Gbps available.
- GPUDirect RDMA — NIC writes directly to GPU memory, bypassing the host. Required for IB at full speed.
- NIXL / NCCL — software layers on top.
Form factors
- SXM — module form factor, higher power envelope (700W+ for H100 SXM5), full NVLink. Found in HGX/DGX systems.
- PCIe — card form factor, lower power (350W typical for H100 PCIe), NVLink bridges between pairs only. More flexible for standard server chassis.
- SXM beats PCIe for big-model training and tensor-parallel serving because of full NVLink. For inference of smaller models or workloads with little inter-GPU traffic, PCIe is fine.
The NVIDIA accelerator line — what's on the menu
| Generation | Architecture | Flagship | HBM | Highlights |
|---|---|---|---|---|
| Ampere | GA100 | A100 (40/80 GB) | HBM2e | TF32, MIG, NVLink-3. Still huge in production. |
| Hopper | GH100 | H100 (80 GB) | HBM3 | FP8, Transformer Engine, NVLink-4, DPX instructions. |
| Hopper-refresh | GH200 | H200 (141 GB) | HBM3e | Bigger HBM & bandwidth — wins on long-context inference. |
| Blackwell | GB100/GB200 | B100, B200, GB200 NVL72 | HBM3e | FP4, 2nd-gen Transformer Engine, NVLink-5. Multi-chip module. |
| Ada Lovelace | AD102 | L40, L40S | GDDR6 | Mid-tier inference; no HBM, but cheaper. |
Non-NVIDIA accelerators worth recognizing
- Google TPU v5p / v5e / v6 (Trillium) — pod-based systolic-array architecture. JAX/PyTorch XLA. Strong for big training.
- AWS Trainium / Trainium2 — AWS's custom training chip. Neuron SDK. Price-competitive for AWS-native workloads.
- AWS Inferentia2 — inference-only sibling.
- Intel Gaudi 2 / Gaudi 3 — Habana labs. HPU. Competitive on certain LLM training workloads, with integrated networking.
- AMD MI300X / MI325X — large HBM (192-256 GB), ROCm software stack. Maturing; competitive for inference of big-context workloads.
- Apple Silicon / Cerebras / Groq / SambaNova — niche or specialized.
You won't be deployed on all of them, but say their names confidently when asked.
KV-cache, prefix caching, page tables
The plumbing language specific to LLM serving:
- KV-cache — per-layer K and V projections retained on GPU memory for an active sequence. The reason decode is fast.
- Paged attention — vLLM's idea of treating KV-cache as virtual memory: fixed-size blocks in HBM, with a block table (page table) per sequence mapping logical positions to physical blocks.
- Block size — typically 16 tokens; tunable. Smaller = less fragmentation, more table overhead.
- Prefix caching — when two requests share a prefix (e.g. the same system prompt), point both sequences' early block-table entries at the same physical blocks. Copy-on-write when they diverge.
- Chunked prefill — split a long prompt into chunks and process them across multiple iterations alongside ongoing decodes, so big prompts don't starve other requests.
- Swap to CPU / disk — vLLM can spill blocks to host memory when GPU memory is tight; trades latency for capacity.
- KV-cache quantization — store cache in FP8 / INT8 to double effective capacity at small quality cost.
Checkpoint formats
- safetensors — Hugging Face's safer alternative to
.pt. No pickle deserialization; bounded memory mapping. The de facto serving format. - pickle / .pt / .pth — original PyTorch format; opens arbitrary code execution surface. Avoid for untrusted inputs.
- GGUF — llama.cpp's quantized format. Common for CPU/edge inference, less so for cluster serving.
- TensorRT plan files — compiled engines, GPU-architecture-specific. Not portable.
- Sharded checkpoints — PyTorch DCP (Distributed Checkpoint), DeepSpeed shards, FSDP shards. One file per rank; metadata describes how to reassemble.
- Megatron checkpoint format — used by Megatron-LM with tensor + pipeline parallelism.
Cloud accelerator pricing tiers
You should be able to discuss these without specific numbers — the numbers change.
- On-demand — pay-per-second, highest unit cost, no commitment.
- Spot / preemptible — significant discount (50-80%), interruptible.
- Savings plans (AWS) / committed-use discounts (GCP) — commit to N hours per month for 1-3 years for a discount. Applies across instance types in the same family.
- Reserved instances — capacity reservation tied to a specific region/AZ/family. Stronger capacity guarantee than savings plans.
- Capacity blocks (AWS) — pre-purchase GPU-hours in a future contiguous window. Designed for training runs.
- Dedicated hosts — entire physical server; useful for licensing or strict isolation.
- On-prem amortization — bought hardware spread across its useful life (typically 3-4 years for GPUs). Add power, cooling, datacenter, ops loaded cost. Compare to cloud blended cost.
Blended cost model
Real cost = (reserved capacity fixed cost / hours used) + (on-demand burst hours × $/hour) + (spot hours × spot rate) + observability overhead + control-plane overhead. The "$/GPU-hour" question is meaningless without that decomposition.
Quick glossary — the rest
- CUDA — NVIDIA's parallel compute platform and language.
- cuDNN — accelerated primitives for deep learning.
- NCCL — NVIDIA Collective Communications Library; all-reduce, broadcast, all-gather across GPUs.
- UCX / NVSHMEM — lower-level communication libraries.
- cuBLAS / cuBLASLt — matrix-multiply libraries.
- NVRTC — runtime CUDA compilation.
- DCGM — Data Center GPU Manager; telemetry, diagnostics, health.
- MIG — Multi-Instance GPU partitioning.
- MPS — Multi-Process Service for GPU sharing.
- UVM — Unified Virtual Memory between host and device.
- PTX / SASS — NVIDIA assembly languages (PTX virtual, SASS native).
- RDMA — Remote Direct Memory Access; bypass kernel for network transfers.
- RoCE — RDMA over Converged Ethernet.
- QSFP — connector form factor for high-speed networking.
- TFLOPS — Tera-FLOPS; vendor headline number, usually peak FP at a precision.
- Token — unit of LLM input/output; a piece of a word in the model's tokenizer.
- TTFT / TPOT / ITL — time-to-first-token, time-per-output-token, inter-token-latency.
- SGLang / TGI / LMDeploy — alternative LLM serving stacks worth knowing exist.