Practice Interview Questions

vLLM treats KV-cache as virtual memory, divided into fixed-size blocks (e.g. 16 tokens). Each sequence has a block table mapping logical positions to physical blocks. Benefits: near-zero fragmentation (only allocate what's used, not max-context per sequence); copy-on-write sharing of identical prefixes between requests; admission control is exact (we know how many blocks remain). It dramatically improves throughput vs. contiguous KV-cache allocation, especially under varying sequence lengths.

Static batching collects requests until a batch fills or a timeout, then runs the whole batch through. Slowest request determines batch latency; GPU goes idle at batch boundaries. Continuous batching operates at the iteration level — every forward pass, the scheduler adds new sequences and removes finished ones, so the GPU is never idle waiting for slow requests, and new requests join immediately. Operationally: lower TTFT, higher throughput, but the static dynamic-batching knobs (max_queue_delay) become irrelevant; the new knobs are max-num-seqs and KV-cache utilization.

vLLM if the workload is one or a few LLMs with OpenAI-compatible API and latency-sensitive chat. Triton if you have a portfolio of models (tokenizers, embeddings, classifiers, several LLMs), mixed frameworks (TensorRT, ONNX, PyTorch, Python), or want one unified server with model-repository semantics and dynamic batching for non-LLM models. You can also run vLLM inside Triton via the vLLM backend to get both.

Tensor parallel TP=4 over NVLink puts ~35 GB of weights on each GPU, plus shared KV-cache pool and activation memory. Tradeoffs: TP adds an all-reduce per layer (cheap over NVLink, expensive across nodes); pull TP>1 only intra-node for inference. Alternatives: FP8 quantization (~70 GB total → fits on 2× H100); INT4 (~35 GB → fits on 1× H100 with room for cache, at some quality cost); H200/B200 with bigger HBM. The senior answer is to ask what TTFT/throughput target makes the precision/parallelism choice obvious.

Exclusive for any workload that saturates a GPU (big-model serving, training). MIG when many small inference workloads share a node and you need hardware-isolated SLOs (e.g. multi-tenant inference platform — each tenant gets a guaranteed slice). MPS when you have many small co-trusted clients sharing a GPU and want better utilization than time-slicing, accepting softer isolation. Default time-slicing only for dev/notebooks where utilization matters more than predictability.

Distributed training requires all N workers to start together — if 6 of 8 land and 2 are pending, the 6 idle GPUs are unusable. Vanilla K8s starts the 6 happily. Gang scheduling holds back the start until all N can be scheduled. Implement via Volcano (PodGroup), Kueue (Workload), or Yunikorn. The scheduler reserves capacity for the whole group atomically and only places pods when N can fit.

Hierarchical queues with priority classes. Production gets the highest priority class with preemption rights. Research gets a guaranteed quota plus borrowing rights up to a cohort cap when production is idle. Preemption is graceful: SIGTERM + checkpoint window, not SIGKILL, so research jobs lose at most N steps. Quota enforced via ResourceQuota; queue-level fairness via Kueue / Volcano. Surface a daily "what each team used" report for accountability.

Pending pod age per priority class (online >0 is a page). Time-to-bind p95 rising. Fragmentation — nodes with idle GPUs no pending pod can use due to topology/taints. Preemption rate spikes. Gap between allocatable and SM-active: high gap = bad placement, low utilization despite full allocation. Quota saturated for a team that's still queueing means you need to renegotiate.

Three different things are called utilization. nvidia-smi's "GPU-Util" is the fraction of time any kernel ran — a tiny kernel looping shows 100%. DCGM's SM-active is better. SM-occupancy (warps issued / max) is best for compute-bound work. For LLM decode, the right metric is HBM bandwidth utilization because decode is memory-bound, not compute-bound. A senior answer names which metric you mean.

Top half: fleet SM-active heatmap by node, fleet HBM used heatmap. Middle: top-five services p95 latency + TTFT, request rate, error rate. Queue depth in front of each service by priority class. Bottom: $/million-tokens trendline; quota usage by team; pending pods. The rule: if any panel turns red, that's the team's afternoon.

OpenTelemetry. The gateway opens the root span. Each downstream (auth, router, retriever, embedder, reranker, LLM, response post-process) opens child spans with attributes — model, tokens, cache hit, GPU node. Tail-based sampling keeps all traces with errors or latency > threshold so you have high signal at low storage cost. Export to Tempo/Jaeger/Honeycomb; correlate via the request ID in logs and metrics.

Averages hide tail. A service with p50 of 100ms and p99 of 5s can have a "300ms average" that suggests health when 1% of users see a broken experience. Histograms let you compute any percentile after the fact and roll up across instances. Use Prometheus histogram buckets sized to the actual distribution; compute histogram_quantile(0.99, ...) for tail metrics.

(1) What changed: recent deploy, driver, traffic shape? (2) Request mix: longer prompts, lower prefix-cache hit rate? (3) KV-cache utilization: cache thrashing under more concurrency? (4) dmesg for Xids; the GPU could be silently throttling on thermal or ECC. (5) Compare to neighbor replicas — if only this one is bad, restart it; if all are bad, look upstream. (6) p95 SM-active and DRAM-active to see whether it's compute or memory bound. The discipline: change → confirm → isolate.

Set NCCL_DEBUG=INFO, re-run, read which rank is waiting for which peer. Usual culprits: firewall blocking NCCL ports between nodes; NCCL_SOCKET_IFNAME picked a management NIC; one rank's pod is still pulling its image; IB link down (ibstat); subnet manager issue. In production, set NCCL_ASYNC_ERROR_HANDLING=1 and TORCH_NCCL_BLOCKING_WAIT=1 so hangs error out within a watchdog timeout instead of stalling forever.

For pure inference: yes — at fixed seed/temp the output is deterministic; otherwise differing samples are acceptable. For inference with tool calls: not without idempotency keys on the tools — the agent layer handles this. For billing: don't bill twice for retried requests that failed before the first token. Accept a client-provided request ID at the gateway, de-dupe within a short window. Streaming retries are usually "retry from scratch" in the contract.

Reject loudly at the edge with 429 + Retry-After. Per-tenant rate limits so one team can't starve others. Queue-time SLO — drop requests that have queued past their deadline (the user gave up). Load-shed low-priority traffic before high-priority. Circuit breaker in front of a melting backend. The principle: predictable 429s beat wandering p99 timeouts.

Frame the triangle first: latency-bounded, customer-facing, so optimize TTFT and tail; accept higher cost.

1. Edge: TLS termination, auth (mTLS or API key), per-tenant rate limiting, request validation, observability injection.
2. Router: model name → backend pool; weighted routing for canaries; circuit breaker; load shedding.
3. Backend pools: vLLM with prefix caching enabled, FP8 KV-cache, TP=4 on H100 SXM nodes. Sized for P50 from reserved + P25 on-demand + spot for batch elsewhere.
4. Autoscaler: KEDA on num_requests_waiting metric, scale-in with long cooldown to preserve KV-cache.
5. Observability: OTel traces, RED metrics per route, DCGM per node, cost dashboard per tenant.
6. Failure modes: 429 on saturation, retry-with-jitter advice, circuit breaker on backend health, blue/green for changes.
7. Capacity math: 10k RPS / 1.6 RPS-per-GPU-equivalent at our token profile = ~6300 GPUs equivalent at saturation; size to 30% headroom = ~9000. Mix of reserved/on-demand based on $/GPU-hour spread.

Pool the hardware by accelerator type; tenants are namespaces with quota + borrowing. Isolation: full GPU for big-model serving, MIG for small models that need SLOs, MPS only where tenants trust each other. Router routes by model name to the right pool. Observability: per-tenant dashboards (latency, throughput, GPU-hours, cost). Self-service "deploy a model" via KServe CRDs. Governance: weekly cost review; quota review quarterly.

Pin via GPU Operator. Build a new driver container image. Deploy first to a dev node, run smoke + perf regression. Then to a single canary in pre-prod, soak 24h, compare DCGM Xids/temp/throughput baselines. Then to 5% of prod (one AZ), soak 24h with on-call watching. Then 25%, 50%, 100% staged across AZs with soak between gates. Pre-defined rollback at each gate. Communicate via incident-style status to stakeholders.

Spot for ~70% of the fleet, on-demand reserve for the rest. Async checkpointing every N minutes to durable storage (object + Lustre cache). On spot warning (2-min for AWS), drain the worker: stop accepting steps, flush state, exit. Re-launch picks up latest checkpoint. Use elastic training (TorchElastic) so jobs survive worker churn without dying. Hedge across instance families and AZs. Track "% time productive" as the headline metric; alert when productivity drops below threshold.

Five-factor: (1) privacy — does the workload touch regulated data? (2) latency — is the API round trip + tail compatible with the SLO? (3) reliability — is the external API a tolerable dependency? (4) cost — at expected volume, blended in-house $/M-tokens vs API $/M-tokens, plus operating overhead. (5) differentiation — is the model fine-tuned or custom? At the company, (1) usually pushes toward in-house even when (4) is close. The cost math always assumes high utilization; idle reserved GPUs flip the answer.

Idle reserved capacity (paying for GPUs no one used). Over-provisioned replicas with rounded-up GPU counts. Spot capacity unused because batch workloads weren't migrated. Cold prefix-cache loss from frequent scale-in. Inefficient quantization (FP16 where FP8 is fine). Lone-tenant nodes with poor packing. The dashboard is "idle GPU-hours per team per day" with a top-N table.

Depends. For LLM decode, 30% SM-active is normal — decode is memory-bandwidth-bound, not compute-bound. Look at DRAM-active; if that's 70-80%, you're saturated and SM-active doesn't matter. For a vision model or training step, 30% SM-active is bad — likely data-loader starvation, small batch size, or sync overhead. The same number means different things depending on the workload shape; the senior framing is "is the bottleneck resource saturated?"

State: capacity, refill_per_sec, current tokens, last_ts. On allow(now, cost): refill = min(capacity, tokens + (now - last_ts) × refill_per_sec); if refill ≥ cost, debit and return True. Constant time, constant memory. Clamp time backwards. Generalize cost for variable-size LLM requests.

Exact: deque of (ts, latency); evict head while head_ts < now − window; sort and pick the 99th element on query. O(n log n) per query. Production: HDR histogram or t-digest for bounded memory and O(log n) updates with small error. Bucketed histogram per second + merge across window also works.

Async loop. submit(item) appends to a buffer and sets a wakeup event. The loop awaits either the event (buffer non-empty) or a timeout (W ms). When B items are present, flush immediately; otherwise on timeout flush whatever is there. Return per-item futures so callers get their result back. Handle shutdown by flushing partials.

First-fit decreasing: sort jobs by size descending; for each, place on first bin with space; open a new bin if none fits. Within 11/9 of optimal for bin packing. Extension to discuss: multi-dimensional (compute + memory) is NP-hard in general; LP-relaxation or Kueue topology-aware schedulers handle real production. Mention this; don't try to derive it.

STAR. Situation: scope, scale, severity. Task: your role, what was at stake. Action: how you triaged, what you escalated, the call you made under uncertainty. Result: time to mitigate, blast radius, what changed durably (PIR action that shipped). Bonus: a sentence on what you'd do differently. Avoid heroism; emphasize coordination and the durable fix.

Useful to have a story where you said "no" or "not yet" to a researcher and proposed a better path. Show that you respect the user, understand their goal, and protect the substrate. Resolution: a small experiment that resolved the disagreement with data, not opinion.

Read claims, but verify on your workload — your prompt distribution, your model, your GPUs. Reproduce one benchmark to ground-truth. Stand up a soak in pre-prod. Measure quality regression alongside performance. Cost the operational debt (new tool means new on-call runbooks, new failure modes). Decide on a small bet (a single use case) before fleet adoption. Don't be first; don't be last.

Specific. the company is building in-house AI compute precisely because privacy, latency, and cost matter for a regulated exchange — that's the framing where this team's leverage is highest. Be concrete about what attracts you (the in-house mandate, the senior small team, the chance to own the substrate). Avoid generic "I love AI" — talk about this team's mandate. End with a question.