Deployment & Ops

Cluster deploy patterns

A production GPU cluster isn't one Kubernetes cluster. Most production fleets run several:

• Production inference — strict SLAs, reserved capacity, change-controlled.
• Research / experimentation — looser, faster iteration, can absorb churn.
• Pre-prod / staging — production-shaped, used as canary.
• Dev / batch — cheaper hardware, spot, mixed.

Separation isolates blast radius and lets each cluster carry its own change cadence. Cluster lifecycle managed via IaC: Terraform for the cluster shell, Helm/Kustomize for the workloads, GitOps (Argo CD / Flux) as the reconciler.

Node pools by accelerator

Within a cluster, separate node pools per accelerator type. Workloads use nodeSelector/affinity + taints to land on the right pool.

# Pinning a workload to the right pool
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-70b
spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-h100-sxm
        node-pool: h100-sxm-online
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: vllm-llama3-70b }

Image management with model weights

Three deployment patterns, each with a tradeoff. Pick deliberately.

Default for production: sidecar with hostPath NVMe cache. Weights tagged by content hash. The first pod on a node pulls; subsequent pods hardlink.

Blue/green & canary inference deploys

Two patterns for shipping a model or serving-stack change without taking the service down.

Blue/green

Run both old (blue) and new (green) at the same time. Traffic flipped at the gateway. Fast rollback (flip back). Expensive (2× capacity briefly). Good for big-bang changes.

Canary

Weighted traffic split — 1% to new, then 10%, 50%, 100%. Watch metrics at each gate. Cheaper than blue/green; longer rollouts; finer-grained.

# Argo Rollouts canary for a vLLM service
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama3-70b }
spec:
  replicas: 8
  strategy:
    canary:
      maxSurge: "25%"
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: latency-and-error-rate
        - setWeight: 25
        - pause: { duration: 30m }
        - setWeight: 50
        - pause: { duration: 1h }
        - setWeight: 100

The analysis template hits Prometheus and fails the rollout if p95 latency or 5xx rate exceeds thresholds.

Rollback strategy

The minimum bar:

• Every deploy is rollbackable with a single command or a single revert PR.
• Previous artifact retained — image, weights, plan files — for at least the duration of the canary plus soak.
• Practiced — at least once per quarter, do an intentional rollback in staging.
• Automated triggers — analysis template, error-budget burn-rate alert, manual button.
• Communicate — the rollback is in the incident channel within 60s of being decided, with the reason.

The observability stack

The boring but essential parts. You should be able to name the pieces and explain what each does.

• Metrics — Prometheus + DCGM exporter + service /metrics endpoints. Long-term storage via Mimir/Thanos/VM.
• Logs — Fluent-bit / Vector → Loki / Elastic / Datadog. Structured JSON.
• Traces — OpenTelemetry collector → Tempo / Jaeger / Honeycomb. Tail-based sampling.
• Dashboards — Grafana. One per service, one per team, one for fleet health.
• Alerts — Alertmanager → PagerDuty / Opsgenie. Tied to SLOs, not raw metrics.
• Synthetics — a probe service hitting each model with canned prompts every minute, watching TTFT and correctness.
• Profiling — Nsight Systems for timeline traces; Nsight Compute for kernel-level. Continuous profiling (Parca / Pyroscope) for long-running services.

Profilers — Nsight, PyTorch, dcgmi, py-spy

You will be asked at some point: "How would you profile this?" Have the toolkit by name.

Incident response

The team will get paged. The discipline keeps the pages short and the learning durable.

1. Acknowledge fast. The page system tracks ack time. Two minutes is the target.
2. Start an incident channel. Slack/Teams channel named with the date. Pin the alert.
3. Assign an Incident Commander. One person owns coordination — even on a small team, one person is "in charge of the incident."
4. Open the runbook. If one exists, work it.
5. Communicate hourly. Status update in the channel even if "still investigating."
6. Stop the bleeding before finding the root cause. Roll back the deploy, drain the bad node, increase capacity — the goal is to restore service first.
7. Once service is restored: mark incident resolved, schedule the PIR (post-incident review) within a week.
8. PIR produces actions. Each action has an owner and a date. Action items show up in the next sprint.

Anatomy of a good runbook

A runbook should be runnable in the middle of the night by someone who is not the expert. Components:

1. Title — what alert triggers this.
2. Severity — P0/P1/P2.
3. Symptoms — what you'll see.
4. Likely causes in order of probability.
5. Diagnostic commands — copy-pasteable, with expected output.
6. Remediation steps — copy-pasteable, idempotent.
7. Escalation — who to wake up and when.
8. Related runbooks — links.
9. Last updated, last verified.

# Excerpt of a "node Xid storm" runbook
#
# Symptom: Alert "DCGMXidErrorsHigh" on node-X
# Likely:
#   1) Failing HBM (rate of Xid 63/64) — drain & replace
#   2) Driver bug (rate of Xid 119/120) — drain, file ticket
#   3) Power glitch (Xid 79 isolated event) — power-cycle node

# Step 1: confirm
ssh node-X dmesg | grep -i 'NVRM: Xid' | tail -20
ssh node-X dcgmi diag -r 1     # quick health check

# Step 2: drain (PDB-respecting)
kubectl cordon node-X
kubectl drain node-X --ignore-daemonsets --delete-emptydir-data --timeout=10m

# Step 3: dispatch
# If diag fails: tag node out of service; open hardware ticket
# If diag passes: gather logs, reboot, run smoke workload

On-call sustainability

A senior engineer protects on-call quality. In an interview, you can demonstrate this by talking about:

• Page-budget hygiene — measure pages/week/engineer; investigate when it rises.
• Symptom-based alerts — only page on user-impacting symptoms.
• Runbook-or-don't-page rule — every page either has a runbook or shouldn't exist.
• Follow-the-sun if the team spans time zones; otherwise rotation cadence that respects sleep.
• Action-item velocity from PIRs — if PIR actions aren't shipping, the same incident will recur.

✓ Check yourself

Answer from memory before opening each one.

1. Why run several clusters for production inference, research and staging instead of one large shared one?

   Show answer

   Separate clusters isolate blast radius and let each carry its own change cadence: production is change-controlled and reserved, while research absorbs churn. A bad driver or runaway job in one cannot take down the others, and staging gives a production-shaped canary. See cluster deploy patterns.

2. Forty pods serve a 70B model and restart often. Compare baking weights into the image, a sidecar loader to local NVMe, and an init container download. Which do you pick?

   Show answer

   Pick the sidecar with a node-local NVMe cache: the first pod on a node pulls and later pods reuse it, and the image stays slim. Baking gives atomic deploys but images of 50 to 200 GB. An init download makes every restart slow and depends on the registry being up. See image management with weights.

3. You must ship a new vLLM version for eight replicas that each use four GPUs. Do you choose blue/green or canary, and what does each cost?

   Show answer

   A canary needs a small surge, around a quarter more, and gives metric gates at each step but a longer rollout. Blue/green briefly doubles capacity to 64 GPUs and gives an instant flip back. Pick blue/green only for a big-bang change when spare capacity is cheap. See blue/green and canary.

4. A new model version passes its canary and soak. Forty minutes later quality complaints arrive. What must already exist for a safe rollback?

   Show answer

   The previous image, weights and engine files must still be available, and rollback must be one command or revert PR that the team has practised. An analysis template or burn-rate alert can trigger it, and the rollback is announced in the incident channel with the reason. See rollback strategy.

5. A training job is slower than expected. Which tool do you reach for first, which next, and why not attach one to a live production replica?

   Show answer

   Start with nvidia-smi or dcgmi for a coarse view, then Nsight Systems for the whole-app timeline across kernels and NCCL events. Use Nsight Compute only for a specific kernel. Profiler overhead is real, so reproduce load on a canary or drained node. See profilers.

6. Latency alarms fire minutes after a deploy. In what order do you act, and why not find the root cause first?

   Show answer

   Acknowledge, open an incident channel, name an incident commander, then stop the bleeding: roll back the deploy or drain the bad node. Restoring service comes before understanding, and the post-incident review finds the cause and produces owned actions. See incident response.

Takeaway

• Production fleets run several clusters, split by purpose, so blast radius and change cadence stay separate.
• Node pools split by accelerator, and workloads reach them through selectors, taints and spread constraints.
• Ship weights through a sidecar loader to a node-local NVMe cache, and pin images and weights by digest.
• Canary gives metric gates at lower cost, and blue/green gives an instant flip back at the price of double capacity.
• A rollback is a command that has been practised, with the previous artifacts kept through canary and soak.
• Name the profiler for each question: nvidia-smi, dcgmi, Nsight Systems, Nsight Compute, the PyTorch profiler and py-spy.
• In an incident, stop the bleeding first, and treat runbooks and page budgets as part of the system.

Next the guide covers the company-specific context that frames every answer. → Domain Context

Further reading

• Canary deployment strategy — Argo Rollouts docs. Weighted steps, pauses and automated analysis for progressive delivery.
• Safely drain a node — Kubernetes docs. Cordon, drain and how disruption budgets are respected.
• Nsight Systems user guide — NVIDIA docs. Timeline profiling of CUDA, NCCL and CPU activity.
• Nsight Compute — NVIDIA docs. Kernel-level analysis of throughput, occupancy and stalls.
• Managing Incidents — Google SRE Book, 2016. Roles, communication and the case for mitigating before diagnosing.
• Postmortem Culture — Google SRE Book, 2016. Blameless reviews and action items that get done.