Deployment & Ops

Cluster deploy patterns

A production GPU cluster isn't one Kubernetes cluster. Most production fleets run several:

• Production inference — strict SLAs, reserved capacity, change-controlled.
• Research / experimentation — looser, faster iteration, can absorb churn.
• Pre-prod / staging — production-shaped, used as canary.
• Dev / batch — cheaper hardware, spot, mixed.

Separation isolates blast radius and lets each cluster carry its own change cadence. Cluster lifecycle managed via IaC: Terraform for the cluster shell, Helm/Kustomize for the workloads, GitOps (Argo CD / Flux) as the reconciler.

Node pools by accelerator

Within a cluster, separate node pools per accelerator type. Workloads use nodeSelector/affinity + taints to land on the right pool.

# Pinning a workload to the right pool
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-70b
spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-h100-sxm
        node-pool: h100-sxm-online
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: vllm-llama3-70b }

Image management with model weights

Three deployment patterns, each with a tradeoff. Pick deliberately.

Default for production: sidecar with hostPath NVMe cache. Weights tagged by content hash. The first pod on a node pulls; subsequent pods hardlink.

Blue/green & canary inference deploys

Two patterns for shipping a model or serving-stack change without taking the service down.

Blue/green

Run both old (blue) and new (green) at the same time. Traffic flipped at the gateway. Fast rollback (flip back). Expensive (2× capacity briefly). Good for big-bang changes.

Canary

Weighted traffic split — 1% to new, then 10%, 50%, 100%. Watch metrics at each gate. Cheaper than blue/green; longer rollouts; finer-grained.

# Argo Rollouts canary for a vLLM service
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama3-70b }
spec:
  replicas: 8
  strategy:
    canary:
      maxSurge: "25%"
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: latency-and-error-rate
        - setWeight: 25
        - pause: { duration: 30m }
        - setWeight: 50
        - pause: { duration: 1h }
        - setWeight: 100

The analysis template hits Prometheus and fails the rollout if p95 latency or 5xx rate exceeds thresholds.

Rollback strategy

The minimum bar:

• Every deploy is rollbackable with a single command or a single revert PR.
• Previous artifact retained — image, weights, plan files — for at least the duration of the canary plus soak.
• Practiced — at least once per quarter, do an intentional rollback in staging.
• Automated triggers — analysis template, error-budget burn-rate alert, manual button.
• Communicate — the rollback is in the incident channel within 60s of being decided, with the reason.

The observability stack

The boring but essential parts. You should be able to name the pieces and explain what each does.

• Metrics — Prometheus + DCGM exporter + service /metrics endpoints. Long-term storage via Mimir/Thanos/VM.
• Logs — Fluent-bit / Vector → Loki / Elastic / Datadog. Structured JSON.
• Traces — OpenTelemetry collector → Tempo / Jaeger / Honeycomb. Tail-based sampling.
• Dashboards — Grafana. One per service, one per team, one for fleet health.
• Alerts — Alertmanager → PagerDuty / Opsgenie. Tied to SLOs, not raw metrics.
• Synthetics — a probe service hitting each model with canned prompts every minute, watching TTFT and correctness.
• Profiling — Nsight Systems for timeline traces; Nsight Compute for kernel-level. Continuous profiling (Parca / Pyroscope) for long-running services.

Profilers — Nsight, PyTorch, dcgmi, py-spy

You will be asked at some point: "How would you profile this?" Have the toolkit by name.

Incident response

The team will get paged. The discipline keeps the pages short and the learning durable.

1. Acknowledge fast. The page system tracks ack time. Two minutes is the target.
2. Start an incident channel. Slack/Teams channel named with the date. Pin the alert.
3. Assign an Incident Commander. One person owns coordination — even on a small team, one person is "in charge of the incident."
4. Open the runbook. If one exists, work it.
5. Communicate hourly. Status update in the channel even if "still investigating."
6. Stop the bleeding before finding the root cause. Roll back the deploy, drain the bad node, increase capacity — the goal is to restore service first.
7. Once service is restored: mark incident resolved, schedule the PIR (post-incident review) within a week.
8. PIR produces actions. Each action has an owner and a date. Action items show up in the next sprint.

Anatomy of a good runbook

A runbook should be runnable in the middle of the night by someone who is not the expert. Components:

1. Title — what alert triggers this.
2. Severity — P0/P1/P2.
3. Symptoms — what you'll see.
4. Likely causes in order of probability.
5. Diagnostic commands — copy-pasteable, with expected output.
6. Remediation steps — copy-pasteable, idempotent.
7. Escalation — who to wake up and when.
8. Related runbooks — links.
9. Last updated, last verified.

# Excerpt of a "node Xid storm" runbook
#
# Symptom: Alert "DCGMXidErrorsHigh" on node-X
# Likely:
#   1) Failing HBM (rate of Xid 63/64) — drain & replace
#   2) Driver bug (rate of Xid 119/120) — drain, file ticket
#   3) Power glitch (Xid 79 isolated event) — power-cycle node

# Step 1: confirm
ssh node-X dmesg | grep -i 'NVRM: Xid' | tail -20
ssh node-X dcgmi diag -r 1     # quick health check

# Step 2: drain (PDB-respecting)
kubectl cordon node-X
kubectl drain node-X --ignore-daemonsets --delete-emptydir-data --timeout=10m

# Step 3: dispatch
# If diag fails: tag node out of service; open hardware ticket
# If diag passes: gather logs, reboot, run smoke workload

On-call sustainability

A senior engineer protects on-call quality. In an interview, you can demonstrate this by talking about:

• Page-budget hygiene — measure pages/week/engineer; investigate when it rises.
• Symptom-based alerts — only page on user-impacting symptoms.
• Runbook-or-don't-page rule — every page either has a runbook or shouldn't exist.
• Follow-the-sun if the team spans time zones; otherwise rotation cadence that respects sleep.
• Action-item velocity from PIRs — if PIR actions aren't shipping, the same incident will recur.