Governance & Audit

Why governance matters at the company specifically

The company is a regulated financial-services platform. Even though this role isn't compliance-facing, the environment around it is:

• Customer data — KYC documents, transaction records, support tickets. If inference touches them, the trail must be auditable.
• Internal model weights — fine-tuned models on company data are intellectual property and potentially regulated artifacts.
• SOC 2, ISO 27001, jurisdictional licenses — controls and evidence required.
• Always-on availability — the exchange runs 24/7. Compute outages have a customer impact, not just an engineering inconvenience.

A senior infra engineer here demonstrates that they think about how the system is run, audited, and recovered, not just how it's built.

Capacity governance

GPUs are expensive and scarce. Governance is how you decide who gets how much and verify it.

• Quota per team — declared, reviewed quarterly, enforced via Kueue/Volcano queues + ResourceQuota.
• Forecasted demand — each team submits a quarterly demand plan; you reconcile to the cluster.
• Reservation strategy — reserved capacity covers baseline; on-demand/spot covers peaks. Document the mix and the breakeven.
• Capacity reviews — monthly: actual vs forecast, top wasters, top growers, idle reclaim opportunities.
• Procurement input — when usage trends up, you brief finance on the case for more reserved capacity or a new contract.

Cost attribution & chargeback

Without per-team cost visibility, no team has reason to optimize. Build the attribution.

• Tag everything — every pod, namespace, node group carries a team, cost-center, product label.
• Compute the unit cost — GPU-hours per team per day from Prometheus, multiplied by blended $/GPU-hour from your reservation mix.
• Cost dashboards per team, surfaced to engineering leads weekly.
• Showback (visibility) before chargeback (real money) — surface costs internally first; only formalize chargeback once data is reliable.
• Top-N waste reports — idle reserved GPUs, low-utilization deployments, over-provisioned replicas.

Tools: Kubecost, OpenCost, custom Prometheus + Grafana. The principle is the same: unit cost × allocated units, attributed to the smallest accountable owner.

Multi-tenant isolation — what to actually promise

Be precise about isolation, both internally and in interviews. Vague promises ("don't worry, it's isolated") will be picked apart.

Change management for driver/kernel/runtime upgrades

The team's most production-shaking actions are upgrades. The discipline:

1. Versioned changelog — every driver, CUDA toolkit, container toolkit, kubelet, vLLM, Triton version that ships, with a change description and risk note.
2. Gated rollout — dev → staging → 1% canary → 10% → 50% → 100%. Soak time per gate.
3. Pre-defined rollback — every upgrade ticket includes the rollback command. Practiced, not theoretical.
4. Owner + reviewer on every change touching the GPU stack.
5. Change freeze windows around critical business events (market events, releases, audits).
6. Post-change verification — perf regression suite + 24h soak + a "go/no-go" status report before the next gate.

SLOs, error budgets, alert hygiene

Define SLOs per service tier; defend them with error budgets.

Alerting rules:

• Alert on user-visible symptoms, not internal saturation. p99 latency breach pages. CPU at 90% on its own doesn't.
• Burn-rate alerts on error budget — page when you'd exhaust the month's budget in < 1 hour.
• Severity tiers — P0 (paging), P1 (next-business-hour), P2 (ticket). Tune the page count; on-call sustainability is part of the SLO.
• Silenced alerts during planned change with audit trail.

Runbooks

If it happens twice, it gets a runbook. The runbook lives next to the alert and contains:

• Symptom — what the alert says.
• Likely causes in priority order.
• Diagnostic commands — copy-pasteable.
• Remediation steps — copy-pasteable, idempotent.
• Escalation criteria — when to wake someone else up.
• Last-updated timestamp — stale runbooks are dangerous.

Examples of GPU-fleet runbooks every team needs:

• "NVIDIA driver upgrade rollout"
• "Node Xid storm — drain and quarantine"
• "vLLM replica stuck — restart procedure"
• "Online inference p95 spike — diagnostic tree"
• "Cluster running out of GPUs — load-shed playbook"
• "NCCL hang in production training — triage"

Security — exfil, supply chain, secrets

• Model exfil prevention — model weights are expensive IP. Egress controls from training nodes; signed artifacts; access logs on weight pulls.
• Image supply chain — sign container images; only trusted registries; pin digests, not tags. vllm:latest is a vulnerability.
• Secrets — model registry credentials, S3 keys, observability tokens. Never in env vars committed to git; use the platform's secret manager.
• Audit logging — every kubectl access, every model deployment, every weight pull, durable and queryable.
• Inbound network policy — gateways only; serving pods are not reachable from arbitrary sources.
• Driver and runtime CVEs — subscribe to NVIDIA security bulletins; patch cadence aligned with the kernel CVE process.

Prompt logging policy

Subtle but important. Inference servers can log every prompt and every response. That's useful for debugging and evals — and dangerous from a privacy and compliance perspective.

• Default off for production-customer-facing services. Default on for internal eval traffic.
• PII handling — if you log, redact or hash known PII fields before persistence.
• Retention — bounded. 14 days for debug logs is typical; longer needs justification.
• Per-tenant opt-in — some teams need rich logging for evals; some can't have it.
• Access controls — prompt logs are sensitive; ACL them tighter than ordinary application logs.

Regulated-environment considerations

You won't be quizzed on regulatory minutiae, but you should sound aware.

• Data residency — EU customer data shouldn't be processed in US-only inference. Multi-region serving has cost and architecture implications.
• Right to erasure — if a customer deletes their account, the systems handling their data — including inference logs — need a path to comply.
• Auditor reproducibility — "show me, for this past investigation, every model run, every prompt, every response, every decision." Your logging and storage have to support that for the retention window.
• Vendor/contract review — sending data to an external API often requires a contract review with the security/legal team. In-house inference avoids that recurring overhead, which is part of the in-house ROI story.