Governance & Audit

Why governance matters at the company specifically

The company is a regulated financial-services platform. Even though this role isn't compliance-facing, the environment around it is:

• Customer data — KYC documents, transaction records, support tickets. If inference touches them, the trail must be auditable.
• Internal model weights — fine-tuned models on company data are intellectual property and potentially regulated artifacts.
• SOC 2, ISO 27001, jurisdictional licenses — controls and evidence required.
• Always-on availability — the exchange runs 24/7. Compute outages have a customer impact, not just an engineering inconvenience.

A senior infra engineer here demonstrates that they think about how the system is run, audited, and recovered, not just how it's built.

Capacity governance

GPUs are expensive and scarce. Governance is how you decide who gets how much and verify it.

• Quota per team — declared, reviewed quarterly, enforced via Kueue/Volcano queues + ResourceQuota.
• Forecasted demand — each team submits a quarterly demand plan; you reconcile to the cluster.
• Reservation strategy — reserved capacity covers baseline; on-demand/spot covers peaks. Document the mix and the breakeven.
• Capacity reviews — monthly: actual vs forecast, top wasters, top growers, idle reclaim opportunities.
• Procurement input — when usage trends up, you brief finance on the case for more reserved capacity or a new contract.

Cost attribution & chargeback

Without per-team cost visibility, no team has reason to optimize. Build the attribution.

• Tag everything — every pod, namespace, node group carries a team, cost-center, product label.
• Compute the unit cost — GPU-hours per team per day from Prometheus, multiplied by blended $/GPU-hour from your reservation mix.
• Cost dashboards per team, surfaced to engineering leads weekly.
• Showback (visibility) before chargeback (real money) — surface costs internally first; only formalize chargeback once data is reliable.
• Top-N waste reports — idle reserved GPUs, low-utilization deployments, over-provisioned replicas.

Tools: Kubecost, OpenCost, custom Prometheus + Grafana. The principle is the same: unit cost × allocated units, attributed to the smallest accountable owner.

Multi-tenant isolation — what to actually promise

Be precise about isolation, both internally and in interviews. Vague promises ("don't worry, it's isolated") will be picked apart.

Change management for driver/kernel/runtime upgrades

The team's most production-shaking actions are upgrades. The discipline:

1. Versioned changelog — every driver, CUDA toolkit, container toolkit, kubelet, vLLM, Triton version that ships, with a change description and risk note.
2. Gated rollout — dev → staging → 1% canary → 10% → 50% → 100%. Soak time per gate.
3. Pre-defined rollback — every upgrade ticket includes the rollback command. Practiced, not theoretical.
4. Owner + reviewer on every change touching the GPU stack.
5. Change freeze windows around critical business events (market events, releases, audits).
6. Post-change verification — perf regression suite + 24h soak + a "go/no-go" status report before the next gate.

SLOs, error budgets, alert hygiene

Define SLOs per service tier; defend them with error budgets.

Alerting rules:

• Alert on user-visible symptoms, not internal saturation. p99 latency breach pages. CPU at 90% on its own doesn't.
• Burn-rate alerts on error budget — page when you'd exhaust the month's budget in < 1 hour.
• Severity tiers — P0 (paging), P1 (next-business-hour), P2 (ticket). Tune the page count; on-call sustainability is part of the SLO.
• Silenced alerts during planned change with audit trail.

Runbooks

If it happens twice, it gets a runbook. The runbook lives next to the alert and contains:

• Symptom — what the alert says.
• Likely causes in priority order.
• Diagnostic commands — copy-pasteable.
• Remediation steps — copy-pasteable, idempotent.
• Escalation criteria — when to wake someone else up.
• Last-updated timestamp — stale runbooks are dangerous.

Examples of GPU-fleet runbooks every team needs:

• "NVIDIA driver upgrade rollout"
• "Node Xid storm — drain and quarantine"
• "vLLM replica stuck — restart procedure"
• "Online inference p95 spike — diagnostic tree"
• "Cluster running out of GPUs — load-shed playbook"
• "NCCL hang in production training — triage"

Security — exfil, supply chain, secrets

• Model exfil prevention — model weights are expensive IP. Egress controls from training nodes; signed artifacts; access logs on weight pulls.
• Image supply chain — sign container images; only trusted registries; pin digests, not tags. vllm:latest is a vulnerability.
• Secrets — model registry credentials, S3 keys, observability tokens. Never in env vars committed to git; use the platform's secret manager.
• Audit logging — every kubectl access, every model deployment, every weight pull, durable and queryable.
• Inbound network policy — gateways only; serving pods are not reachable from arbitrary sources.
• Driver and runtime CVEs — subscribe to NVIDIA security bulletins; patch cadence aligned with the kernel CVE process.

Prompt logging policy

Subtle but important. Inference servers can log every prompt and every response. That's useful for debugging and evals — and dangerous from a privacy and compliance perspective.

• Default off for production-customer-facing services. Default on for internal eval traffic.
• PII handling — if you log, redact or hash known PII fields before persistence.
• Retention — bounded. 14 days for debug logs is typical; longer needs justification.
• Per-tenant opt-in — some teams need rich logging for evals; some can't have it.
• Access controls — prompt logs are sensitive; ACL them tighter than ordinary application logs.

Regulated-environment considerations

You won't be quizzed on regulatory minutiae, but you should sound aware.

• Data residency — EU customer data shouldn't be processed in US-only inference. Multi-region serving has cost and architecture implications.
• Right to erasure — if a customer deletes their account, the systems handling their data — including inference logs — need a path to comply.
• Auditor reproducibility — "show me, for this past investigation, every model run, every prompt, every response, every decision." Your logging and storage have to support that for the retention window.
• Vendor/contract review — sending data to an external API often requires a contract review with the security/legal team. In-house inference avoids that recurring overhead, which is part of the in-house ROI story.

✓ Check yourself

Answer from memory before opening each one.

1. Demand for GPUs swings between 40 and 100 over a month, with a median near 70. How do you split reserved, on-demand and spot, and what do you track?

   Show answer

   Reserve to cover about the median, use savings plans or on-demand for the next band, and send everything above that, plus training and batch, to spot. Track idle reserved GPU-hours as the waste measure and refresh the mix each quarter. Reserving for the peak buys capacity that sits idle most of the month. See capacity governance.

2. Why surface cost as showback before turning on chargeback?

   Show answer

   Chargeback moves real money, so it needs data teams accept as correct. Showback lets you fix tagging gaps and the unit-cost method while nobody is billed, and it gives teams a reason to optimise early. Attribute unit cost times allocated units to the smallest accountable owner. See cost attribution.

3. Three tenants want to share GPUs: KYC document processing, internal notebooks and a batch eval team. Which isolation promise do you make to each, and what does each cost?

   Show answer

   KYC: physical separation or MIG, at the price of bin-packing efficiency or static partitions. Notebooks: logical separation with sharing, accepting no performance guarantee. Batch evals: MPS or time-slicing on shared pools. Promise only what the mechanism guarantees. See isolation promises.

4. A 99.9 percent monthly availability target leaves about 43 minutes of budget. Why page on a p99 breach but not on CPU at 90 percent, and what does a burn-rate alert add?

   Show answer

   Alerts should track user-visible symptoms, since high CPU alone may hurt nobody. A burn-rate alert pages when the budget would run out in about an hour, so a fast burn wakes someone and a slow burn becomes a ticket. See SLOs, error budgets and alert hygiene.

5. An evals team asks for full prompt and response logging on a production service that handles customer data. What is your default, and what would change it?

   Show answer

   Default to metadata only, with content redacted and bounded retention. Allow content logging as a per-service opt-in with a stated reason, a privacy review, redaction of known PII and tight access control. Metadata covers capacity and performance debugging. See prompt logging policy.

6. An auditor wants every model run for a past investigation, and a customer has asked for erasure. How do the two pull against each other?

   Show answer

   Reproducibility needs prompts and responses kept for the retention window, while erasure needs a path to remove one person's data from those logs. The answer is to keep what the audit needs, index it so a customer's records can be found and deleted, and settle the legal basis with compliance. See regulated-environment considerations.

Takeaway

• A regulated, always-on environment means the trail of who ran what, on which data, must be auditable.
• Capacity governance has quotas per team, quarterly demand plans, a reserved baseline sized near the median, and idle reserved hours as the waste measure.
• Without cost attribution no team has a reason to optimise: tag everything, show costs first, and charge back only once the data is trusted.
• Promise only the isolation the mechanism gives, from namespaces up to a dedicated cluster, and name what each level costs.
• Upgrades are the riskiest actions: use a versioned changelog, gated rollout with soak time, a practised rollback and freeze windows.
• Alert on user-visible symptoms and error-budget burn, and write a runbook for anything that happens twice.
• Log request metadata by default and content only by explicit opt-in, and pin images by digest.

Next the guide moves to the coding half of the loop, starting with the fundamentals you will write on a timer. → Coding Fundamentals

Further reading

• Alerting on SLOs — Google SRE Workbook, 2018. Burn-rate alerting and multi-window thresholds.
• OpenCost documentation — OpenCost project. Allocating Kubernetes cost to namespaces and labels.
• Multi-tenancy — Kubernetes docs. What namespaces, quotas and network policy do and do not isolate.
• Resource Quotas — Kubernetes docs. Per-namespace caps, including extended resources such as GPUs.
• Cosign overview — Sigstore docs. Signing and verifying container images for a supply chain control.
• NVIDIA Product Security — NVIDIA. Where driver and runtime security bulletins are published.