The Role, Decoded
What a Senior AI Compute Infrastructure Engineer at the company actually owns, why "in-house" matters for a regulated exchange, and how the team's mandate shapes every design answer you'll give.
The team's mandate
"The company is building a dedicated AI Compute and Infrastructure team to power the next generation of model training, inference, evaluation, and experimentation across the exchange. The mandate is simple: make the company's AI ambitions real by building compute infrastructure that is fast, dependable, efficient, and production-grade."
Translation: this team is the compute substrate. Other teams — researchers, ML engineers, product — train and ship models. This team makes sure they can. If a GPU node panics at 3am, the pager goes to this team. If inference p99 latency creeps from 600ms to 1.8s under burst load, this team owns the diagnosis. If next quarter's training run needs 256 H100s for two weeks, this team plans the capacity, signs the reservation, and lights up the cluster.
You're not the modeler. You're the floor under the modelers.
What this team actually owns
Pulled directly from the "opportunity" bullets, with translation:
| JD bullet | What it means in practice |
|---|---|
| "Own and operate GPU and accelerator clusters... drivers, runtimes, kernels, device plugins, node configuration, scheduling primitives, workload isolation" | You touch the NVIDIA driver version, the CUDA runtime, the container toolkit, the k8s device plugin, MIG profiles, taints/tolerations, gang scheduling. When the cluster gets a new node type, you bring it in. |
| "Run models locally on GPUs where strategically and economically preferable" | Build the in-house alternative to OpenAI/Anthropic API calls for use cases that justify it. Maintain a TCO model that says when in-house wins. |
| "Scheduling, orchestration, placement, quota management, utilization systems" | K8s + custom controllers + a fair-share scheduler. Quotas per team, preemption, priority classes, queue depth visibility. |
| "Optimize inference pipelines... vLLM, Triton, TensorRT or equivalent" | Tune paged attention, continuous batching, KV-cache reuse, tensor parallelism shapes, quantization for the workload. |
| "Partner with ML engineers to remove bottlenecks in training, evaluation, batch inference, online inference, production debugging" | You're a force multiplier. When a researcher's job is mysteriously slow, you can nsys profile, read NCCL traces, find the kernel that's stalling on memory. |
| "Build observability for GPU utilization, memory pressure, queue depth, saturation, token throughput, request latency, failed workloads, capacity pressure, spend" | DCGM exporter → Prometheus, Grafana boards per team, cost-per-million-tokens dashboards, alerts on queue depth. |
| "Drive reliability, incident response, alerting, runbooks, post-incident improvements" | On-call rotation, runbooks for CUDA OOM / NCCL hangs / driver mismatches, PIRs that change the system. |
| "Evaluate and integrate new hardware, instance families, accelerators, runtimes, schedulers, serving frameworks" | When B100s ship, you benchmark them. When TensorRT-LLM gets a new release, you decide if you upgrade. |
| "Tooling that makes GPU usage visible, accountable, easier for internal teams to consume" | Self-serve: launch a training job, see your team's quota, deploy an inference endpoint, all without filing a ticket. |
Why running models in-house matters at the company
Why does a crypto exchange build its own GPU compute team rather than send every prompt to a third-party API? Five reasons. You should be able to recite them in order.
- Privacy. KYC data, transaction history, customer PII, internal investigations. Sending this material through a public LLM API creates contractual, regulatory, and reputational exposure. In-house inference keeps the data inside the trust boundary.
- Latency. Real-time risk decisions, fraud signals, and trading-adjacent UX have hard latency budgets. A 1.5s round trip to a hosted API plus its tail variance is incompatible with those budgets. A locally-served fine-tune on a co-located GPU can answer in 80ms p99.
- Reliability. Hosted APIs have outages and rate limits that the company doesn't control. If a critical compliance review depends on inference, the dependency has to be a first-class operational concern, not an external vendor's status page.
- Cost. At high token volume, hosted APIs become structurally expensive. A well-utilized H100 cluster running an open-weights model can be 3-10x cheaper per million tokens than a frontier API for tasks where the open model is good enough.
- Differentiation. The interesting things the company can build — fine-tuned trading copilots, custom embedding models for KYC matching, domain-specific safety classifiers — require infrastructure that supports custom models and custom serving stacks.
When someone asks "why would you serve in-house instead of just calling Anthropic?", lead with privacy and latency, then mention reliability, cost at scale, and differentiation. Don't say "because it's cheaper" — that's wrong at small scale and oversimplifies at large scale.
Who you interface with
- AI/ML researchers — they need GPUs, fast iteration, clean profiling tools. They are your first customers. If they can't get an interactive H100 session in five minutes, you've failed.
- ML engineers — they ship the inference services. You give them serving frameworks, deploy patterns, capacity, and the dashboard that proves it's healthy.
- Platform engineering — owns the general Kubernetes / CI / observability stack. You inherit and extend their substrate; you don't fork it.
- Security — they care about model exfil, supply chain, secrets, multi-tenant isolation. They'll review your IAM, your image provenance, and your logging policy.
- Product — they don't care how many SMs an H100 has. They care that the latency budget for a feature is met within the agreed cost. You translate.
- Finance / capacity planning — reserved capacity, savings plans, capacity blocks, depreciation on on-prem hardware. You're the input.
The stack, decoded
You won't be quizzed on every line item. You should be able to draw a defensible diagram with these blocks labeled correctly.
Hardware substrate
NVIDIA H100 / H200 SXM nodes wired with NVLink and a high-bandwidth fabric (InfiniBand or RoCE), behind PCIe Gen5. A100s for cheaper inference and dev. Possibly Trainium / TPU / Gaudi for specific workloads. Local NVMe for KV-cache spill and model artifact staging.
Runtime layer
Linux + NVIDIA driver + CUDA + NCCL + cuDNN. Containers via nvidia-container-toolkit. Drivers and toolkit versions are operationally significant — a mismatch is a half-day outage.
Orchestration
Kubernetes with the NVIDIA device plugin, gang scheduling (Volcano / Kueue / Yunikorn), MIG/MPS for partitioning, priority classes, taints/tolerations for accelerator type. Operators for serving frameworks (KServe).
Serving frameworks
vLLM for LLM serving with paged attention and continuous batching. Triton Inference Server for multi-model, multi-backend (TensorRT, ONNX, PyTorch, Python). TensorRT / TensorRT-LLM for compiled, quantized, kernel-fused execution. KServe as the CRD layer for "deploy a model" UX. Ray Serve for complex multi-stage Python pipelines.
Observability
DCGM exporter → Prometheus → Grafana. OpenTelemetry traces for inference request paths. Logs into Loki/ELK/Datadog. Profilers: Nsight Systems, Nsight Compute, PyTorch profiler.
Cost & capacity
Reserved capacity contracts, capacity blocks (AWS), committed-use discounts (GCP), per-team chargeback, idle-GPU alerts, savings-plan modeling.
The cost-perf-reliability triangle
Every interesting design question in this role is a tradeoff between three pressures:
- Cost — $/GPU-hour, idle utilization, reserved vs on-demand mix.
- Performance — latency (p50/p95/p99, TTFT), throughput (tokens/sec, RPS).
- Reliability — availability, predictable tail, recovery time, blast radius.
When asked to design something, name the triangle out loud. "Before I pick a serving stack I want to clarify the dominant pressure here — is this a latency-bounded interactive workload, a throughput-bounded batch workload, or a reliability-critical compliance path? My answer changes." That single sentence demonstrates seniority.
The seniority clause
The JD asks for 5+ years of infrastructure engineering with significant GPU compute, ML infrastructure, distributed systems, or HPC time. If you don't have all of that, the reframe lives in chapter 02.
The short version: this is a senior IC role on a small, high-leverage team. They are hiring for judgment under operational pressure, not for bingo coverage of every framework. Your job in the interview is to demonstrate that judgment with the surface area you do have, name your gaps cleanly, and show that you close them fast.
What to ask them
- "What does the current fleet look like — node types, scale, mix of training vs inference, on-prem vs cloud — and what's the next hardware change you anticipate?"
- "What workloads are running in-house today vs going to external APIs, and what's the criteria for moving a workload from external to in-house?"
- "What's the current GPU utilization picture across the fleet — average SM-active, how much capacity is reserved vs spot, where are the biggest idle pockets?"
- "What does an on-call shift on this team usually look like — what fraction is driver/runtime issues vs scheduling vs serving vs capacity?"
- "Where is the biggest unsolved problem in the team's first six months? If I joined, where would I have the most leverage?"
These questions show you're thinking like the engineer who will own the system, not the candidate trying to pass the round.