Section A · Orient

The Role, Decoded

What a Senior AI Compute Infrastructure Engineer at the company actually owns, why "in-house" matters for a regulated exchange, and how the team's mandate shapes every design answer you'll give.

The team's mandate

Representative JD language

"The company is building a dedicated AI Compute and Infrastructure team to power the next generation of model training, inference, evaluation, and experimentation across the exchange. The mandate is simple: make the company's AI ambitions real by building compute infrastructure that is fast, dependable, efficient, and production-grade."

Translation: this team is the compute substrate. Other teams — researchers, ML engineers, product — train and ship models. This team makes sure they can. If a GPU node panics at 3am, the pager goes to this team. If inference p99 latency creeps from 600ms to 1.8s under burst load, this team owns the diagnosis. If next quarter's training run needs 256 H100s for two weeks, this team plans the capacity, signs the reservation, and lights up the cluster.

You're not the modeler. You're the floor under the modelers.

What this team actually owns

Pulled directly from the "opportunity" bullets, with translation:

JD bullet	What it means in practice
"Own and operate GPU and accelerator clusters... drivers, runtimes, kernels, device plugins, node configuration, scheduling primitives, workload isolation"	You touch the NVIDIA driver version, the CUDA runtime, the container toolkit, the k8s device plugin, MIG profiles, taints/tolerations, gang scheduling. When the cluster gets a new node type, you bring it in.
"Run models locally on GPUs where strategically and economically preferable"	Build the in-house alternative to OpenAI/Anthropic API calls for use cases that justify it. Maintain a TCO model that says when in-house wins.
"Scheduling, orchestration, placement, quota management, utilization systems"	K8s + custom controllers + a fair-share scheduler. Quotas per team, preemption, priority classes, queue depth visibility.
"Optimize inference pipelines... vLLM, Triton, TensorRT or equivalent"	Tune paged attention, continuous batching, KV-cache reuse, tensor parallelism shapes, quantization for the workload.
"Partner with ML engineers to remove bottlenecks in training, evaluation, batch inference, online inference, production debugging"	You're a force multiplier. When a researcher's job is mysteriously slow, you can `nsys profile`, read NCCL traces, find the kernel that's stalling on memory.
"Build observability for GPU utilization, memory pressure, queue depth, saturation, token throughput, request latency, failed workloads, capacity pressure, spend"	DCGM exporter → Prometheus, Grafana boards per team, cost-per-million-tokens dashboards, alerts on queue depth.
"Drive reliability, incident response, alerting, runbooks, post-incident improvements"	On-call rotation, runbooks for CUDA OOM / NCCL hangs / driver mismatches, PIRs that change the system.
"Evaluate and integrate new hardware, instance families, accelerators, runtimes, schedulers, serving frameworks"	When B100s ship, you benchmark them. When TensorRT-LLM gets a new release, you decide if you upgrade.
"Tooling that makes GPU usage visible, accountable, easier for internal teams to consume"	Self-serve: launch a training job, see your team's quota, deploy an inference endpoint, all without filing a ticket.

Why running models in-house matters at the company

Why does a crypto exchange build its own GPU compute team rather than send every prompt to a third-party API? Five reasons. You should be able to recite them in order.

Privacy. KYC data, transaction history, customer PII, internal investigations. Sending this material through a public LLM API creates contractual, regulatory, and reputational exposure. In-house inference keeps the data inside the trust boundary.
Latency. Real-time risk decisions, fraud signals, and trading-adjacent UX have hard latency budgets. A 1.5s round trip to a hosted API plus its tail variance is incompatible with those budgets. A locally-served fine-tune on a co-located GPU can answer in 80ms p99.
Reliability. Hosted APIs have outages and rate limits that the company doesn't control. If a critical compliance review depends on inference, the dependency has to be a first-class operational concern, not an external vendor's status page.
Cost. At high token volume, hosted APIs become structurally expensive. A well-utilized H100 cluster running an open-weights model can be 3-10x cheaper per million tokens than a frontier API for tasks where the open model is good enough.
Differentiation. The interesting things the company can build — fine-tuned trading copilots, custom embedding models for KYC matching, domain-specific safety classifiers — require infrastructure that supports custom models and custom serving stacks.

How to use this in an interview

When someone asks "why would you serve in-house instead of just calling Anthropic?", lead with privacy and latency, then mention reliability, cost at scale, and differentiation. Don't say "because it's cheaper" — that's wrong at small scale and oversimplifies at large scale.

Who you interface with

AI/ML researchers — they need GPUs, fast iteration, clean profiling tools. They are your first customers. If they can't get an interactive H100 session in five minutes, you've failed.
ML engineers — they ship the inference services. You give them serving frameworks, deploy patterns, capacity, and the dashboard that proves it's healthy.
Platform engineering — owns the general Kubernetes / CI / observability stack. You inherit and extend their substrate; you don't fork it.
Security — they care about model exfil, supply chain, secrets, multi-tenant isolation. They'll review your IAM, your image provenance, and your logging policy.
Product — they don't care how many SMs an H100 has. They care that the latency budget for a feature is met within the agreed cost. You translate.
Finance / capacity planning — reserved capacity, savings plans, capacity blocks, depreciation on on-prem hardware. You're the input.

The stack, decoded

You won't be quizzed on every line item. You should be able to draw a defensible diagram with these blocks labeled correctly.

Hardware substrate

NVIDIA H100 / H200 SXM nodes wired with NVLink and a high-bandwidth fabric (InfiniBand or RoCE), behind PCIe Gen5. A100s for cheaper inference and dev. Possibly Trainium / TPU / Gaudi for specific workloads. Local NVMe for KV-cache spill and model artifact staging.

Runtime layer

Linux + NVIDIA driver + CUDA + NCCL + cuDNN. Containers via nvidia-container-toolkit. Drivers and toolkit versions are operationally significant — a mismatch is a half-day outage.

Orchestration

Kubernetes with the NVIDIA device plugin, gang scheduling (Volcano / Kueue / Yunikorn), MIG/MPS for partitioning, priority classes, taints/tolerations for accelerator type. Operators for serving frameworks (KServe).

Serving frameworks

vLLM for LLM serving with paged attention and continuous batching. Triton Inference Server for multi-model, multi-backend (TensorRT, ONNX, PyTorch, Python). TensorRT / TensorRT-LLM for compiled, quantized, kernel-fused execution. KServe as the CRD layer for "deploy a model" UX. Ray Serve for complex multi-stage Python pipelines.

Observability

DCGM exporter → Prometheus → Grafana. OpenTelemetry traces for inference request paths. Logs into Loki/ELK/Datadog. Profilers: Nsight Systems, Nsight Compute, PyTorch profiler.

Cost & capacity

Reserved capacity contracts, capacity blocks (AWS), committed-use discounts (GCP), per-team chargeback, idle-GPU alerts, savings-plan modeling.

The cost-perf-reliability triangle

Every interesting design question in this role is a tradeoff between three pressures:

Cost — $/GPU-hour, idle utilization, reserved vs on-demand mix.
Performance — latency (p50/p95/p99, TTFT), throughput (tokens/sec, RPS).
Reliability — availability, predictable tail, recovery time, blast radius.

The interview test

When asked to design something, name the triangle out loud. "Before I pick a serving stack I want to clarify the dominant pressure here — is this a latency-bounded interactive workload, a throughput-bounded batch workload, or a reliability-critical compliance path? My answer changes." That single sentence demonstrates seniority.

The seniority clause

The JD asks for 5+ years of infrastructure engineering with significant GPU compute, ML infrastructure, distributed systems, or HPC time. If you don't have all of that, the reframe lives in chapter 02.

The short version: this is a senior IC role on a small, high-leverage team. They are hiring for judgment under operational pressure, not for bingo coverage of every framework. Your job in the interview is to demonstrate that judgment with the surface area you do have, name your gaps cleanly, and show that you close them fast.

What to ask them

"What does the current fleet look like — node types, scale, mix of training vs inference, on-prem vs cloud — and what's the next hardware change you anticipate?"
"What workloads are running in-house today vs going to external APIs, and what's the criteria for moving a workload from external to in-house?"
"What's the current GPU utilization picture across the fleet — average SM-active, how much capacity is reserved vs spot, where are the biggest idle pockets?"
"What does an on-call shift on this team usually look like — what fraction is driver/runtime issues vs scheduling vs serving vs capacity?"
"Where is the biggest unsolved problem in the team's first six months? If I joined, where would I have the most leverage?"

These questions show you're thinking like the engineer who will own the system, not the candidate trying to pass the round.

✓ Check yourself

Answer from memory before opening each one.

A product lead asks why the company should not send every prompt, including KYC review, to a hosted model API. Which two reasons do you lead with, and why is "it is cheaper" the weakest opening?

Show answer
Lead with privacy and latency. KYC data and investigation material should stay inside the trust boundary, and risk decisions have latency budgets a hosted round trip cannot meet. Cost only favours in-house at high volume with well-utilised GPUs, so at small scale the claim is simply wrong. See why in-house matters.
A team's API bill is large and they propose moving the workload onto your cluster. What do you ask before agreeing?

Show answer
Ask about token volume, whether an open-weights model is good enough for the task, and what utilisation the GPUs would reach. The 3 to 10 times saving only holds when the hardware is busy and the open model meets the quality bar. Idle reserved GPUs turn the saving into a loss, which is why the team keeps a TCO model. See what the team owns.
A research group tells you next quarter's run needs 256 H100s for two weeks. Which parts of that request are yours to solve, and who else do you pull in?

Show answer
You own the capacity plan, the reservation and bringing the cluster up for the run. Finance and capacity planning decide how the reservation is paid for, and the researchers supply the job shape. Platform engineering stays the owner of the base Kubernetes and observability stack that you extend. See who you interface with.
Why does the guide treat a driver and toolkit version mismatch as a half-day outage rather than a minor annoyance?

Show answer
The driver, CUDA runtime and container toolkit must agree before a pod can use a GPU. A mismatch leaves pods without working GPUs across every node that took the change, and fixing it means draining and replacing nodes. That is why you control rollout of these versions with a canary. See the stack.
Three workloads share the fleet: an interactive assistant, an overnight embedding job and a compliance review path. Which side of the triangle dominates for each, and what do you give up?

Show answer
The assistant is bound by latency, so you pay for headroom and a higher steady cost. The overnight job is bound by throughput, so you pack tightly and accept preemption. The compliance path is bound by reliability, so you accept lower utilisation and higher cost for a predictable tail. Say which pressure dominates before you pick a stack. See the triangle.

Takeaway

This team is the compute substrate: researchers and ML engineers train and ship models, and this team makes sure the GPUs, schedulers and serving stacks under them work.
The mandate covers clusters and runtimes, scheduling and quota, inference optimisation, observability, reliability, hardware evaluation and self-serve tooling, not modelling.
In-house inference wins on privacy and latency first, then reliability, cost at scale and differentiation. Cost alone is a weak argument.
Driver, CUDA and toolkit versions are operationally significant, because a mismatch takes GPUs away from every pod on the affected nodes.
Every design answer sits inside cost, performance and reliability. Name the dominant pressure out loud before you choose.
The role hires for judgment under operational pressure. Show it with the experience you have and name your gaps plainly.

Chapter 02 covers how to present your own background honestly against this role. → Positioning From Scratch