The Role, Decoded

What the team actually does

The AI Infrastructure team at a high-scale fintech is the production layer that turns model weights and clever agent prototypes into systems that survive real users and real traffic. The JD frames it cleanly:

Concretely, that means owning:

• The inference gateway — a high-performance Rust service in front of model servers (vLLM, Triton, TGI, internal). Handles auth, request validation, batching, streaming, retries, cost accounting.
• The orchestration layer — services that execute multi-step agent plans, dispatch tool calls, manage run state, persist trajectories, and resume after crashes.
• The execution sandbox — where agent-issued actions actually run (a tool dispatcher, a sandboxed code-exec environment, a queue for side-effecting actions awaiting approval).
• The observability spine — structured logs, traces, metrics, audit events, replay tooling. Every agent action is traceable.
• The reliability primitives — circuit breakers, retries, hedging, bulkheads, rate limits, kill switches, feature flags.

This is production-flavored Rust SWE work. Not training. Not modeling. Not research prototyping. It's the stuff that runs at 3am when someone's agent is in a retry storm against a partner's API and you have ten seconds before the queue fills.

What this role builds

Strip the JD bullets and the day-to-day looks like:

Position relative to the Agent Systems team

This is the single most important framing to get right. Agent Systems and AI Infrastructure are partner teams, not the same team. The JD calls the handoff out explicitly:

The split looks roughly like:

In interviews, when someone asks an agent-design question, your move is to reframe toward the infrastructure: "That's an Agent Systems decision I'd want their input on; on my side I'd be thinking about how to bound retries, propagate cancellation, and audit each tool call."

Position relative to AI Compute Infrastructure

Many orgs have an adjacent AI Compute team that owns GPUs, schedulers, training clusters, vLLM deployments themselves. The boundary is roughly:

• AI Compute: bare-metal GPU fleets, k8s schedulers for inference servers, model weight management, KV cache infrastructure inside vLLM, low-level CUDA-adjacent concerns.
• AI Infrastructure (you): the Rust services that call those inference servers, multiplex traffic over them, run the agent orchestration above them, and present the public-facing API.

If a question is about model parallelism, KV cache eviction strategies inside vLLM, or GPU memory pressure, that's Compute. If a question is about request batching at the gateway, p99 tail latency under load, or how to fail-over between two inference clusters, that's you.

Why Rust for this role

The JD pins Rust hard ("Strong proficiency in Rust and systems-level programming"). It's not a fashion choice. Three real reasons:

1. Tail-latency control. AI inference gateways sit on the request path for millions of users. P99 latency dominates UX. Garbage-collected runtimes (Go, JVM, Python) introduce GC pauses that show up as p99/p999 spikes. Rust gives predictable latency with no GC.
2. Memory safety without GC. You get the low-overhead profile of C/C++ with compile-time guarantees against the bugs that bring down high-throughput services — use-after-free, data races, double-frees.
3. Concurrency that scales. Tokio + the broader async-Rust ecosystem (tonic, hyper, tower, axum) is engineered for hundreds of thousands of in-flight tasks per process with very low per-task overhead. For agent orchestration where a single user request fans into many tool calls and inference calls, this matters.

The "high-scale, high-stakes" culture

The JD uses the words "high-scale and high-stakes" explicitly. In context that means:

• Millions of users — your service is on the critical path for money-adjacent features. Outages have customer and regulatory cost.
• Strict SLOs — success rate, latency, correctness. Error budgets are real.
• Production-first culture — code review takes failure modes seriously. "What happens when this times out?" is the first question, not the last.
• Crypto context — actions can be irreversible (on-chain), regulated, and high-loss. Bias toward conservative defaults, reversibility, human approval for high-tier actions.

If you walk into the interview talking about cool features and not about failure modes, you're flagged as junior regardless of years on a resume.

The tech stack — what each piece is for

Rust + Tokio

The runtime. Tokio is the dominant async runtime — work-stealing executor, async I/O, timers, channels, sync primitives. You will be expected to talk about it fluently: tasks vs threads, blocking vs cooperative, spawn vs spawn_blocking, the runtime model. See 04-deep-dive-primary.

tonic + Protocol Buffers

gRPC implementation for Rust. The standard way services in this stack talk to each other. Strongly-typed contracts via protobuf, bidi streaming, deadline propagation, interceptors. You'll define .proto files and generate Rust code.

tower + hyper

Tower is the middleware framework — services are Service<Request, Response> and you layer middleware (timeout, retry, rate limit, tracing) onto them. Hyper is the underlying HTTP implementation. Axum is a popular tower-based web framework on top.

OpenTelemetry

Standard for traces, metrics, and (increasingly) logs. tracing crate is the Rust idiom. Spans for every request, propagated trace IDs through every tool call so you can reconstruct what an agent did. See 09-governance-and-audit.

vLLM / Triton / TGI (you call these, you don't run them)

Open-source model servers. As an AI Infra SWE, you call them from Rust — typically over HTTP or gRPC — and worry about batching, streaming responses (SSE), prompt-cache awareness, and failover. You do not typically operate the GPU layer underneath.

Postgres + Redis

Persistence. Postgres for durable run state and audit events. Redis for caches, distributed counters, rate limit token buckets, ephemeral state. See 12-data-pipelines.

Kafka or NATS

Event streams for audit, fan-out, async work. Agent action logs, eval traces, training-data collection often flow through here.

Soft signals in the JD

• "5+ years of high-scale production systems" — they want someone who has been on-call for systems with real load. If you have that, lead with it.
• "High ownership mindset in high-stakes production environment" — they expect you to own the incident, not hand it off.
• "Strong collaboration skills across infrastructure and applied engineering" — translation: you need to work with both ML researchers and platform SREs without alienating either.
• "Nice to have: agent-based or LLM-powered systems" — they know fewer people have done this; it's a bonus, not a gate.
• "Nice to have: 0→1 or platform-building" — they want platform mindset, not just feature-team mindset.

The seniority clause

JDs ask for 5+ years; many qualified candidates don't have exactly that shape. The reframe:

Don't apologize, don't claim experience you don't have. If asked directly, acknowledge once cleanly, redirect to what you do bring (recent Rust depth, distributed-systems instincts, platform thinking, motivation), and let the conversation continue. See 02-positioning-from-scratch for specific language.

What to ask them

Strong, role-fit questions:

1. "Where's the AI Infrastructure / Agent Systems boundary in practice today, and where is it the most fuzzy?"
2. "What's your current p99 latency SLO on the inference gateway, and what's the dominant tail-latency contributor?"
3. "How is the team approaching backpressure between the orchestrator and the inference fleet under traffic spikes?"
4. "What's the audit/replay story for an autonomous agent action a regulator might later question?"
5. "Where do you see the biggest leverage for a new senior engineer in the first 90 days — gateway perf, orchestration reliability, or eval/observability tooling?"
6. "How much of the current stack is Rust, and where are you still on legacy Python services you'd want to migrate?"

These signal you're thinking as a senior engineer about the surface area, not just hunting for a job.