Practice Interview Questions

"Ownership gives you memory safety without garbage collection. Every value has one owner, references are either one mutable or many shared but never both, and the compiler enforces the rules. For a gateway sitting on the request path for millions of users, that means no GC pauses contributing to p99 tail latency, no use-after-free or data race bugs that would cause non-deterministic outages, and predictable performance under load. It's why Rust is the right pick for this workload over Go or Java."

"Box is single owner, heap allocation, no sharing. Used for trait objects and recursive types. Rc is reference counted with multiple owners but single-threaded — non-atomic counter, not Send. Arc is the same but atomic — works across threads, slightly more overhead. In service code spanning Tokio tasks you basically always reach for Arc; Rc shows up rarely because tasks move across threads and require Send. None of these allow mutation through the shared reference; for that you pair with Mutex or RwLock for cross-thread, or RefCell for single-thread."

"The guard is held across the suspension point, but the task might move to a different worker thread when it resumes. With std::sync::Mutex that creates a deadlock risk and definitely blocks the worker thread during the await — defeating the point of async. Two fixes: use tokio::sync::Mutex if you really must hold a lock across .await, or — better — restructure to drop the guard before awaiting. Pattern: lock, copy out the data you need, drop the guard, then do the I/O."

"Allocations on the hot path are tail-latency tax. Junior Rust takes Strings everywhere. Senior Rust takes &str at API boundaries unless ownership is needed. Cow<str> for the maybe-borrow-maybe-own case. SmallVec and Box<str> have niches. The discipline shows you've thought about per-request allocation cost — which on a gateway running tens of thousands of QPS is real money."

"Multi-threaded executor — by default one OS thread per CPU core. Each thread has a local task queue. Tasks yield at .await points; the thread picks up another task. If a thread runs out of work, it steals from another thread's local queue. Result: hundreds of thousands of in-flight tasks on a small thread pool, as long as they're mostly I/O-bound. The footguns are (1) any CPU-bound work without yields starves the worker, (2) any blocking call blocks the worker — use spawn_blocking for that. For an inference gateway where most tasks wait on a model server, this is the right runtime."

"Cancelling a parent task cascades to all its children. Rust doesn't enforce this; you build it with CancellationToken from tokio-util. The parent passes a token down; each layer races its work against token.cancelled() with select!. Calling cancel() on the parent propagates. For HTTP servers, client disconnect drops the request future, which drops children — so the natural shape of tonic/axum gives you this for free, unless you spawn tasks that escape the request scope. The principle is: don't spawn long-lived work without a token that ties it to its caller."

"Backpressure is the downstream signaling 'slow down' to the upstream. The primary tool in async Rust is the bounded channel — sending on a full channel awaits, that await propagates back through the call chain, eventually the HTTP layer rejects new connections with 503. Other tools: tower's ConcurrencyLimit, LoadShed, RateLimit, and Semaphore for explicit caps. The anti-pattern is unbounded channels — those are infinite buffers that become OOMs under load. If I see an unbounded_channel in a service, that's a red flag."

"For genuinely synchronous CPU-bound or blocking work that you can't make async — tokenization, crypto, large serializations, calling a sync C library. spawn_blocking moves the work to a dedicated thread pool (default 512 threads) so it doesn't starve the async workers. Don't use it as a hammer — wrapping every sync call in spawn_blocking has overhead and doesn't scale infinitely. And don't spawn-blocking trivial work — a few microseconds of sync compute is cheaper to just run inline."

"Client hits the gateway over HTTPS. Edge terminates TLS, mints a request ID. Auth layer validates JWT, picks the tenant. Rate limiter checks the bucket. Routing picks a backend pool based on model; within the pool, prefix-cache-aware consistent hashing picks a node so we hit warm caches. Tower layers add a deadline, a circuit breaker. We post to vLLM's OpenAI-compatible endpoint with stream=true. Tokens come back as SSE; we wrap each into our protocol, push through a bounded channel to the client, stream back. Every step gets a span; the audit event records prompt hash, response hash, model version, cost. On cancellation we drop the upstream future to release GPU time."

"Modern inference servers cache the KV-state of common prompt prefixes — your system prompt, your in-context examples — and reuse it across requests. Cache hits skip the prefill compute for those tokens, which is a big TTFT win. The gateway has to route consistently so that requests sharing a prefix land on the same backend node — random load balancing destroys the cache. Consistent hashing on a prefix hash gives you that without a leader. The other thing: don't mutate the system prompt per request, you'll thrash the cache."

"SSE between gateway and client; the upstream model server typically streams over its own protocol — vLLM is SSE-flavored, Triton can be gRPC server-streaming. The gateway consumes upstream, transforms frames, pushes into a bounded channel, drains to the client. Backpressure: if the client is slow, the channel fills, the upstream consumer awaits, eventually the model server slows down — that's good, it means we're not wasting GPU on a client that's not consuming. Cancellation: client disconnect drops the request future, which cancels the upstream call, which frees the GPU. Heartbeats: emit periodic comments so proxies don't kill idle connections."

"Clarify: SLO targets for TTFT, ITL, success rate. Model mix — short vs long context? Then layer it: edge LB terminates TLS and handles request IDs. Auth + tenant routing as a tower layer. Bounded-concurrency per-tenant semaphore for fairness. Per-model backend pool — vLLM nodes, each with prefix-cache-aware affinity routing via consistent hashing. Per-backend circuit breaker. Streaming over SSE; bounded channel to the client. Per-request span; metrics on RED + tokens + cost; audit event on completion. Postgres for run state if agent runs; Redis for rate-limit token buckets. Capacity: 100k QPS at ~50ms median, ~500ms p99 first-token — that's probably 100-200 gateway pods plus the GPU fleet. I'd validate with a load test before committing the number."

"Run state in Postgres with optimistic concurrency. Events appended to an audit log — Kafka or a Postgres append-only table — after every step. One executor instance owns one run at a time via a Postgres advisory lock or lease. Each step persists before invoking the next model call, so on restart we know exactly where we were. Side-effecting tool calls carry an idempotency key generated at intent time; the tool router records the key on dispatch and returns the recorded result on retry. On startup, executor scans for runs whose lease has expired and resumes them. Stuck-state detection: if a run is in 'executing step N' for more than 5× the median step time, alert and either retry or fail."

"Agent Systems decides what the agent does — prompts, tool schema design, plan strategy, agent-quality evals. I decide how the agent runs at scale — service design, executor reliability, tool dispatcher, state persistence, audit, observability, kill switches. The handoff is: their prototype defines the contract — what tools, what plan shape, what success looks like — and my service provides a hardened executor that runs many of them in parallel with SLO guarantees, idempotency, audit, and recovery. The friction zone is usually 'agent quality eval' vs 'service reliability eval' — task success vs SLOs. I'd own the infra-tier metrics (runs that hit limits, tool failure rates) and partner with them on whatever bridges into agent quality."

"Three states. Closed — calls pass through, failures counted. Open — failure threshold exceeded, calls fail fast without hitting the backend. Half-open — after a cooldown, one probe is allowed; success returns to closed, failure back to open. Tuning matters: too sensitive and you open on transient blips; too lax and you don't actually protect. I'd attach breakers per-backend node so I can route around a single bad node while the pool remains healthy. Combined with a retry budget, you get good protection against cascading failure during partial outages."

"thiserror for library / service code where callers branch on the error — custom enum, named variants, retryable bit, source preserved via #[from]. anyhow for application code where you just want easy chaining with .context() and a single boxed error — top-level binaries, scripts, tests. In a service crate I'd use thiserror at every internal boundary; anyhow only at main(). The reason is that retries, circuit breakers, and audit decisions need to match on error kind — an opaque anyhow doesn't let you do that."

"Five layers. Exponential backoff so you're not hammering the recovering service. Jitter so the fleet doesn't retry in lockstep. Retry budget — limit retries as a fraction of total traffic, like ≤10% — so retries can't amplify load fleet-wide. Only retry safe errors — 5xx and network, never blind 4xx. And circuit breakers above all that, so once a backend is clearly bad, you fail fast and stop trying for a while. Idempotency keys on writes so retries don't duplicate side effects."

"In a distributed system you can't tell if a failed call actually executed or not — the network might have eaten your reply. So writes get idempotency keys. The orchestrator generates a UUID at intent time; the tool router persists key → in_progress, executes, then key → result. On retry with the same key, returns the recorded result. The downstream API should also accept the key for end-to-end idempotency. For agent infra this is non-negotiable — without it, a retried side-effecting tool call duplicates the action, which in a fintech could mean duplicate transfers or duplicate notices."

"In Rust, cancellation = drop. When you drop a future, work stops at the next .await; destructors release resources. For HTTP: client disconnect drops the request future, which drops nested calls. For explicit structured cancellation, use CancellationToken from tokio-util — pass a child token down through each layer, race each layer's work against token.cancelled() in a select. Cancelling the parent token cascades to children. The thing that breaks this is spawning tasks that don't carry the token — those become orphans the parent can't cancel. The rule: don't tokio::spawn long work without a cancellation pathway."

"One trace ID per run, propagated to every step. The root span is orchestrator.run; child spans for each model.call and each tool.dispatch. Tool dispatches that call downstream services propagate W3C trace context, so the trace stays connected. OTel + tracing crate in Rust. Structured logs at info level — run_id, step, tool, status, latency, cost — with prompt and response hashes (bodies stored elsewhere keyed by hash). Metrics: per-tool success rate, per-tool latency histogram, per-run step count, per-run cost. Audit events on a separate append-only stream with the full sequence — that's what a regulator queries six months later."

"RED — Rate, Errors, Duration — for request-driven services. The three you'd put on every endpoint dashboard. USE — Utilization, Saturation, Errors — for resources: CPU, memory, queue depth, KV cache. Together they cover the two questions you actually ask in an incident: 'are users seeing problems' (RED) and 'where's the bottleneck' (USE)."

"Don't log raw prompts and responses by default — log their hashes and lengths. Store bodies in a separately-access-controlled content-addressed store keyed by hash. Tokenize PII identifiers at the boundary — account_id=12345 becomes account_id=ACCT_T0a8f — with a separate mapping that's tightly access-controlled. Redact known patterns (emails, SSNs) before logs leave the app. The audit log keeps the tokens and metadata; the resolver mapping is what gets deleted on a GDPR erasure request, leaving aggregate stats intact but no longer connecting to the individual."

State the approach first: "Lazy refill — compute current tokens on demand based on elapsed time. Wrap in a Mutex for simplicity; for hot keys, switch to atomic CAS or sharded. try_acquire reads now, refills, checks, decrements, returns bool." Then write the code in 11 P1. Mention: distributed version uses a Lua script in Redis.

State the approach: "DashMap of in-flight broadcast::Sender per key. On call: check cache, else check inflight, else become the producer, others subscribe. Producer runs, broadcasts result, removes inflight entry. Failures broadcast as Err to subscribers. Won't cache failures by default." Then write it; pattern in 11 P2 senior variant.

State the approach: "CancellationToken passed to axum's with_graceful_shutdown. Signal handler on SIGTERM/SIGINT triggers cancel. Wrap in timeout for force-exit safety. preStop hook in k8s should sleep ~5s before SIGTERM so the LB has time to drain." Then write it; pattern in 11 P10.

Use a real story. The shape that wins: impact → what you saw → how you scoped → the actual fix → the follow-up.

"At [scale], our [system] started 5xx-ing at [time] — about [N]% of requests over [duration]. I was on-call. First moves: looked at the RED dashboard, saw error rate climbing on one endpoint; checked the trace samples, saw they were all timing out at the same downstream call. Pulled up the downstream's metrics — it was saturated, queue depth climbing. The fix was to engage the circuit breaker manually for that backend (we had a feature flag for it), which immediately failed fast instead of queueing, restoring most of the traffic. Then we drained the backlog. Follow-up: a post-mortem identified that our retry budget was too high — we'd amplified the original problem. We tightened the budget and added a multi-window burn-rate alert that would have paged us 8 minutes earlier."

If you don't have a real incident, be honest:

"I haven't been on-call for a comparable production incident. My closest is [smaller event]. What I'd bring is the playbook — RED dashboard, traces, circuit breaker, post-mortem with retry-budget tightening — which I've internalized but not yet exercised at this scale."

"They own agent quality, I own service quality. When the conversations clash — say, they want a longer max-step budget for better completion rates, I want a lower one for cost predictability — I bring the data: current cost distribution, tail-step-count contribution to p99 latency, error budget impact. The decision is then quantitative: 'with budget B we hit 95% completion but burn $X/day; with B/2 we hit 90% and $X/4.' Then we pick together with product input. I don't fight on opinion; I bring numbers. If I'm worried about a safety property and they're worried about completion, I make the safety case explicit and we either find a design that does both or escalate to leadership."

"Listen and inventory before changing anything. Week 1: shadow on-call, read the most recent five post-mortems, get added to the trace and metrics dashboards, read the gateway code top-to-bottom. Week 2: map the current architecture in a doc, identify the highest-leverage observability or reliability gap that wouldn't require team buy-in to fix. Week 3: ship that — small, useful, demonstrates production seriousness. Week 4: write a 60-day plan with the team, pick one bigger workstream — likely either an orchestrator reliability gap, a perf hotspot in the gateway, or eval/replay tooling. I'd avoid shipping anything in the first month I haven't seen the failure cases of."