Core Fundamentals

Ownership and borrowing

Rust's core idea: every value has a single owner; references are either one mutable &mut T or any number of shared &T at a time, never both. The compiler enforces this. The consequence: no data races at compile time, no use-after-free, no double-free.

fn consume(s: String) {
    println!("{}", s);
} // s dropped here

fn borrow(s: &str) {
    println!("{}", s);
}

let owned = String::from("hello");
borrow(&owned);   // OK — shared borrow
consume(owned);   // moves ownership
// borrow(&owned); // compile error — owned is gone

The interview answer when asked "what's special about Rust": ownership + borrowing give you memory safety without a garbage collector. That's the whole pitch for the language in latency-sensitive infra.

Lifetimes

A lifetime is the compiler's way of asking "does this reference live long enough?" Most of the time they're inferred. They show up explicitly when a return value's reference depends on input lifetimes:

// The returned slice can't outlive the input slice.
fn first_word<'a>(s: &'a str) -> &'a str {
    s.split_whitespace().next().unwrap_or("")
}

For interviews: know that 'static means "lives for the whole program," know elision rules exist (you don't need to memorize them), and know that lifetimes are not runtime — they're compile-time annotations the borrow checker uses.

Arc, Mutex, RwLock — sharing state across tasks

Tokio tasks run on a shared pool of threads, so shared state must be thread-safe. The standard kit:

Channels

Channels are how Rust tasks communicate without sharing state. Four flavors you'll see:

Bounded vs unbounded: always prefer bounded. Unbounded channels are infinite buffers — under load they become unbounded memory growth. Bounded channels apply backpressure (sender awaits when full), which is what you want.

use tokio::sync::mpsc;

let (tx, mut rx) = mpsc::channel::<Request>(1024); // bounded

// producer
tokio::spawn(async move {
    for req in incoming {
        tx.send(req).await.expect("receiver dropped"); // awaits when full
    }
});

// consumer
while let Some(req) = rx.recv().await {
    handle(req).await;
}

Tokio async basics

You will be expected to talk about this fluently. The minimum:

• Future — a value that, when polled, eventually produces a result. Rust futures are lazy: nothing happens until something polls them (or they're spawned on a runtime).
• async fn — desugars to a function returning impl Future<Output = T>. The compiler generates a state machine.
• .await — yields control if the future isn't ready, lets the executor run other tasks.
• Task — a future the runtime owns and polls. Created with tokio::spawn. Cheap (~hundreds of bytes), can have millions concurrent.
• Executor — the thing that polls futures. Tokio's default is a multi-threaded work-stealing executor.

The core insight: async Rust gives you concurrency without paying per-task threads. Idle tasks don't occupy a thread, so you can have hundreds of thousands of in-flight requests waiting on I/O on a small thread pool. Critical for an inference gateway where most tasks spend their life waiting on a model server.

#[tokio::main]
async fn main() {
    let handle = tokio::spawn(async {
        do_some_io().await
    });
    let result = handle.await.unwrap();
    println!("{:?}", result);
}

See 04-deep-dive-primary for the work-stealing internals, blocking vs cooperative, and backpressure patterns.

The actor pattern

An actor is a task that owns some state and processes messages from a channel. Used heavily in Rust services to avoid shared-mutable state.

use tokio::sync::{mpsc, oneshot};

enum CacheMsg {
    Get { key: String, reply: oneshot::Sender<Option<String>> },
    Put { key: String, value: String },
}

async fn cache_actor(mut rx: mpsc::Receiver<CacheMsg>) {
    let mut store = std::collections::HashMap::new();
    while let Some(msg) = rx.recv().await {
        match msg {
            CacheMsg::Get { key, reply } => {
                let _ = reply.send(store.get(&key).cloned());
            }
            CacheMsg::Put { key, value } => {
                store.insert(key, value);
            }
        }
    }
}

Why this matters: in agent orchestration you'll often want per-run state (the plan, the partial outputs, the cancellation flag) owned by exactly one task. The actor pattern is the idiomatic way.

CAP and consistency models

The interview-grade version, in 90 seconds:

• CAP — in a network partition, you choose either consistency (CP) or availability (AP). You cannot have both. Most production systems are AP with eventual consistency for most state and CP for a small set of critical writes.
• Strong consistency — every read returns the latest committed write. Postgres single-leader, Spanner. Expensive.
• Eventual consistency — reads may be stale, will converge. DynamoDB, Cassandra defaults. Cheap, scalable.
• Read-your-writes — a user's own writes are visible to their subsequent reads, even if others see stale.
• Linearizable — global real-time order. Hard.

For AI infra: your run state (which step is the agent on?) wants strong consistency — Postgres or Raft-backed KV. Your eval/audit log is append-only and tolerates eventual consistency. Your inference response cache tolerates staleness.

Idempotency

An operation is idempotent if doing it twice has the same effect as doing it once. In a distributed system, where retries are inevitable and you can't tell "did the network swallow my reply, or did the operation fail?" — idempotency is what saves you.

The pattern: the caller generates a unique key, sends it with the request, the server records "I already did this key → here's the result," and on retry returns the recorded result instead of re-executing.

// Pseudocode
async fn submit(req: ToolCall, idem_key: Uuid) -> Result<Response> {
    if let Some(cached) = store.get(&idem_key).await? {
        return Ok(cached);
    }
    let result = execute(req).await?;
    store.put(&idem_key, &result, TTL).await?;
    Ok(result)
}

For agent infra: every side-effecting tool call gets an idempotency key. Plan-step retries do not duplicate side effects.

Retries and backoff

Naive retry — "if it failed, try again immediately" — causes retry storms. The whole fleet retries at the same time after a partial outage, and the recovering service falls over again. The mitigations:

• Exponential backoff — 100ms, 200ms, 400ms, 800ms... cap at some max.
• Jitter — randomize within the backoff window. Decorrelates retries across the fleet.
• Retry budget — limit total retries as a fraction of total requests (e.g., ≤10%). Stops retry from amplifying load.
• Retry only safe-to-retry errors — 5xx and network errors generally; never blindly retry 4xx.
• Idempotency keys — see above; required for write retries.
• Circuit breaker — see 08-error-handling. After enough failures, stop trying for a while.

use rand::Rng;
use std::time::Duration;
use tokio::time::sleep;

async fn with_backoff<F, Fut, T, E>(mut op: F, max_attempts: u32) -> Result<T, E>
where
    F: FnMut() -> Fut,
    Fut: std::future::Future<Output = Result<T, E>>,
{
    let mut delay_ms = 100u64;
    for attempt in 1..=max_attempts {
        match op().await {
            Ok(v) => return Ok(v),
            Err(e) if attempt == max_attempts => return Err(e),
            Err(_) => {
                let jitter: u64 = rand::thread_rng().gen_range(0..delay_ms);
                sleep(Duration::from_millis(delay_ms + jitter)).await;
                delay_ms = (delay_ms * 2).min(10_000);
            }
        }
    }
    unreachable!()
}

Timeouts and deadlines

Every network call needs a timeout. Every nested call needs a deadline, not just a local timeout. The difference matters:

• Timeout — "I'll wait up to N ms for this call." Local to one call.
• Deadline — "the whole request must finish by wall-clock time T." Propagated through every downstream call. If a downstream call has 50ms budget left, that's what it gets, not the default 5s.

gRPC supports deadlines natively (and tonic forwards them). Without deadline propagation, a slow upstream cascades into wasted downstream work — the user already gave up but the inference call is still running.

use std::time::Duration;
use tokio::time::timeout;

let result = timeout(Duration::from_millis(500), call_model(req)).await;
match result {
    Ok(Ok(resp)) => { /* success */ }
    Ok(Err(e))   => { /* call failed */ }
    Err(_elapsed) => { /* timed out */ }
}

✓ Check yourself

Answer from memory before opening each one.

1. A handler holds a std::sync::Mutex guard while it awaits a call to the model server. What goes wrong, and what are your two options for fixing it?

   Show answer

   The guard stays held while the task is suspended, so other tasks that need the lock block their executor threads and throughput stalls. Either switch to the Tokio mutex for a critical section that must await, or better, restructure so the lock is released before the await. See Arc, Mutex, RwLock.

2. A worker pool reads from an unbounded channel in front of a model server. Traffic doubles for ten minutes. What happens, and what do you change?

   Show answer

   The queue absorbs the excess, so memory and queueing latency grow without limit while the workers stay saturated. Use a bounded channel: when it fills, senders await, and that pushes backpressure to the caller instead of hiding the overload. See Channels.

3. A request has a one-second budget and makes three sequential downstream calls, each with a fixed 500 ms timeout. Describe a failure, and the fix.

   Show answer

   If the first call takes 800 ms, the second and third still wait up to 500 ms each, so work continues long after the caller gave up. Propagate a deadline so each call gets only the time that remains, and fail fast when the budget is spent. See Timeouts and deadlines.

4. A partial outage ends and 500 clients retry with exponential backoff, yet the recovering service falls over again. What two properties are missing, and why does each help?

   Show answer

   Jitter and a retry budget. Without jitter the clients retry in synchronised waves; jitter spreads them within each window. A retry budget caps retries as a fraction of total requests, so retries cannot multiply load on a service that is already weak. See Retries and backoff.

5. A send-payment tool call times out on the client, and the orchestrator retries it. What does the server need so the payment is made once, and what weakness does the simple check-then-execute sketch have?

   Show answer

   The caller sends a unique idempotency key, and the server records the key with its result so a retry returns the stored result instead of executing again. The sketch checks, executes and then stores, which leaves a window where two attempts both run, so a real store reserves the key atomically first. See Idempotency.

6. For an agent platform, choose a consistency level for the run state, the audit log and the inference response cache, and say why they differ.

   Show answer

   Run state needs strong consistency, because two workers must not both believe they own step four. The audit log is append-only and tolerates eventual consistency. The response cache tolerates staleness. Pay for strong consistency only where a wrong read causes a wrong action. See CAP and consistency models.

Takeaway

• Ownership and borrowing, with one owner and either one &mut or many &, give memory safety without a garbage collector, and that is the case for Rust in latency-sensitive infrastructure.
• Async tasks are cheap state machines, so a small thread pool holds hundreds of thousands of requests waiting on a model server.
• Never hold a std mutex guard across an await. Prefer to restructure, or use the Tokio mutex when the critical section must await.
• Use bounded channels, because a full channel makes senders wait and so turns overload into backpressure instead of memory growth.
• Retries need backoff, jitter, a budget and idempotency keys, or they turn a partial outage into a storm and duplicate side effects.
• Propagate a deadline through every nested call; a fixed local timeout wastes work on requests the caller has abandoned.
• Timeout, retry, idempotency and backpressure are the four primitives, and a failure question is answered by combining them for the specific call.

The Tokio runtime, backpressure and tower middleware get a full chapter next. → Deep dive: primary topic

Further reading

• Understanding Ownership — The Rust Programming Language, Rust project. The reference explanation of moves, borrows and the aliasing rule.
• Shared state — Tokio tutorial. When to use a std mutex, a Tokio mutex or a restructure, with the await pitfall.
• tokio::sync::mpsc — docs.rs. Bounded and unbounded channel semantics and the backpressure behaviour.
• Deadlines — gRPC documentation. How deadlines propagate across calls and what a server should do when one expires.
• Timeouts, retries and backoff with jitter — Amazon Builders' Library. Why retries amplify load and how jitter and limits tame it.
• Designing robust and predictable APIs with idempotency — Stripe, 2017. A practical idempotency-key design for write retries.