Core Fundamentals

Ownership and borrowing

Rust's core idea: every value has a single owner; references are either one mutable &mut T or any number of shared &T at a time, never both. The compiler enforces this. The consequence: no data races at compile time, no use-after-free, no double-free.

fn consume(s: String) {
    println!("{}", s);
} // s dropped here

fn borrow(s: &str) {
    println!("{}", s);
}

let owned = String::from("hello");
borrow(&owned);   // OK — shared borrow
consume(owned);   // moves ownership
// borrow(&owned); // compile error — owned is gone

The interview answer when asked "what's special about Rust": ownership + borrowing give you memory safety without a garbage collector. That's the whole pitch for the language in latency-sensitive infra.

Lifetimes

A lifetime is the compiler's way of asking "does this reference live long enough?" Most of the time they're inferred. They show up explicitly when a return value's reference depends on input lifetimes:

// The returned slice can't outlive the input slice.
fn first_word<'a>(s: &'a str) -> &'a str {
    s.split_whitespace().next().unwrap_or("")
}

For interviews: know that 'static means "lives for the whole program," know elision rules exist (you don't need to memorize them), and know that lifetimes are not runtime — they're compile-time annotations the borrow checker uses.

Arc, Mutex, RwLock — sharing state across tasks

Tokio tasks run on a shared pool of threads, so shared state must be thread-safe. The standard kit:

Channels

Channels are how Rust tasks communicate without sharing state. Four flavors you'll see:

Bounded vs unbounded: always prefer bounded. Unbounded channels are infinite buffers — under load they become unbounded memory growth. Bounded channels apply backpressure (sender awaits when full), which is what you want.

use tokio::sync::mpsc;

let (tx, mut rx) = mpsc::channel::<Request>(1024); // bounded

// producer
tokio::spawn(async move {
    for req in incoming {
        tx.send(req).await.expect("receiver dropped"); // awaits when full
    }
});

// consumer
while let Some(req) = rx.recv().await {
    handle(req).await;
}

Tokio async basics

You will be expected to talk about this fluently. The minimum:

• Future — a value that, when polled, eventually produces a result. Rust futures are lazy: nothing happens until something polls them (or they're spawned on a runtime).
• async fn — desugars to a function returning impl Future<Output = T>. The compiler generates a state machine.
• .await — yields control if the future isn't ready, lets the executor run other tasks.
• Task — a future the runtime owns and polls. Created with tokio::spawn. Cheap (~hundreds of bytes), can have millions concurrent.
• Executor — the thing that polls futures. Tokio's default is a multi-threaded work-stealing executor.

The core insight: async Rust gives you concurrency without paying per-task threads. Idle tasks don't occupy a thread, so you can have hundreds of thousands of in-flight requests waiting on I/O on a small thread pool. Critical for an inference gateway where most tasks spend their life waiting on a model server.

#[tokio::main]
async fn main() {
    let handle = tokio::spawn(async {
        do_some_io().await
    });
    let result = handle.await.unwrap();
    println!("{:?}", result);
}

See 04-deep-dive-primary for the work-stealing internals, blocking vs cooperative, and backpressure patterns.

The actor pattern

An actor is a task that owns some state and processes messages from a channel. Used heavily in Rust services to avoid shared-mutable state.

use tokio::sync::{mpsc, oneshot};

enum CacheMsg {
    Get { key: String, reply: oneshot::Sender<Option<String>> },
    Put { key: String, value: String },
}

async fn cache_actor(mut rx: mpsc::Receiver<CacheMsg>) {
    let mut store = std::collections::HashMap::new();
    while let Some(msg) = rx.recv().await {
        match msg {
            CacheMsg::Get { key, reply } => {
                let _ = reply.send(store.get(&key).cloned());
            }
            CacheMsg::Put { key, value } => {
                store.insert(key, value);
            }
        }
    }
}

Why this matters: in agent orchestration you'll often want per-run state (the plan, the partial outputs, the cancellation flag) owned by exactly one task. The actor pattern is the idiomatic way.

CAP and consistency models

The interview-grade version, in 90 seconds:

• CAP — in a network partition, you choose either consistency (CP) or availability (AP). You cannot have both. Most production systems are AP with eventual consistency for most state and CP for a small set of critical writes.
• Strong consistency — every read returns the latest committed write. Postgres single-leader, Spanner. Expensive.
• Eventual consistency — reads may be stale, will converge. DynamoDB, Cassandra defaults. Cheap, scalable.
• Read-your-writes — a user's own writes are visible to their subsequent reads, even if others see stale.
• Linearizable — global real-time order. Hard.

For AI infra: your run state (which step is the agent on?) wants strong consistency — Postgres or Raft-backed KV. Your eval/audit log is append-only and tolerates eventual consistency. Your inference response cache tolerates staleness.

Idempotency

An operation is idempotent if doing it twice has the same effect as doing it once. In a distributed system, where retries are inevitable and you can't tell "did the network swallow my reply, or did the operation fail?" — idempotency is what saves you.

The pattern: the caller generates a unique key, sends it with the request, the server records "I already did this key → here's the result," and on retry returns the recorded result instead of re-executing.

// Pseudocode
async fn submit(req: ToolCall, idem_key: Uuid) -> Result<Response> {
    if let Some(cached) = store.get(&idem_key).await? {
        return Ok(cached);
    }
    let result = execute(req).await?;
    store.put(&idem_key, &result, TTL).await?;
    Ok(result)
}

For agent infra: every side-effecting tool call gets an idempotency key. Plan-step retries do not duplicate side effects.

Retries and backoff

Naive retry — "if it failed, try again immediately" — causes retry storms. The whole fleet retries at the same time after a partial outage, and the recovering service falls over again. The mitigations:

• Exponential backoff — 100ms, 200ms, 400ms, 800ms... cap at some max.
• Jitter — randomize within the backoff window. Decorrelates retries across the fleet.
• Retry budget — limit total retries as a fraction of total requests (e.g., ≤10%). Stops retry from amplifying load.
• Retry only safe-to-retry errors — 5xx and network errors generally; never blindly retry 4xx.
• Idempotency keys — see above; required for write retries.
• Circuit breaker — see 08-error-handling. After enough failures, stop trying for a while.

use rand::Rng;
use std::time::Duration;
use tokio::time::sleep;

async fn with_backoff<F, Fut, T, E>(mut op: F, max_attempts: u32) -> Result<T, E>
where
    F: FnMut() -> Fut,
    Fut: std::future::Future<Output = Result<T, E>>,
{
    let mut delay_ms = 100u64;
    for attempt in 1..=max_attempts {
        match op().await {
            Ok(v) => return Ok(v),
            Err(e) if attempt == max_attempts => return Err(e),
            Err(_) => {
                let jitter: u64 = rand::thread_rng().gen_range(0..delay_ms);
                sleep(Duration::from_millis(delay_ms + jitter)).await;
                delay_ms = (delay_ms * 2).min(10_000);
            }
        }
    }
    unreachable!()
}

Timeouts and deadlines

Every network call needs a timeout. Every nested call needs a deadline, not just a local timeout. The difference matters:

• Timeout — "I'll wait up to N ms for this call." Local to one call.
• Deadline — "the whole request must finish by wall-clock time T." Propagated through every downstream call. If a downstream call has 50ms budget left, that's what it gets, not the default 5s.

gRPC supports deadlines natively (and tonic forwards them). Without deadline propagation, a slow upstream cascades into wasted downstream work — the user already gave up but the inference call is still running.

use std::time::Duration;
use tokio::time::timeout;

let result = timeout(Duration::from_millis(500), call_model(req)).await;
match result {
    Ok(Ok(resp)) => { /* success */ }
    Ok(Err(e))   => { /* call failed */ }
    Err(_elapsed) => { /* timed out */ }
}