Error Handling & Failure Modes

Rust-style error handling

Rust's Result<T, E> + the ? operator give you explicit, typed error propagation. The interview shorthand:

• Errors are values. They appear in the type signature. Callers cannot ignore them by accident.
• Two error styles: thiserror for libraries (structured, named variants), anyhow for application code (single boxed error, easy chaining).
• Panics are for "the program is in an impossible state" — not for normal failure paths.

// Library-style: thiserror

#[derive(thiserror::Error, Debug)]
pub enum BackendError {
    #[error("network: {0}")]
    Network(#[from] reqwest::Error),
    #[error("timeout after {0:?}")]
    Timeout(std::time::Duration),
    #[error("upstream {status}: {body}")]
    Status { status: u16, body: String },
    #[error("invalid response: {0}")]
    Decode(#[from] serde_json::Error),
}

async fn call(client: &reqwest::Client, url: &str) -> Result<Resp, BackendError> {
    let resp = client.get(url).send().await?; // Network from
    if !resp.status().is_success() {
        return Err(BackendError::Status {
            status: resp.status().as_u16(),
            body: resp.text().await.unwrap_or_default(),
        });
    }
    Ok(resp.json().await?) // Decode from
}

For service code, prefer thiserror — the caller needs to match on the error to make decisions (retry? circuit-break? log and continue?). An untyped anyhow::Error loses that.

thiserror vs anyhow — when to use which

Common pattern in a service: thiserror for the public service API, anyhow inside test code and CLI binaries.

Never panic in service code

A panic inside an async task aborts the task. In a multi-tenant service, that's one user's request failing — not the whole process — but it leaves the system in an unpredictable state (held locks, half-emitted events). Discipline:

• No .unwrap() in service code. Use ? with an explicit error type, or .expect("invariant: ...") only for genuinely-impossible cases.
• No array indexing without bounds checks; use .get() + ?.
• Validate inputs at the boundary. Once past the front door, types should make invalid states unrepresentable.
• Catch panics at the task boundary with tokio::task::JoinHandle; log them as critical alerts.
• Set panic = "abort" in release profile if you'd rather crash the process than continue in an unknown state.

Timeouts and cancellation in Tokio

Cancellation in Rust is cooperative — a future can only be cancelled at an .await point. Dropping the future cancels it.

use std::time::Duration;
use tokio::time::timeout;

// timeout returns Err(Elapsed) if the inner future hasn't completed.
// Dropping the inner future cancels it.
match timeout(Duration::from_millis(500), call_backend()).await {
    Ok(Ok(resp))   => serve(resp),
    Ok(Err(e))     => log_and_fail(e),
    Err(_elapsed)  => serve_timeout_error(),
}

Tower has a Timeout layer that wraps the service-level future:

use tower::ServiceBuilder;
use std::time::Duration;

let svc = ServiceBuilder::new()
    .timeout(Duration::from_secs(5))
    .service(backend);

For LLM streaming, "timeout" is more nuanced — you want a TTFT timeout (no token in 2s = bail) and a no-progress-for-X timeout (no token in 30s mid-stream = bail), not just a total-time timeout.

Structured cancellation

"Structured" means: when a parent task is cancelled, all its child tasks are cancelled too. Achievable in Rust with explicit cancellation tokens.

use tokio_util::sync::CancellationToken;

async fn agent_run(token: CancellationToken) -> Result<(), Error> {
    let child = token.child_token();
    let backend_call = tokio::spawn(call_backend(child.clone()));

    tokio::select! {
        _ = token.cancelled() => {
            child.cancel();
            backend_call.await.ok();
            Err(Error::Cancelled)
        }
        r = backend_call => r.unwrap(),
    }
}

async fn call_backend(token: CancellationToken) -> Result<Resp, Error> {
    tokio::select! {
        _ = token.cancelled() => Err(Error::Cancelled),
        r = actually_call() => r,
    }
}

The pattern: pass a cancellation token down through every async function. Each layer races its work against token.cancelled(). Cancelling the parent token cascades.

For HTTP servers: when the client disconnects, the request future is dropped, which drops your nested calls, which cancels them. Tonic and hyper handle this natively if you don't spawn work that escapes the request future.

Retry budgets

Per-call retry is local — "I'll try this 3 times." Retry budget is global — "across the fleet, retries can be at most 10% of total traffic." Without the budget, a partial outage triggers fleet-wide retry, which amplifies load and prevents recovery.

pub struct RetryBudget {
    success_count: std::sync::atomic::AtomicU64,
    retry_count: std::sync::atomic::AtomicU64,
    max_ratio: f64, // e.g. 0.1
}

impl RetryBudget {
    pub fn try_consume(&self) -> bool {
        use std::sync::atomic::Ordering::Relaxed;
        let s = self.success_count.load(Relaxed).max(1) as f64;
        let r = self.retry_count.load(Relaxed) as f64;
        if r / s > self.max_ratio { return false; }
        self.retry_count.fetch_add(1, Relaxed);
        true
    }
}

Combine with per-call retry: per-call decides "should I retry this specific call?", budget decides "is the fleet allowed to retry right now?"

Circuit breakers

Already introduced in 06. The pattern in three states:

• Closed — calls pass through; failures counted.
• Open — failure threshold exceeded; calls fail fast without hitting backend.
• Half-open — after cool-down, a single probe is allowed. Success → closed. Failure → open.

Tune carefully. Too sensitive: legitimate transient blips open the breaker for everyone. Too lax: doesn't actually protect under sustained partial outage.

Bulkheads

Borrowed from naval engineering: isolate failure domains so one flooded compartment doesn't sink the ship. In a service:

• Separate thread pools / executors for unrelated workloads. A storm of slow agent runs shouldn't starve the inference-gateway path.
• Separate connection pools per backend. A saturated Postgres pool shouldn't block Redis calls.
• Per-tenant semaphores. One noisy tenant can't consume 100% of the gateway's concurrency budget.
• Per-model-pool concurrency limits. A spike in long-context requests doesn't starve short-context ones.

use std::sync::Arc;
use std::collections::HashMap;
use tokio::sync::Semaphore;

pub struct PerTenantLimiter {
    sems: tokio::sync::RwLock<HashMap<TenantId, Arc<Semaphore>>>,
    default_quota: usize,
}

impl PerTenantLimiter {
    pub async fn acquire(&self, t: TenantId) -> tokio::sync::OwnedSemaphorePermit {
        let sem = {
            let r = self.sems.read().await;
            if let Some(s) = r.get(&t) { s.clone() } else { drop(r); self.create(t).await }
        };
        sem.acquire_owned().await.expect("never closed")
    }
    async fn create(&self, t: TenantId) -> Arc<Semaphore> {
        let mut w = self.sems.write().await;
        w.entry(t)
            .or_insert_with(|| Arc::new(Semaphore::new(self.default_quota)))
            .clone()
    }
}

Preventing cascading failure

Cascade pattern: backend A slows → callers queue up → caller threads/tasks exhaust → upstream service times out → its callers queue up → repeat. Fixes:

• Fail fast under saturation. Tower's LoadShed + ConcurrencyLimit. Return 503 instead of queueing forever.
• Deadline propagation. If the upstream gave you 100ms, don't spend 5s on retries. The work is wasted.
• Bound queues. Bounded channels; bounded HTTP buffers.
• Throttle retries (retry budget).
• Bulkheads so cascades stay contained to one workload.
• Backpressure from downstream → upstream → ingress, all the way out.

Idempotency keys for agent steps

The single most important pattern for agent infra. Every side-effecting tool call carries an idempotency key. Generated by the orchestrator at intent-time, persisted, scoped to the run.

#[derive(Clone, Copy, Eq, PartialEq, Hash)]
pub struct IdempotencyKey(pub uuid::Uuid);

pub struct IdempotencyStore { /* Postgres or Redis */ }

impl IdempotencyStore {
    pub async fn run_once<F, Fut, T>(&self, key: IdempotencyKey, f: F)
        -> Result<T, IdemError>
    where
        F: FnOnce() -> Fut,
        Fut: std::future::Future<Output = Result<T, IdemError>>,
        T: serde::Serialize + serde::de::DeserializeOwned + Clone,
    {
        if let Some(prev) = self.lookup(key).await? {
            return Ok(serde_json::from_value(prev)?);
        }
        // Mark in-progress, with a TTL on the lock
        self.mark_in_progress(key).await?;
        let out = f().await?;
        self.record_result(key, serde_json::to_value(&out)?).await?;
        Ok(out)
    }
}

Failure modes to mention in an interview:

• Crash mid-call — next attempt sees "in progress"; wait or override with a grace period.
• Two-phase commit with the downstream — for fully end-to-end idempotency, the downstream API also takes the key.
• Key collision across runs — namespace by (run_id, step_id).
• Retention — TTLs balance memory vs the latest retry window your callers might use.