Error Handling & Failure Modes

Rust-style error handling

Rust's Result<T, E> + the ? operator give you explicit, typed error propagation. The interview shorthand:

• Errors are values. They appear in the type signature. Callers cannot ignore them by accident.
• Two error styles: thiserror for libraries (structured, named variants), anyhow for application code (single boxed error, easy chaining).
• Panics are for "the program is in an impossible state" — not for normal failure paths.

// Library-style: thiserror

#[derive(thiserror::Error, Debug)]
pub enum BackendError {
    #[error("network: {0}")]
    Network(#[from] reqwest::Error),
    #[error("timeout after {0:?}")]
    Timeout(std::time::Duration),
    #[error("upstream {status}: {body}")]
    Status { status: u16, body: String },
    #[error("invalid response: {0}")]
    Decode(#[from] serde_json::Error),
}

async fn call(client: &reqwest::Client, url: &str) -> Result<Resp, BackendError> {
    let resp = client.get(url).send().await?; // Network from
    if !resp.status().is_success() {
        return Err(BackendError::Status {
            status: resp.status().as_u16(),
            body: resp.text().await.unwrap_or_default(),
        });
    }
    Ok(resp.json().await?) // Decode from
}

For service code, prefer thiserror — the caller needs to match on the error to make decisions (retry? circuit-break? log and continue?). An untyped anyhow::Error loses that.

thiserror vs anyhow — when to use which

Common pattern in a service: thiserror for the public service API, anyhow inside test code and CLI binaries.

Never panic in service code

A panic inside an async task aborts the task. In a multi-tenant service, that's one user's request failing — not the whole process — but it leaves the system in an unpredictable state (held locks, half-emitted events). Discipline:

• No .unwrap() in service code. Use ? with an explicit error type, or .expect("invariant: ...") only for genuinely-impossible cases.
• No array indexing without bounds checks; use .get() + ?.
• Validate inputs at the boundary. Once past the front door, types should make invalid states unrepresentable.
• Catch panics at the task boundary with tokio::task::JoinHandle; log them as critical alerts.
• Set panic = "abort" in release profile if you'd rather crash the process than continue in an unknown state.

Timeouts and cancellation in Tokio

Cancellation in Rust is cooperative — a future can only be cancelled at an .await point. Dropping the future cancels it.

use std::time::Duration;
use tokio::time::timeout;

// timeout returns Err(Elapsed) if the inner future hasn't completed.
// Dropping the inner future cancels it.
match timeout(Duration::from_millis(500), call_backend()).await {
    Ok(Ok(resp))   => serve(resp),
    Ok(Err(e))     => log_and_fail(e),
    Err(_elapsed)  => serve_timeout_error(),
}

Tower has a Timeout layer that wraps the service-level future:

use tower::ServiceBuilder;
use std::time::Duration;

let svc = ServiceBuilder::new()
    .timeout(Duration::from_secs(5))
    .service(backend);

For LLM streaming, "timeout" is more nuanced — you want a TTFT timeout (no token in 2s = bail) and a no-progress-for-X timeout (no token in 30s mid-stream = bail), not just a total-time timeout.

Structured cancellation

"Structured" means: when a parent task is cancelled, all its child tasks are cancelled too. Achievable in Rust with explicit cancellation tokens.

use tokio_util::sync::CancellationToken;

async fn agent_run(token: CancellationToken) -> Result<(), Error> {
    let child = token.child_token();
    let backend_call = tokio::spawn(call_backend(child.clone()));

    tokio::select! {
        _ = token.cancelled() => {
            child.cancel();
            backend_call.await.ok();
            Err(Error::Cancelled)
        }
        r = backend_call => r.unwrap(),
    }
}

async fn call_backend(token: CancellationToken) -> Result<Resp, Error> {
    tokio::select! {
        _ = token.cancelled() => Err(Error::Cancelled),
        r = actually_call() => r,
    }
}

The pattern: pass a cancellation token down through every async function. Each layer races its work against token.cancelled(). Cancelling the parent token cascades.

For HTTP servers: when the client disconnects, the request future is dropped, which drops your nested calls, which cancels them. Tonic and hyper handle this natively if you don't spawn work that escapes the request future.

Retry budgets

Per-call retry is local — "I'll try this 3 times." Retry budget is global — "across the fleet, retries can be at most 10% of total traffic." Without the budget, a partial outage triggers fleet-wide retry, which amplifies load and prevents recovery.

pub struct RetryBudget {
    success_count: std::sync::atomic::AtomicU64,
    retry_count: std::sync::atomic::AtomicU64,
    max_ratio: f64, // e.g. 0.1
}

impl RetryBudget {
    pub fn try_consume(&self) -> bool {
        use std::sync::atomic::Ordering::Relaxed;
        let s = self.success_count.load(Relaxed).max(1) as f64;
        let r = self.retry_count.load(Relaxed) as f64;
        if r / s > self.max_ratio { return false; }
        self.retry_count.fetch_add(1, Relaxed);
        true
    }
}

Combine with per-call retry: per-call decides "should I retry this specific call?", budget decides "is the fleet allowed to retry right now?"

Circuit breakers

Already introduced in 06. The pattern in three states:

• Closed — calls pass through; failures counted.
• Open — failure threshold exceeded; calls fail fast without hitting backend.
• Half-open — after cool-down, a single probe is allowed. Success → closed. Failure → open.

Tune carefully. Too sensitive: legitimate transient blips open the breaker for everyone. Too lax: doesn't actually protect under sustained partial outage.

Bulkheads

Borrowed from naval engineering: isolate failure domains so one flooded compartment doesn't sink the ship. In a service:

• Separate thread pools / executors for unrelated workloads. A storm of slow agent runs shouldn't starve the inference-gateway path.
• Separate connection pools per backend. A saturated Postgres pool shouldn't block Redis calls.
• Per-tenant semaphores. One noisy tenant can't consume 100% of the gateway's concurrency budget.
• Per-model-pool concurrency limits. A spike in long-context requests doesn't starve short-context ones.

use std::sync::Arc;
use std::collections::HashMap;
use tokio::sync::Semaphore;

pub struct PerTenantLimiter {
    sems: tokio::sync::RwLock<HashMap<TenantId, Arc<Semaphore>>>,
    default_quota: usize,
}

impl PerTenantLimiter {
    pub async fn acquire(&self, t: TenantId) -> tokio::sync::OwnedSemaphorePermit {
        let sem = {
            let r = self.sems.read().await;
            if let Some(s) = r.get(&t) { s.clone() } else { drop(r); self.create(t).await }
        };
        sem.acquire_owned().await.expect("never closed")
    }
    async fn create(&self, t: TenantId) -> Arc<Semaphore> {
        let mut w = self.sems.write().await;
        w.entry(t)
            .or_insert_with(|| Arc::new(Semaphore::new(self.default_quota)))
            .clone()
    }
}

Preventing cascading failure

Cascade pattern: backend A slows → callers queue up → caller threads/tasks exhaust → upstream service times out → its callers queue up → repeat. Fixes:

• Fail fast under saturation. Tower's LoadShed + ConcurrencyLimit. Return 503 instead of queueing forever.
• Deadline propagation. If the upstream gave you 100ms, don't spend 5s on retries. The work is wasted.
• Bound queues. Bounded channels; bounded HTTP buffers.
• Throttle retries (retry budget).
• Bulkheads so cascades stay contained to one workload.
• Backpressure from downstream → upstream → ingress, all the way out.

Idempotency keys for agent steps

The single most important pattern for agent infra. Every side-effecting tool call carries an idempotency key. Generated by the orchestrator at intent-time, persisted, scoped to the run.

#[derive(Clone, Copy, Eq, PartialEq, Hash)]
pub struct IdempotencyKey(pub uuid::Uuid);

pub struct IdempotencyStore { /* Postgres or Redis */ }

impl IdempotencyStore {
    pub async fn run_once<F, Fut, T>(&self, key: IdempotencyKey, f: F)
        -> Result<T, IdemError>
    where
        F: FnOnce() -> Fut,
        Fut: std::future::Future<Output = Result<T, IdemError>>,
        T: serde::Serialize + serde::de::DeserializeOwned + Clone,
    {
        if let Some(prev) = self.lookup(key).await? {
            return Ok(serde_json::from_value(prev)?);
        }
        // Mark in-progress, with a TTL on the lock
        self.mark_in_progress(key).await?;
        let out = f().await?;
        self.record_result(key, serde_json::to_value(&out)?).await?;
        Ok(out)
    }
}

Failure modes to mention in an interview:

• Crash mid-call — next attempt sees "in progress"; wait or override with a grace period.
• Two-phase commit with the downstream — for fully end-to-end idempotency, the downstream API also takes the key.
• Key collision across runs — namespace by (run_id, step_id).
• Retention — TTLs balance memory vs the latest retry window your callers might use.

✓ Check yourself

Answer from memory before opening each one.

1. A function does two seconds of CPU work with no await inside, and you wrap it in a 100 ms timeout. What does the caller observe, and why?

   Show answer

   The work is not interrupted; the timeout fires only when the future next yields. Cancellation in Rust is cooperative, so a future can be dropped only at an await point. Long CPU work needs a yield point or the blocking pool. See Timeouts and cancellation in Tokio.

2. Why is a single total-time timeout the wrong tool for a streamed completion, and which timeouts replace it?

   Show answer

   A healthy long generation and a stalled one look the same to a total timer. Use a time-to-first-token timeout, for example two seconds, and a no-progress timeout between tokens, for example thirty seconds. Each bails on the failure it detects without cutting off good long streams. See Timeouts and cancellation in Tokio.

3. The service API returns anyhow::Error. A caller must choose between retrying, tripping a breaker and failing the request. Why does this design block that, and what do you change?

   Show answer

   anyhow erases the cause into one boxed type, so callers cannot match on it. Define a thiserror enum with named variants for network, timeout, status and decode failures, so the caller branches on each. Keep anyhow for binaries and tests. See thiserror vs anyhow.

4. A backend fails every call and each of 1,000 instances retries three times. Roughly how much load does the backend see, and what changes with a 10% retry budget?

   Show answer

   Each call is attempted four times, so the backend sees about four times the traffic just when it is weakest. A fleet-wide budget of 10% caps retries at about a tenth of requests, so load stays near 1.1 times. Per-call retries decide whether to retry; the budget decides whether the fleet may. See Retry budgets.

5. One tenant sends a burst of long-context requests and another tenant's short requests start timing out. Which isolation do you add, and where?

   Show answer

   Add bulkheads: a per-tenant semaphore so no tenant can take the whole concurrency budget, and a per-model-pool limit so long-context requests cannot starve short ones. Also keep separate connection pools per backend so one saturated pool does not block the rest. See Bulkheads.

6. The idempotency store shows a key as in progress left by a crashed attempt. Should the retry wait or override after a grace period, and what decides?

   Show answer

   Waiting is safe but can stall the run if the first attempt never finishes. Overriding after a grace period unblocks it but may repeat the side effect, so it is acceptable only if the downstream takes the same key and dedupes. Choose by the cost of a duplicate against the cost of a stall. See Idempotency keys for agent steps.

Takeaway

• Rust errors are values in the type signature: use a thiserror enum where callers branch on the cause and anyhow where they do not, and never unwrap in service code.
• Cancellation is cooperative and happens at await points; propagate a token down so cancelling a parent cancels every child.
• Streaming calls need a time-to-first-token timeout and a no-progress timeout, not only a total timer.
• Retries need a fleet-wide budget as well as a per-call limit, because without one a partial outage multiplies load by the retry count.
• Contain failure with circuit breakers per backend, bulkheads per tenant and pool, bounded queues, load shedding and deadline propagation.
• Every side-effecting tool call carries an idempotency key scoped to the run and step, and the downstream should honour the same key.

Failure handling needs a record of what happened and bounds on what can happen, which is the subject of the next chapter. → Governance & audit

Further reading

• Error Handling — The Rust Programming Language, Rust project. Result, the question-mark operator and when to panic.
• thiserror — docs.rs. Deriving structured error enums with source chaining.
• CancellationToken — docs.rs. Child tokens and cascading cancellation.
• Timeouts, retries and backoff with jitter — Amazon Builders' Library. Retry amplification and how backoff and jitter limit it.
• Addressing Cascading Failures — Google SRE Book, 2016. How overload spreads and what stops it, including load shedding and retry limits.
• Handling Overload — Google SRE Book, 2016. Per-client limits, criticality and graceful degradation.