Applied Patterns

Inference gateway with circuit breakers

The inference gateway sits in front of one or more model backends. Its responsibilities, by layer:

Circuit breakers: after N failures in a window, open the breaker — subsequent calls fail fast without hitting the backend. After a cool-down, half-open: let one probe through. If it succeeds, close.

use std::sync::atomic::{AtomicU64, Ordering};

pub struct CircuitBreaker {
    failure_count: AtomicU64,
    last_failure_ms: AtomicU64,
    threshold: u64,
    cooldown_ms: u64,
}

impl CircuitBreaker {
    pub fn allow(&self) -> bool {
        let failures = self.failure_count.load(Ordering::Relaxed);
        if failures < self.threshold { return true; }
        let last = self.last_failure_ms.load(Ordering::Relaxed);
        let now = now_ms();
        now - last > self.cooldown_ms
    }
    pub fn record_success(&self) {
        self.failure_count.store(0, Ordering::Relaxed);
    }
    pub fn record_failure(&self) {
        self.failure_count.fetch_add(1, Ordering::Relaxed);
        self.last_failure_ms.store(now_ms(), Ordering::Relaxed);
    }
}

In tower, tower::limit and community crates like tower-circuit-breaker give you this as a Layer.

Request hedging

Hedging fires a duplicate request to a second backend after a small delay, takes whichever returns first, cancels the loser. Trades 2x cost for the slow tail to halve tail latency in many real distributions.

use tokio::{select, time::{sleep, Duration}};

async fn hedged_call(backends: &[Backend], req: Req, hedge_after: Duration)
    -> Result<Resp, Error>
{
    let primary = backends[0].call(&req);
    let secondary = async {
        sleep(hedge_after).await;
        backends[1].call(&req).await
    };
    tokio::pin!(primary, secondary);
    select! {
        r = &mut primary   => r,
        r = &mut secondary => r,
    }
}

Rules of thumb:

• Hedge only idempotent (read-only or idempotency-keyed) calls.
• Hedge at the p95 of the latency distribution, not earlier — earlier doubles cost without benefit.
• Cap hedging as a fraction of total traffic (e.g., ≤10%) so a slow backend doesn't trigger fleet-wide doubling.

Agent execution sandbox

When an agent's tool dispatches code, you need isolation. Options, in increasing strength:

1. In-process function call — fine for read-only tools you wrote yourself.
2. WASM sandbox (wasmtime, wasmer) — capability-controlled, fast startup, memory-isolated. Good for arbitrary tool plugins.
3. Container per call (gVisor, Firecracker microVMs) — strong isolation, slow startup. Reserve for "agent runs untrusted code."
4. Out-of-process RPC — the tool is its own service. Isolation by process boundary plus normal RPC controls.

For this fintech/crypto-exchange context, you're most likely doing (1) and (4) — well-defined internal tools called via RPC, with the orchestrator enforcing auth, idempotency, and audit.

Tool router

The tool router is the registry-plus-dispatcher for the agent's available tools. It:

• Looks up tool by name → handler.
• Validates args against the tool's JSON schema.
• Checks the caller's permissions against the tool's auth requirements.
• Enforces approval gates (some tools require human approval).
• Dispatches with an idempotency key and a deadline.
• Records an audit event (intent + outcome).

#[async_trait::async_trait]
pub trait Tool: Send + Sync {
    fn name(&self) -> &str;
    fn schema(&self) -> &serde_json::Value;
    fn risk_tier(&self) -> RiskTier;
    async fn call(&self, args: &serde_json::Value, ctx: &CallCtx)
        -> Result<ToolResult, ToolError>;
}

pub struct ToolRouter {
    tools: std::collections::HashMap<String, Box<dyn Tool>>,
    auth: AuthService,
    audit: AuditLog,
    approval: ApprovalQueue,
}

impl ToolRouter {
    pub async fn dispatch(&self, name: &str, args: serde_json::Value, ctx: CallCtx)
        -> Result<ToolResult, RouterError>
    {
        let tool = self.tools.get(name).ok_or(RouterError::UnknownTool)?;
        self.auth.authorize(&ctx.user, name).await?;
        validate_against(&args, tool.schema())?;
        if tool.risk_tier() == RiskTier::High {
            self.approval.wait_for_approval(&ctx.run_id, name, &args).await?;
        }
        self.audit.record_intent(&ctx, name, &args).await?;
        let result = tool.call(&args, &ctx).await;
        self.audit.record_outcome(&ctx, name, &result).await?;
        Ok(result?)
    }
}

Plan executor with persistence

The plan executor is the loop in chapter 05 turned into a real Rust service. Key features:

• State stored in Postgres with optimistic concurrency control (a version column).
• Events appended to a separate audit log (Kafka or Postgres append-only table).
• One executor instance owns one run at a time (Postgres advisory lock).
• Resumable — on startup or after crash, executor scans for runs whose lock has expired and resumes them.
• Bounded — every run has a deadline, a step limit, and a cost budget. Exceeding any halts the run.
• Cancellable — a "cancel" signal published to a topic; the executor checks before each step.

states: pending -> running -> awaiting_approval -> running -> complete | failed | cancelled

transitions:
  pending     +start          -> running
  running     +tool_call      -> running (after dispatch)
  running     +needs_approval -> awaiting_approval
  awaiting_approval +approved -> running
  awaiting_approval +denied   -> failed
  running     +final_answer   -> complete
  running     +error          -> failed (after exhausted retries)
  *           +cancel_signal  -> cancelled

Idempotent retries

The pattern combining everything above:

1. Orchestrator generates a UUID per intended tool call. This is the idempotency key.
2. Tool router persists (idem_key) -> in_progress before invoking.
3. On success, writes (idem_key) -> result. TTL = retention window.
4. On retry with the same key, returns the recorded result without re-invoking.
5. On crash mid-call: the next attempt sees in_progress; either waits (cooperative recovery) or treats it as failed and retries with a new key after a grace period.

Failure mode to discuss in interviews: the "doubly executed but never recorded" case. Happens when the server completes the side effect but crashes before persisting the result. The mitigation is making the underlying side effect idempotent end-to-end — e.g., the downstream API takes the same idem key.

Deterministic replay for debugging

When a regulator or an internal investigator asks "why did the agent do X at 2:14pm on Tuesday?", you need replay. The architecture:

• Every step is an event in an append-only log — model call (prompt + model version + sampling config + response), tool call (name + args + idem key + result + version), approval (who, when, decision).
• External-data snapshots — what the retrieval index returned at that moment is recorded, not re-fetched.
• Replay tool — load the events, walk them in order, show the prompts, responses, and decisions in a UI.
• Counterfactual replay — optionally re-run with a different prompt or model version and diff the trajectory. Useful for "did the new prompt version regress on case X?"

Things to prototype before the loop

If you have a weekend, build one of these. The interview conversation gets categorically stronger when you can say "I built one of these last weekend."

1. A toy inference gateway in axum that fronts an OpenAI-compatible endpoint, adds a tower timeout layer, a per-key rate limit, and a structured-log layer with request IDs.
2. An async LRU cache with a request-coalescing front (chapter 11's "dedup/coalescing" problem).
3. A tonic streaming server that returns 100 fake tokens with a small sleep between them, with a client that cancels mid-stream.
4. A bounded worker pool with backpressure — 10 workers, channel of 100, producer that benchmarks throughput against varying worker count.
5. A retry-with-backoff combinator (chapter 11's problem) with jitter and a retry budget.

None take more than 4 hours each. All directly map to interview questions you'll get.