Build an Inference Gateway in Rust
From cargo new to a production-shaped streaming gateway in front of vLLM — with batching, circuit breakers, request hedging, OpenTelemetry tracing, and a multi-stage Dockerfile. Three hours, ten chapters, one working repo at the end.
What we're building
An inference gateway is the thin, fast, reliability-shaped layer between your application clients and a fleet of LLM workers (vLLM, TGI, Triton, or anything OpenAI-compatible). It's where you put the cross-cutting concerns that don't belong inside the model server: tenant routing, retries, hedging, batching, observability, auth, rate limits, the works.
By the end of this guide you'll have a Rust binary called infergw that:
- Speaks HTTP and OpenAI-compatible
/v1/chat/completionson the front, with Server-Sent Events streaming. - Proxies to one or more upstream vLLM-compatible endpoints (we'll use a local mock so no GPU is required).
- Coalesces identical in-flight prompts so duplicate requests share a single upstream call.
- Wraps the upstream call in a circuit breaker, retry budget, and request hedging.
- Exposes Prometheus metrics on
/metricsand emits OTLP traces. - Ships as a ~15 MB distroless container with a healthcheck and a Kubernetes Deployment example.
An inference gateway is mostly I/O — you sit between two HTTP endpoints, push bytes, and update counters. Rust gives you predictable tail latency under load (no GC pauses), a precise async story via tokio, and a binary you can ship without runtime dependencies. The cost is a steeper learning curve, but the streaming-proxy shape is one of Rust's strongest use cases.
Final architecture
Here's what you'll have running on your laptop by Chapter 6:
┌─────────────────────────────────────────────┐
client (curl, │ infergw (this guide) │
app, agent loop) │ │
│ │ ┌──────────┐ ┌─────────────────────┐ │
│ POST /v1/ │ │ axum │ │ proxy service │ │
├──chat/c. ───►├──►│ router ├───►│ • coalesce │ │ ┌──────────────┐
│ (SSE) │ │ │ │ • circuit breaker │ ├──────►│ mock vLLM │
◄──── tokens ──┤ │ │ │ • retry + hedge │ │ HTTP │ (or real) │
│ └──────────┘ └─────────────────────┘ │ └──────────────┘
│ │ │ │
│ ▼ ▼ │
│ /healthz /metrics ─── prom │
│ /readyz │ │
│ ▼ │
│ OTLP traces → jaeger │
└─────────────────────────────────────────────┘
Every box is a thing you'll write or wire up yourself. The only "magic" is what's inside tokio and axum, and we'll poke at those too.
Prereqs
- Rust 1.75+ via
rustup. Verify withrustc --version. - Docker 24+ and
docker compose. Needed for Chapter 6 and the optional Jaeger/Prometheus sidecars. - curl and jq for poking at endpoints.
- oha or vegeta for Chapter 7 load tests (you'll install it when you get there).
- Comfortable with basic Rust:
Result,async/await,Arc. You don't need to knowtower,hyper, oraxum— we'll build the mental model as we go. - No GPU required. We use a tiny local mock upstream that returns canned SSE chunks. If you have a real vLLM endpoint you want to point at, every chapter notes how.
Time estimate
About three hours of focused work. Each chapter ends in a runnable checkpoint, so it's safe to stop mid-guide and come back.
| Chapter | Time | Checkpoint |
|---|---|---|
| 00 — Prereqs & mock upstream | ~15 min | curl the mock and see SSE chunks stream |
| 01 — Skeleton: axum + tracing + shutdown | ~20 min | /healthz returns OK; Ctrl-C drains cleanly |
| 02 — Proxy & streaming | ~30 min | tokens stream from mock through the gateway |
| 03 — Batching & coalescing | ~25 min | two concurrent identical requests hit upstream once |
| 04 — Circuit breaker & hedging | ~30 min | upstream flaps; gateway sheds and recovers |
| 05 — Observability | ~25 min | Prometheus scrapes; traces appear in Jaeger |
| 06 — Docker & deploy | ~20 min | ~15 MB distroless image runs healthy |
| 07 — Load test & tuning | ~15 min | p99 numbers in hand; you've tuned worker threads |
| 08 — Where to next | ~5 min | a stretch idea you want to try |
Chapter index
Install checklist, set up the project layout, and build a 30-line mock vLLM that returns deterministic SSE chunks so you can develop without a GPU.
CHAPTER 01 Skeleton: axum + tracing + shutdowncargo new, wire up axum and tokio, add structured tracing, and implement graceful shutdown that drains in-flight requests before exiting.
Forward chat completion requests to the upstream and stream the SSE response back to the client chunk-by-chunk using reqwest and Stream.
Add in-flight request coalescing for identical prompts using a singleflight pattern. Measure the speedup on duplicate concurrent requests.
CHAPTER 04 Circuit breaker & hedgingWrap the upstream call in a retry budget, a simple circuit breaker, and request hedging that fires a backup call after p95 latency.
CHAPTER 05 ObservabilityPrometheus metrics on /metrics (request counts, latency histograms, queue depth, upstream errors) and OTLP traces wired into Jaeger via Docker Compose.
Multi-stage Dockerfile (builder → distroless), release profile tuning, healthcheck wiring, env-var config, and a sample Kubernetes Deployment.
CHAPTER 07 Load test & tuningDrive the gateway with oha, read your own Prometheus metrics under load, and tune tokio worker threads against tail latency.
Stretch ideas — auth, multi-tenancy, KV-cache aware routing, model fallback, structured output validation — and a reading list to go deeper.
After you finish
You'll have a real, runnable Rust gateway you can point a coding interviewer or a hiring manager at. More usefully, you'll have built the things that ML serving infrastructure engineers talk about — coalescing, hedging, circuit breaking, multi-stage builds — instead of having only read about them. The vocabulary will feel different on the way out.
If you only have an hour, do 00→02 and you'll have a working streaming proxy. If you only care about reliability patterns, skim 01 and go straight to 03/04. Each chapter's checkpoint is a clean stopping point.