Where to Next
You have a working gateway. Here are the directions that turn it into something you'd actually put in production — pick whichever interests you most.
What you built
Three hours ago you had an empty directory. Now you have:
- A Rust HTTP gateway that proxies OpenAI-compatible chat-completion requests with SSE streaming.
- In-flight request coalescing that turns N concurrent identical requests into one upstream call.
- A circuit breaker, retry budget primitives, and request hedging.
- Prometheus metrics, OpenTelemetry traces, structured JSON logs.
- A multi-stage Docker build that produces a ~35 MB distroless image.
- A Kubernetes Deployment manifest with proper probes and graceful shutdown.
- A load-tested baseline of the gateway's behavior under concurrency.
That's a real piece of infrastructure. Not a toy.
Stretch ideas
Each of these is its own evening's work. Pick by interest, not order.
1 · Auth and per-tenant rate limiting
Add a tower middleware that extracts a tenant from a bearer token (JWT validation with jsonwebtoken) and tags every metric / log / span with tenant_id. Then add a token-bucket rate limiter (tower::limit or governor) keyed by tenant.
The harder part is doing this without making the hot path slow. Validate JWTs against a cached JWK set, not a per-request HTTPS call. Rate-limit before doing any upstream-touching work.
2 · Multi-upstream load balancing with KV-cache awareness
Currently the gateway points at a single upstream. Real deployments have N replicas. Pick a load-balancing policy: round-robin is fine, but for LLMs you can do meaningfully better by routing requests with shared prefixes to the same replica so vLLM's prefix cache hits warm.
Sketch: hash a prefix of the prompt (say, first 200 tokens), use that hash to select a replica from the pool with bounded-load consistent hashing. Per-upstream breakers (the stretch idea from Chapter 04) is the table stakes here.
3 · Model routing and fallback
Accept arbitrary model values from clients. Route to different upstreams based on the model: llama-3-70b goes to your big-model fleet, llama-3-8b goes to your small-model fleet. On a per-model breaker trip, optionally fall back to a different model (gpt-4o → gpt-4o-mini) with a header indicating the substitution happened.
This is where the architecture earns its keep — the routing logic is in the gateway, the model servers stay simple.
4 · Token counting and budget enforcement
Pre-flight: estimate input tokens with a fast tokenizer (tiktoken-rs). Compare against a per-tenant token budget. Reject upfront if it would blow the budget. Post-flight: parse the SSE stream as it passes through and count actual output tokens against the same budget.
The tokenizer is the hard part — keeping it in sync with the model's actual tokenization. The right answer for production is to have the model server return token counts and trust them, but doing it client-side once is illuminating.
5 · Structured-output validation
If clients request a response_format with a JSON schema, validate the streamed output as it's produced — bail and re-prompt the upstream if the partial response is provably invalid (e.g. it's generated {"name": "alice", "name": and you can already see "name" is repeated). This is finicky but it's exactly the kind of work a gateway is uniquely positioned to do.
6 · Replay-able request log
For every request, write {tenant, model, prompt, params, completion, latency, hedged?, coalesced?} to an append-only log (start with a local file, evolve to S3 / Kafka). Two payoffs: (a) when something goes wrong, you can replay exactly what happened; (b) you can mine the log to discover patterns (top prompts, slowest prompts, prompts that triggered hedging) and feed those back into routing / caching decisions.
7 · A proper request cache
Coalescing handles in-flight duplicates. A cache handles recent duplicates. With temperature=0, identical prompts produce identical outputs — cache the completion for ~5 minutes in Redis (or even an in-process moka cache). The hit rate on agent workloads with retries and replays is often 20%+ — a huge cost / latency win. The interactions with billing, abuse, and tenant isolation are the design work.
8 · WebSocket / bidirectional streaming
OpenAI's Realtime API uses WebSockets, not SSE. Adding a /v1/realtime endpoint that proxies WS-to-WS lets your gateway handle voice agents and low-latency interactive use cases. Reach for tokio-tungstenite. The bidirectional flow control story is a chapter on its own — frames in both directions, with the same backpressure properties you got for free with SSE.
Talking about this in an interview
If "have you built ML infrastructure?" comes up, this guide gives you a 90-second answer that's specific and defensible:
"I built an inference gateway in Rust — axum and tokio on the front, reqwest streaming to a vLLM-compatible upstream. A few things became real by doing it: streaming SSE end-to-end requires being deliberate about NOT setting a full-request timeout, otherwise it kills the stream; in-flight request coalescing on identical prompts is the single highest-impact optimization for agent workloads, often 100x+ savings on duplicate work; circuit breakers and hedging interact in ways you have to think about — you never hedge through an open breaker. The observability layer is what made tuning feasible — Prometheus histograms with the right buckets for LLM-shaped latencies, OTLP traces so I could see which span was eating the tail. Shipped as a ~35 MB distroless container with a graceful-shutdown handshake against the K8s readiness probe."
Three things make that answer land: specific numbers (35 MB, 100x), specific tradeoffs (don't hedge through open breakers, full-request timeout kills streams), a coherent shape (front, middle, back, observability, deploy). All three come from having actually built it, not having read about it.
Further reading
- The Tail at Scale — Dean & Barroso, 2013. The paper that put hedging on the map. Short and worth the time.
- Circuit breakers and load shedding — Cloudflare engineering blog. Practical patterns at scale.
- tokio.rs tutorials — the missing manual for async Rust runtime behavior.
- vLLM docs — continuous batching, PagedAttention, prefix caching. The companion PagedAttention deep-dive in this site goes one level below.
- OpenTelemetry Rust — the docs are dense; the examples are the fastest path in.
- K8s graceful shutdown — the official write-up of the SIGTERM / readiness / preStop dance.
Thanks for building this with me. The repo you've got at the end is yours — extend it, deploy it, break it, learn from it. The point was never the gateway itself; it was the muscle memory of "I can build infrastructure like this when the situation calls for it."