Chapter 00 · ~15 minutes

Prereqs & Mock Upstream

Lay down the workspace and stand up a deterministic SSE-emitting fake vLLM so the rest of the guide can develop offline.

Install checklist

Run each, confirm output, then move on. We'll add more later as needed.

rustc --version    # rustc 1.75.0 or newer
cargo --version    # cargo 1.75.0 or newer
docker --version   # Docker version 24.x or newer
curl --version     # any reasonably modern curl
jq --version       # 1.6 or newer
Missing rustup?

Install from rustup.rs — one command, no admin needed. Then rustup default stable.

Project layout

We'll use a Cargo workspace so the gateway and the mock live side-by-side and can share a target directory. Pick a folder, then:

mkdir -p infergw-workspace && cd infergw-workspace
mkdir -p mock-upstream/src infergw/src
touch Cargo.toml mock-upstream/Cargo.toml infergw/Cargo.toml

Put this in the workspace root Cargo.toml:

[workspace]
resolver = "2"
members  = ["infergw", "mock-upstream"]

[workspace.package]
edition = "2021"
rust-version = "1.75"

[profile.release]
lto = "thin"
codegen-units = 1
strip = true

The release profile flags are worth scanning now and re-reading in Chapter 06 — they're what makes the final binary small and fast.

The mock upstream

vLLM exposes an OpenAI-compatible POST /v1/chat/completions endpoint that returns either a single JSON body or, when "stream": true, a Server-Sent Events stream. Our mock returns the streaming form: a series of chunks, then a final data: [DONE].

Drop this into mock-upstream/Cargo.toml:

[package]
name = "mock-upstream"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true

[dependencies]
axum         = "0.7"
tokio        = { version = "1", features = ["macros", "rt-multi-thread", "time", "signal"] }
tokio-stream = "0.1"
futures      = "0.3"
serde        = { version = "1", features = ["derive"] }
serde_json   = "1"
tracing      = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

And mock-upstream/src/main.rs:

//! Mock vLLM-compatible upstream. Returns a deterministic SSE stream so the
//! gateway can be developed and tested without a GPU.

use axum::{
    extract::Json,
    response::sse::{Event, Sse},
    routing::{get, post},
    Router,
};
use futures::stream::{self, Stream};
use serde::Deserialize;
use std::{convert::Infallible, time::Duration};
use tokio_stream::StreamExt;

#[derive(Deserialize, Debug)]
struct ChatRequest {
    model: String,
    messages: Vec<serde_json::Value>,
    #[serde(default)]
    stream: bool,
}

async fn chat(
    Json(req): Json<ChatRequest>,
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
    // Deterministic response: echo back token-by-token based on prompt length.
    let prompt_len: usize = req
        .messages
        .iter()
        .filter_map(|m| m.get("content").and_then(|c| c.as_str()))
        .map(str::len)
        .sum();

    let tokens = [
        "Hello", ",", " this", " is", " a", " mocked", " response", " for", " model", " '",
        req.model.as_str(), "'", ".", " You", " sent", " ", &prompt_len.to_string(), " chars", ".",
    ];

    let chunks: Vec<String> = tokens
        .iter()
        .map(|t| {
            serde_json::json!({
                "choices": [{
                    "delta": { "content": t },
                    "index": 0,
                    "finish_reason": serde_json::Value::Null,
                }]
            })
            .to_string()
        })
        .collect();

    // Append the OpenAI sentinel.
    let mut frames: Vec<String> = chunks;
    frames.push("[DONE]".to_string());

    let stream = stream::iter(frames)
        .throttle(Duration::from_millis(40))
        .map(|payload| Ok(Event::default().data(payload)));

    Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
}

async fn healthz() -> &'static str { "ok" }

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_env_filter(
            tracing_subscriber::EnvFilter::try_from_default_env()
                .unwrap_or_else(|_| "mock_upstream=info".into()),
        )
        .init();

    let app = Router::new()
        .route("/v1/chat/completions", post(chat))
        .route("/healthz", get(healthz));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:9000").await.unwrap();
    tracing::info!("mock-upstream listening on http://127.0.0.1:9000");
    axum::serve(listener, app).await.unwrap();
}
Three details worth noticing

1. tokio_stream::StreamExt::throttle spaces the chunks 40ms apart. Without this the whole response would arrive in one TCP packet and you wouldn't see streaming behavior — which is the whole point of testing this end-to-end.

2. The [DONE] sentinel is part of the OpenAI SSE convention, not an HTTP thing. Your gateway will need to look for it and close the stream.

3. Each frame is Event::default().data(payload), which wraps the JSON in data: <payload>\n\n. That double-newline is the SSE record separator — clients buffer until they see it.

Run it and curl it

cargo run -p mock-upstream

First build will take 30–90 seconds; subsequent builds are fast. You should see:

INFO mock_upstream: mock-upstream listening on http://127.0.0.1:9000

From a second terminal:

curl -N -H 'content-type: application/json' \
  -d '{"model":"llama-3-8b","messages":[{"role":"user","content":"Hi there"}],"stream":true}' \
  http://127.0.0.1:9000/v1/chat/completions

The -N flag is important — it disables curl's output buffering so you actually see the chunks land one at a time. Expected output (with about 40ms between each data: line):

data: {"choices":[{"delta":{"content":"Hello"},"index":0,"finish_reason":null}]}

data: {"choices":[{"delta":{"content":","},"index":0,"finish_reason":null}]}

data: {"choices":[{"delta":{"content":" this"},"index":0,"finish_reason":null}]}
...
data: [DONE]
Not seeing chunks land one at a time?

Without -N, curl waits for the full response before printing anything — your terminal will look frozen for ~800ms then dump everything. That looks like the stream is broken when it isn't. Always use -N when testing SSE.

Why SSE looks like this

Server-Sent Events is an old, plain-text protocol that predates WebSockets and is much simpler. Each event is:

data: <payload>\n
\n

The blank line is the record separator. Clients keep reading until the connection closes or they decide they're done. There's no schema — the data: payload can be anything, and OpenAI happened to pick stringified JSON with [DONE] as the sentinel.

This matters because the gateway can't just forward bytes verbatim if we want to do useful work like counting tokens or splicing frames. We'll need to parse the SSE framing on the way through. Chapter 02 handles that.

If you'd rather point at a real upstream

If you already have a vLLM, Ollama, or OpenAI-compatible endpoint reachable from your laptop, you don't need the mock. Just remember the URL you'll use — somewhere like http://localhost:8000/v1/chat/completions for vLLM, or https://api.openai.com/v1/chat/completions with an Authorization: Bearer ... header for OpenAI. Every subsequent chapter notes where to swap the URL.

Why use the mock anyway?

Deterministic output, no API costs, no GPU, no network flakiness — easier to reason about correctness when the upstream is boring. Plus, in Chapter 04, we'll deliberately make the mock fail in interesting ways to exercise the circuit breaker. Try that against a real provider and they'll rate-limit you.

Checkpoint

You should now have:

  • A Cargo workspace at infergw-workspace/ with mock-upstream compiling.
  • The mock server running on 127.0.0.1:9000.
  • A curl that streams 19 chunks then [DONE].

Leave the mock running — Chapter 01 needs it on the other end of its (nonexistent yet) proxy.

→ Chapter 01: Skeleton — axum, tracing, graceful shutdown