Chapter 01 · ~20 minutes

Skeleton: axum, tracing, graceful shutdown

Stand up the gateway binary with a health endpoint, structured logs, and shutdown semantics that drain in-flight requests instead of dropping them on the floor.

Dependencies

Drop this into infergw/Cargo.toml:

[package]
name = "infergw"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true

[dependencies]
axum               = "0.7"
tokio              = { version = "1", features = ["macros", "rt-multi-thread", "signal", "time", "sync"] }
tower              = { version = "0.4", features = ["timeout", "limit"] }
tower-http         = { version = "0.5", features = ["trace", "timeout"] }
hyper              = { version = "1", features = ["full"] }
tracing            = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
serde              = { version = "1", features = ["derive"] }
serde_json         = "1"
anyhow             = "1"
thiserror          = "1"

That's the floor — we'll add reqwest, futures, metrics, opentelemetry, and friends in later chapters.

A small Config struct

Read everything from env vars at startup. Twelve-factor by default. Create infergw/src/config.rs:

use std::env;

#[derive(Debug, Clone)]
pub struct Config {
    pub bind_addr: String,
    pub upstream_url: String,
    pub upstream_timeout_ms: u64,
    pub shutdown_grace_ms: u64,
}

impl Config {
    pub fn from_env() -> anyhow::Result<Self> {
        Ok(Self {
            bind_addr: env::var("INFERGW_BIND").unwrap_or_else(|_| "0.0.0.0:8080".into()),
            upstream_url: env::var("INFERGW_UPSTREAM")
                .unwrap_or_else(|_| "http://127.0.0.1:9000".into()),
            upstream_timeout_ms: env::var("INFERGW_UPSTREAM_TIMEOUT_MS")
                .ok()
                .and_then(|s| s.parse().ok())
                .unwrap_or(30_000),
            shutdown_grace_ms: env::var("INFERGW_SHUTDOWN_GRACE_MS")
                .ok()
                .and_then(|s| s.parse().ok())
                .unwrap_or(15_000),
        })
    }
}

Defaults match the mock from Chapter 00. Override at runtime when we Docker-ize in Chapter 06.

Structured tracing

One of the points of choosing Rust here is that tracing gives you compile-time-checked structured logging that's also the foundation for OpenTelemetry spans in Chapter 05. Set it up once and benefit forever.

Create infergw/src/telemetry.rs:

use tracing_subscriber::{fmt, prelude::*, EnvFilter};

pub fn init_tracing() {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("infergw=info,tower_http=info"));

    // JSON when INFERGW_LOG_JSON=1, pretty otherwise.
    let json_mode = std::env::var("INFERGW_LOG_JSON")
        .ok()
        .map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
        .unwrap_or(false);

    let registry = tracing_subscriber::registry().with(filter);

    if json_mode {
        registry.with(fmt::layer().json().with_current_span(true)).init();
    } else {
        registry.with(fmt::layer().with_target(true).compact()).init();
    }
}
Why two modes

Pretty output for local cargo run. JSON when INFERGW_LOG_JSON=1 for production where logs get scraped by Loki / Datadog / CloudWatch. One toggle, no fork in the code path.

Router and health endpoints

Two endpoints to start: /healthz (liveness — am I running?) and /readyz (readiness — should traffic come to me?). The K8s difference is real: failing readiness drains traffic, failing liveness restarts the pod.

Create infergw/src/routes.rs:

use axum::{http::StatusCode, response::IntoResponse, routing::get, Router};
use std::sync::{
    atomic::{AtomicBool, Ordering},
    Arc,
};

#[derive(Clone)]
pub struct AppState {
    pub ready: Arc<AtomicBool>,
}

impl AppState {
    pub fn new() -> Self {
        Self {
            ready: Arc::new(AtomicBool::new(true)),
        }
    }
}

pub fn router(state: AppState) -> Router {
    Router::new()
        .route("/healthz", get(healthz))
        .route("/readyz", get(readyz))
        .with_state(state)
}

async fn healthz() -> impl IntoResponse {
    (StatusCode::OK, "ok")
}

async fn readyz(
    axum::extract::State(state): axum::extract::State<AppState>,
) -> impl IntoResponse {
    if state.ready.load(Ordering::Relaxed) {
        (StatusCode::OK, "ready")
    } else {
        (StatusCode::SERVICE_UNAVAILABLE, "draining")
    }
}

Graceful shutdown

This is the bit most tutorials skip. When K8s sends SIGTERM, you want to: flip readyz to SERVICE_UNAVAILABLE (so the load balancer stops sending new requests), wait for in-flight requests to finish, then exit. If you can't drain in 15 seconds, give up and exit anyway.

Create infergw/src/shutdown.rs:

use crate::routes::AppState;
use std::{sync::atomic::Ordering, time::Duration};
use tokio::signal;

/// Future that completes on SIGINT or SIGTERM, flips readiness to false,
/// then yields so axum can finish in-flight requests up to the grace period.
pub async fn shutdown_signal(state: AppState, grace: Duration) {
    let ctrl_c = async {
        signal::ctrl_c()
            .await
            .expect("failed to install Ctrl-C handler");
    };

    #[cfg(unix)]
    let terminate = async {
        signal::unix::signal(signal::unix::SignalKind::terminate())
            .expect("failed to install SIGTERM handler")
            .recv()
            .await;
    };

    #[cfg(not(unix))]
    let terminate = std::future::pending::<()>();

    tokio::select! {
        _ = ctrl_c    => tracing::info!("received Ctrl-C, draining..."),
        _ = terminate => tracing::info!("received SIGTERM, draining..."),
    }

    // Stop accepting new requests at the LB layer.
    state.ready.store(false, Ordering::Relaxed);

    // Give in-flight requests a chance to complete.
    tokio::time::sleep(grace).await;
    tracing::info!("grace period elapsed, exiting");
}
Why we sleep instead of tracking in-flight

You could wire a per-request counter that this function awaits going to zero — and in Chapter 04 we'll do something like that for the connection pool. But for the request loop, axum's with_graceful_shutdown already stops accepting new connections and waits for the existing ones to finish. The sleep is the upper bound: even if some request is misbehaving, we exit eventually.

Putting it together

Now the entry point. infergw/src/main.rs:

mod config;
mod routes;
mod shutdown;
mod telemetry;

use std::time::Duration;
use tower_http::trace::TraceLayer;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    telemetry::init_tracing();
    let cfg = config::Config::from_env()?;
    tracing::info!(?cfg, "starting infergw");

    let state = routes::AppState::new();
    let app = routes::router(state.clone()).layer(TraceLayer::new_for_http());

    let listener = tokio::net::TcpListener::bind(&cfg.bind_addr).await?;
    tracing::info!(addr = %cfg.bind_addr, "listening");

    let grace = Duration::from_millis(cfg.shutdown_grace_ms);
    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown::shutdown_signal(state, grace))
        .await?;

    tracing::info!("infergw shut down cleanly");
    Ok(())
}

That's the whole skeleton. ~150 lines of code, every line earning its keep.

Run it

From the workspace root:

cargo run -p infergw

You should see:

INFO infergw: starting infergw cfg=Config { bind_addr: "0.0.0.0:8080", ... }
INFO infergw: listening addr=0.0.0.0:8080

From another terminal:

curl -i http://127.0.0.1:8080/healthz
# HTTP/1.1 200 OK
# ...
# ok

curl -i http://127.0.0.1:8080/readyz
# HTTP/1.1 200 OK
# ...
# ready

Now press Ctrl-C in the server terminal. Watch what happens:

^C
INFO infergw::shutdown: received Ctrl-C, draining...
INFO infergw::shutdown: grace period elapsed, exiting
INFO infergw: infergw shut down cleanly

During the grace window, hit /readyz again from a third terminal — you'll get 503 Service Unavailable: draining. That's the load balancer's cue to stop routing to this pod.

Try this: prove the drain actually works

Add a deliberately slow route that sleeps 10 seconds before responding. Send a request, then immediately Ctrl-C the server. Watch the slow request still complete before the process exits.

// In routes.rs, add to the router:
.route("/slow", get(slow))

async fn slow() -> impl IntoResponse {
    tokio::time::sleep(std::time::Duration::from_secs(10)).await;
    "done"
}

Run the server, curl http://127.0.0.1:8080/slow &, immediately Ctrl-C the server. The curl will complete with "done" after 10 seconds — even though SIGINT arrived right away.

Checkpoint

  • cargo run -p infergw binds to :8080.
  • GET /healthz returns 200 ok.
  • GET /readyz returns 200 ready, switches to 503 draining on shutdown.
  • Ctrl-C drains in-flight requests up to the grace period then exits cleanly.
  • Logs are structured and respect RUST_LOG.

Keep both processes running into the next chapter — we're about to make the gateway actually do something.

→ Chapter 02: Proxy & streaming