Chapter 05 · ~25 minutes

Observability

Prometheus counters, latency histograms, OTLP traces — wired into a local Grafana + Jaeger stack so you can actually see the numbers move.

What to measure

The minimum useful set for an inference gateway:

  • infergw_requests_total{route, status} — request counter, partitioned by route and HTTP status class.
  • infergw_request_duration_seconds{route} — histogram, so you can derive p50/p95/p99.
  • infergw_upstream_errors_total{reason} — every time the upstream call fails, partition by reason (connect, timeout, status, chunk).
  • infergw_coalesce_followers_total — how often coalescing actually saved a call.
  • infergw_breaker_state — gauge, 0=closed, 1=half-open, 2=open. So you can alert on prolonged open state.
  • infergw_hedges_fired_total — how often the backup request actually fired.
  • infergw_inflight — gauge of currently-active requests, the closest thing to queue depth.
Why these and not more

Every metric costs cardinality, scrape time, and brainspace at 3 AM. Start with the seven above and add one only when you can name the specific question it answers. The metric you'll wish you had during an incident is almost always one of these — request counts by status, latency distribution, upstream error class, breaker state.

Prometheus metrics

The Rust ecosystem has two main choices: the prometheus crate (battle-tested, slightly verbose) and the metrics + metrics-exporter-prometheus facade (more ergonomic, slightly newer). We'll use the latter.

Add to infergw/Cargo.toml:

metrics                     = "0.23"
metrics-exporter-prometheus = "0.15"

Create infergw/src/metrics.rs:

use metrics_exporter_prometheus::{PrometheusBuilder, PrometheusHandle};

pub fn install() -> anyhow::Result<PrometheusHandle> {
    let handle = PrometheusBuilder::new()
        .set_buckets_for_metric(
            metrics_exporter_prometheus::Matcher::Prefix("infergw_request_duration".into()),
            &[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0],
        )?
        .install_recorder()?;

    // Pre-declare gauges so they show up at zero before the first event.
    metrics::describe_counter!("infergw_requests_total",         "request counter");
    metrics::describe_histogram!("infergw_request_duration_seconds", "request duration");
    metrics::describe_counter!("infergw_upstream_errors_total",  "upstream errors");
    metrics::describe_counter!("infergw_coalesce_followers_total","coalesced followers");
    metrics::describe_gauge!("infergw_breaker_state",            "0=closed 1=half 2=open");
    metrics::describe_counter!("infergw_hedges_fired_total",     "hedge backups fired");
    metrics::describe_gauge!("infergw_inflight",                 "in-flight requests");

    Ok(handle)
}

Custom histogram buckets matter: the default Prometheus buckets are tuned for HTTP at human timescales (0.005 → 10 seconds). LLM streaming responses easily run 5–30 seconds, so the bucket at 10 seconds is your "tail" boundary. Add 30s on the high end and you'll catch the truly slow ones.

Wire a /metrics route. Update infergw/src/routes.rs to take a handle:

use metrics_exporter_prometheus::PrometheusHandle;

#[derive(Clone)]
pub struct AppState {
    // ... existing fields
    pub metrics_handle: PrometheusHandle,
}

impl AppState {
    pub fn new(upstream: Upstream, hedge_after_ms: u64, metrics_handle: PrometheusHandle) -> Self {
        Self {
            // ... existing init
            metrics_handle,
        }
    }
}

pub fn router(state: AppState) -> Router {
    Router::new()
        .route("/healthz", get(healthz))
        .route("/readyz", get(readyz))
        .route("/metrics", get(metrics_endpoint))
        .route("/v1/chat/completions", post(crate::proxy::chat_completions))
        .with_state(state)
}

async fn metrics_endpoint(
    axum::extract::State(state): axum::extract::State<AppState>,
) -> impl IntoResponse {
    state.metrics_handle.render()
}

And wire in main.rs:

mod metrics;
// ...
let handle = metrics::install()?;
let state = routes::AppState::new(upstream, cfg.hedge_after_ms, handle);

Instrument the proxy

Sprinkle the events at the right spots. In proxy.rs:

use std::time::Instant;

pub async fn chat_completions(
    State(state): State<AppState>,
    Json(body): Json<Value>,
) -> Result<Response, ProxyError> {
    let started = Instant::now();
    metrics::gauge!("infergw_inflight").increment(1.0);

    let result = chat_completions_inner(state.clone(), body).await;

    let elapsed = started.elapsed().as_secs_f64();
    metrics::histogram!("infergw_request_duration_seconds",
        "route" => "chat_completions").record(elapsed);
    let status = if result.is_ok() { "2xx" } else { "5xx" };
    metrics::counter!("infergw_requests_total",
        "route" => "chat_completions",
        "status" => status).increment(1);
    metrics::gauge!("infergw_inflight").decrement(1.0);

    // Update breaker state gauge.
    let bs = match state.breaker.state() {
        crate::breaker::State::Closed   => 0.0,
        crate::breaker::State::HalfOpen => 1.0,
        crate::breaker::State::Open     => 2.0,
    };
    metrics::gauge!("infergw_breaker_state").set(bs);

    result
}

async fn chat_completions_inner(state: AppState, body: Value)
    -> Result<Response, ProxyError> {
    // ... existing logic from Chapter 03/04, with a few targeted counter bumps
    // (left as edits to the existing function rather than a full rewrite).
}

In the leader's failure path, increment by reason:

// inside spawn_leader, on Err branches:
metrics::counter!("infergw_upstream_errors_total",
    "reason" => "send_failed").increment(1);
// or "status_non_2xx", "chunk_error", "breaker_open", "hedge_failed"

In the coalescer's follower branch:

JoinResult::Follower(inflight) => {
    metrics::counter!("infergw_coalesce_followers_total").increment(1);
    // ... existing follower setup
}

In the hedge module, increment when the backup actually fires:

if winner_tx.lock().await.is_some() {
    metrics::counter!("infergw_hedges_fired_total").increment(1);
    tracing::info!("hedge: firing backup request");
    // ...
}

Now verify it works:

cargo run -p infergw
# In another terminal:
curl -s http://127.0.0.1:8080/metrics | grep -E '^infergw_' | head -30

You should see all your metric families listed with HELP / TYPE comments and current values. Fire a few requests and re-scrape — the counters go up.

OpenTelemetry traces

Metrics tell you what happened in aggregate. Traces tell you why a particular request was slow — by stitching together the time spent in each stage. We'll emit OTLP over gRPC to a Jaeger-compatible collector.

Add to infergw/Cargo.toml:

opentelemetry            = { version = "0.24", features = ["trace"] }
opentelemetry_sdk        = { version = "0.24", features = ["rt-tokio"] }
opentelemetry-otlp       = { version = "0.17", features = ["grpc-tonic", "trace"] }
tracing-opentelemetry    = "0.25"

Update infergw/src/telemetry.rs:

use opentelemetry::trace::TracerProvider;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{trace as sdktrace, Resource};
use tracing_subscriber::{fmt, prelude::*, EnvFilter};

pub fn init_tracing() -> anyhow::Result<()> {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("infergw=info,tower_http=info"));

    let json_mode = std::env::var("INFERGW_LOG_JSON")
        .ok().map(|v| v == "1").unwrap_or(false);

    let otlp_endpoint = std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").ok();
    let registry = tracing_subscriber::registry().with(filter);

    if let Some(endpoint) = otlp_endpoint {
        let exporter = opentelemetry_otlp::new_exporter()
            .tonic()
            .with_endpoint(endpoint);
        let provider = opentelemetry_otlp::new_pipeline()
            .tracing()
            .with_exporter(exporter)
            .with_trace_config(
                sdktrace::Config::default().with_resource(Resource::new(vec![
                    opentelemetry::KeyValue::new("service.name", "infergw"),
                ])),
            )
            .install_batch(opentelemetry_sdk::runtime::Tokio)?;
        let tracer = provider.tracer("infergw");
        let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);
        if json_mode {
            registry.with(otel_layer).with(fmt::layer().json()).init();
        } else {
            registry.with(otel_layer).with(fmt::layer().compact()).init();
        }
    } else if json_mode {
        registry.with(fmt::layer().json()).init();
    } else {
        registry.with(fmt::layer().compact()).init();
    }
    Ok(())
}

pub fn shutdown() {
    opentelemetry::global::shutdown_tracer_provider();
}

And call telemetry::shutdown() at the end of main so the OTLP exporter flushes pending spans before exit.

Now decorate the handler with a span. In proxy.rs:

use tracing::Instrument;

pub async fn chat_completions(
    State(state): State<AppState>,
    Json(body): Json<Value>,
) -> Result<Response, ProxyError> {
    let model = body.get("model").and_then(|v| v.as_str()).unwrap_or("unknown").to_string();
    let span = tracing::info_span!("chat_completions", model = %model);
    chat_completions_inner_instrumented(state, body).instrument(span).await
}

Inside the function, additional spans for upstream calls and coalescer joins:

let _span_guard = tracing::info_span!("upstream_call", url = %url).entered();

Span guards work great in synchronous sections; in async, prefer .instrument(span). The pattern's worth getting right — wrong instrumentation in async produces traces that are technically valid but useless.

Local stack with Docker Compose

Save this as infergw-workspace/observability/docker-compose.yml:

services:
  jaeger:
    image: jaegertracing/all-in-one:1.57
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
  prometheus:
    image: prom/prometheus:v2.52.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
  grafana:
    image: grafana/grafana:10.4.3
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - "3000:3000"

And infergw-workspace/observability/prometheus.yml:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: infergw
    static_configs:
      - targets: ['host.docker.internal:8080']  # macOS/Win
        # on Linux: use the host's LAN IP, or run infergw in a container too

Bring it up:

cd observability
docker compose up -d

Then run the gateway with OTLP enabled:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 \
  cargo run -p infergw

Look at it

Fire some traffic:

for i in $(seq 1 20); do
  curl -sN -o /dev/null \
    -H 'content-type: application/json' \
    -d "{\"model\":\"llama-3-8b\",\"messages\":[{\"role\":\"user\",\"content\":\"req $i\"}],\"stream\":true}" \
    http://127.0.0.1:8080/v1/chat/completions
done

Now visit:

  • Prometheus: http://localhost:9090 → try rate(infergw_requests_total[1m]) or histogram_quantile(0.95, sum by (le) (rate(infergw_request_duration_seconds_bucket[1m]))).
  • Jaeger: http://localhost:16686 → pick service "infergw", click "Find Traces". Each request appears as a span with nested upstream_call children. Click in for the timeline.
  • Grafana: http://localhost:3000 → add Prometheus (http://prometheus:9090) as a data source, then build a quick dashboard from the queries above.
A starter PromQL set

Memorize these four queries and you'll cover 80% of incident response:

  • sum(rate(infergw_requests_total[1m])) by (status) — RPS by status class
  • histogram_quantile(0.99, sum by (le) (rate(infergw_request_duration_seconds_bucket[5m]))) — p99 latency
  • sum(rate(infergw_upstream_errors_total[1m])) by (reason) — what's failing upstream
  • max_over_time(infergw_breaker_state[5m]) — has the breaker been open at all recently?

Checkpoint

  • GET /metrics returns Prometheus-format text, all seven metric families populated.
  • Prometheus scrapes the gateway every 5s and renders charts.
  • Jaeger receives traces and shows per-request span trees with upstream timing.
  • Counters move when you fire requests; the breaker gauge flips from 0 → 2 → 0 if you exercise it.

You can now actually see what your gateway is doing in real time. That's the productivity unlock that makes the next chapter — making it deployable — worthwhile.

→ Chapter 06: Multi-stage Dockerfile and Kubernetes Deployment