Observability

What to measure

The minimum useful set for an inference gateway:

• infergw_requests_total{route, status} — request counter, partitioned by route and HTTP status class.
• infergw_request_duration_seconds{route} — histogram, so you can derive p50/p95/p99.
• infergw_upstream_errors_total{reason} — every time the upstream call fails, partition by reason (connect, timeout, status, chunk).
• infergw_coalesce_followers_total — how often coalescing actually saved a call.
• infergw_breaker_state — gauge, 0=closed, 1=half-open, 2=open. So you can alert on prolonged open state.
• infergw_hedges_fired_total — how often the backup request actually fired.
• infergw_inflight — gauge of currently-active requests, the closest thing to queue depth.

Prometheus metrics

The Rust ecosystem has two main choices: the prometheus crate (battle-tested, slightly verbose) and the metrics + metrics-exporter-prometheus facade (more ergonomic, slightly newer). We'll use the latter.

Add to infergw/Cargo.toml:

metrics                     = "0.23"
metrics-exporter-prometheus = "0.15"

Create infergw/src/metrics.rs:

use metrics_exporter_prometheus::{PrometheusBuilder, PrometheusHandle};

pub fn install() -> anyhow::Result<PrometheusHandle> {
    let handle = PrometheusBuilder::new()
        .set_buckets_for_metric(
            metrics_exporter_prometheus::Matcher::Prefix("infergw_request_duration".into()),
            &[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0],
        )?
        .install_recorder()?;

    // Pre-declare gauges so they show up at zero before the first event.
    metrics::describe_counter!("infergw_requests_total",         "request counter");
    metrics::describe_histogram!("infergw_request_duration_seconds", "request duration");
    metrics::describe_counter!("infergw_upstream_errors_total",  "upstream errors");
    metrics::describe_counter!("infergw_coalesce_followers_total","coalesced followers");
    metrics::describe_gauge!("infergw_breaker_state",            "0=closed 1=half 2=open");
    metrics::describe_counter!("infergw_hedges_fired_total",     "hedge backups fired");
    metrics::describe_gauge!("infergw_inflight",                 "in-flight requests");

    Ok(handle)
}

Custom histogram buckets matter: the default Prometheus buckets are tuned for HTTP at human timescales (0.005 → 10 seconds). LLM streaming responses easily run 5–30 seconds, so the bucket at 10 seconds is your "tail" boundary. Add 30s on the high end and you'll catch the truly slow ones.

Wire a /metrics route. Update infergw/src/routes.rs to take a handle:

use metrics_exporter_prometheus::PrometheusHandle;

#[derive(Clone)]
pub struct AppState {
    // ... existing fields
    pub metrics_handle: PrometheusHandle,
}

impl AppState {
    pub fn new(upstream: Upstream, hedge_after_ms: u64, metrics_handle: PrometheusHandle) -> Self {
        Self {
            // ... existing init
            metrics_handle,
        }
    }
}

pub fn router(state: AppState) -> Router {
    Router::new()
        .route("/healthz", get(healthz))
        .route("/readyz", get(readyz))
        .route("/metrics", get(metrics_endpoint))
        .route("/v1/chat/completions", post(crate::proxy::chat_completions))
        .with_state(state)
}

async fn metrics_endpoint(
    axum::extract::State(state): axum::extract::State<AppState>,
) -> impl IntoResponse {
    state.metrics_handle.render()
}

And wire in main.rs:

mod metrics;
// ...
let handle = metrics::install()?;
let state = routes::AppState::new(upstream, cfg.hedge_after_ms, handle);

Instrument the proxy

Sprinkle the events at the right spots. In proxy.rs:

use std::time::Instant;

pub async fn chat_completions(
    State(state): State<AppState>,
    Json(body): Json<Value>,
) -> Result<Response, ProxyError> {
    let started = Instant::now();
    metrics::gauge!("infergw_inflight").increment(1.0);

    let result = chat_completions_inner(state.clone(), body).await;

    let elapsed = started.elapsed().as_secs_f64();
    metrics::histogram!("infergw_request_duration_seconds",
        "route" => "chat_completions").record(elapsed);
    let status = if result.is_ok() { "2xx" } else { "5xx" };
    metrics::counter!("infergw_requests_total",
        "route" => "chat_completions",
        "status" => status).increment(1);
    metrics::gauge!("infergw_inflight").decrement(1.0);

    // Update breaker state gauge.
    let bs = match state.breaker.state() {
        crate::breaker::State::Closed   => 0.0,
        crate::breaker::State::HalfOpen => 1.0,
        crate::breaker::State::Open     => 2.0,
    };
    metrics::gauge!("infergw_breaker_state").set(bs);

    result
}

async fn chat_completions_inner(state: AppState, body: Value)
    -> Result<Response, ProxyError> {
    // ... existing logic from Chapter 03/04, with a few targeted counter bumps
    // (left as edits to the existing function rather than a full rewrite).
}

In the leader's failure path, increment by reason:

// inside spawn_leader, on Err branches:
metrics::counter!("infergw_upstream_errors_total",
    "reason" => "send_failed").increment(1);
// or "status_non_2xx", "chunk_error", "breaker_open", "hedge_failed"

In the coalescer's follower branch:

JoinResult::Follower(inflight) => {
    metrics::counter!("infergw_coalesce_followers_total").increment(1);
    // ... existing follower setup
}

In the hedge module, increment when the backup actually fires:

if winner_tx.lock().await.is_some() {
    metrics::counter!("infergw_hedges_fired_total").increment(1);
    tracing::info!("hedge: firing backup request");
    // ...
}

Now verify it works:

cargo run -p infergw
# In another terminal:
curl -s http://127.0.0.1:8080/metrics | grep -E '^infergw_' | head -30

You should see all your metric families listed with HELP / TYPE comments and current values. Fire a few requests and re-scrape — the counters go up.

OpenTelemetry traces

Metrics tell you what happened in aggregate. Traces tell you why a particular request was slow — by stitching together the time spent in each stage. We'll emit OTLP over gRPC to a Jaeger-compatible collector.

Add to infergw/Cargo.toml:

opentelemetry            = { version = "0.24", features = ["trace"] }
opentelemetry_sdk        = { version = "0.24", features = ["rt-tokio"] }
opentelemetry-otlp       = { version = "0.17", features = ["grpc-tonic", "trace"] }
tracing-opentelemetry    = "0.25"

Update infergw/src/telemetry.rs:

use opentelemetry::trace::TracerProvider;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{trace as sdktrace, Resource};
use tracing_subscriber::{fmt, prelude::*, EnvFilter};

pub fn init_tracing() -> anyhow::Result<()> {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("infergw=info,tower_http=info"));

    let json_mode = std::env::var("INFERGW_LOG_JSON")
        .ok().map(|v| v == "1").unwrap_or(false);

    let otlp_endpoint = std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").ok();
    let registry = tracing_subscriber::registry().with(filter);

    if let Some(endpoint) = otlp_endpoint {
        let exporter = opentelemetry_otlp::new_exporter()
            .tonic()
            .with_endpoint(endpoint);
        let provider = opentelemetry_otlp::new_pipeline()
            .tracing()
            .with_exporter(exporter)
            .with_trace_config(
                sdktrace::Config::default().with_resource(Resource::new(vec![
                    opentelemetry::KeyValue::new("service.name", "infergw"),
                ])),
            )
            .install_batch(opentelemetry_sdk::runtime::Tokio)?;
        let tracer = provider.tracer("infergw");
        let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);
        if json_mode {
            registry.with(otel_layer).with(fmt::layer().json()).init();
        } else {
            registry.with(otel_layer).with(fmt::layer().compact()).init();
        }
    } else if json_mode {
        registry.with(fmt::layer().json()).init();
    } else {
        registry.with(fmt::layer().compact()).init();
    }
    Ok(())
}

pub fn shutdown() {
    opentelemetry::global::shutdown_tracer_provider();
}

And call telemetry::shutdown() at the end of main so the OTLP exporter flushes pending spans before exit.

Now decorate the handler with a span. In proxy.rs:

use tracing::Instrument;

pub async fn chat_completions(
    State(state): State<AppState>,
    Json(body): Json<Value>,
) -> Result<Response, ProxyError> {
    let model = body.get("model").and_then(|v| v.as_str()).unwrap_or("unknown").to_string();
    let span = tracing::info_span!("chat_completions", model = %model);
    chat_completions_inner_instrumented(state, body).instrument(span).await
}

Inside the function, additional spans for upstream calls and coalescer joins:

let _span_guard = tracing::info_span!("upstream_call", url = %url).entered();

Span guards work great in synchronous sections; in async, prefer .instrument(span). The pattern's worth getting right — wrong instrumentation in async produces traces that are technically valid but useless.

Local stack with Docker Compose

Save this as infergw-workspace/observability/docker-compose.yml:

services:
  jaeger:
    image: jaegertracing/all-in-one:1.57
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
  prometheus:
    image: prom/prometheus:v2.52.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
  grafana:
    image: grafana/grafana:10.4.3
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - "3000:3000"

And infergw-workspace/observability/prometheus.yml:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: infergw
    static_configs:
      - targets: ['host.docker.internal:8080']  # macOS/Win
        # on Linux: use the host's LAN IP, or run infergw in a container too

Bring it up:

cd observability
docker compose up -d

Then run the gateway with OTLP enabled:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 \
  cargo run -p infergw

Look at it

Fire some traffic:

for i in $(seq 1 20); do
  curl -sN -o /dev/null \
    -H 'content-type: application/json' \
    -d "{\"model\":\"llama-3-8b\",\"messages\":[{\"role\":\"user\",\"content\":\"req $i\"}],\"stream\":true}" \
    http://127.0.0.1:8080/v1/chat/completions
done

Now visit:

• Prometheus: http://localhost:9090 → try rate(infergw_requests_total[1m]) or histogram_quantile(0.95, sum by (le) (rate(infergw_request_duration_seconds_bucket[1m]))).
• Jaeger: http://localhost:16686 → pick service "infergw", click "Find Traces". Each request appears as a span with nested upstream_call children. Click in for the timeline.
• Grafana: http://localhost:3000 → add Prometheus (http://prometheus:9090) as a data source, then build a quick dashboard from the queries above.

Checkpoint

• GET /metrics returns Prometheus-format text, all seven metric families populated.
• Prometheus scrapes the gateway every 5s and renders charts.
• Jaeger receives traces and shows per-request span trees with upstream timing.
• Counters move when you fire requests; the breaker gauge flips from 0 → 2 → 0 if you exercise it.

You can now actually see what your gateway is doing in real time. That's the productivity unlock that makes the next chapter — making it deployable — worthwhile.

→ Chapter 06: Multi-stage Dockerfile and Kubernetes Deployment

✓ Check yourself

Answer from memory before opening each one.

1. Send 20 sequential streaming requests, then run curl -s localhost:8080/metrics | grep ^infergw_. What should the request counter and the duration histogram look like, and what does it mean if the histogram shows quantile= lines instead of le= buckets?

   Show answer

   The counter carries route="chat_completions" and status="2xx" with a value of 20. The histogram shows _bucket lines with le from 0.005 to 30 plus +Inf, then _sum and _count. Quantile lines mean the metrics crate fell back to a summary because the bucket matcher did not match the metric name. See Prometheus metrics.

2. Under load, infergw_request_duration_seconds reports a p99 near a few milliseconds while oha measures about 800 ms per stream. Where is the timer measuring, and what else is wrong with the status counter?

   Show answer

   The timer wraps the handler, which returns as soon as the response and its body stream exist. The body is read after that, so the histogram records time to the start of the response, not the length of the stream. The counter has the same gap: a stream that fails later is still counted as 2xx, and the in-flight gauge drops early. Record on stream completion or drop to measure what users feel. See instrument the proxy.

3. Your streams run 25 seconds but the histogram has no bucket above 10 seconds. What does histogram_quantile(0.99, ...) report?

   Show answer

   It returns the upper bound of the highest finite bucket, so 10 seconds, and it cannot tell 11 seconds from 25. Buckets must span the latencies you actually have, which is why the chapter extends them to 30 seconds. See Prometheus metrics.

4. A colleague adds a user_id label to the request counter. There are 50,000 users. What breaks, and where should that detail go instead?

   Show answer

   Every label value creates a new time series, so the counter multiplies by 50,000 and bloats memory in the exporter and in Prometheus, and scrapes slow down. Keep labels to small bounded sets such as route, status class and failure reason. Put per-user detail in logs and trace attributes. See what to measure.

5. Inside an async handler the chapter prefers .instrument(span) to holding a guard from .entered(). Why?

   Show answer

   A held guard marks the span as current on the thread. At an await the task can be suspended and the thread can run other tasks, which then get attributed to your span, while your task may resume on a thread where the span is not entered. instrument enters and exits the span around each poll of the future. See OpenTelemetry traces.

6. Requests succeed and metrics scrape fine, but Jaeger shows no service named infergw. What do you check first?

   Show answer

   Whether OTEL_EXPORTER_OTLP_ENDPOINT is set, because without it the code installs no OpenTelemetry layer and exports nothing, silently. Then check that the value includes the scheme and points at the OTLP gRPC port 4317, and that Jaeger started with OTLP enabled. See local stack with Docker Compose.

Takeaway

• Seven metrics cover traffic, latency, upstream errors, coalescing, hedging, breaker state and load. Add another only when you can name the question it answers.
• Histogram buckets must span your real latencies, because a quantile can never exceed the top finite bucket.
• Every label value multiplies the series count, so labels stay bounded and per-user detail goes in logs and traces.
• A timer around a streaming handler measures time to the start of the response, so record at stream end to measure what users feel.
• Metrics tell you what happened in aggregate and traces tell you why one request was slow.
• In async code attach spans with instrument rather than held guards, and flush the exporter on shutdown or the last spans are lost.

With the gateway observable, it is ready to package. → Docker & deploy

Further reading

• metrics-exporter-prometheus crate documentation — docs.rs, accessed 2026. Bucket matchers, recorders and the render handle used here.
• Histograms and summaries — Prometheus docs, accessed 2026. How quantiles are estimated from buckets and why bucket choice matters.
• Traces — OpenTelemetry docs, accessed 2026. Spans, context and what a trace is meant to answer.
• tracing-opentelemetry crate documentation — docs.rs, accessed 2026. The layer that turns tracing spans into OpenTelemetry spans.
• opentelemetry-otlp crate documentation — docs.rs, accessed 2026. Exporter options and environment variables for OTLP.
• tracing::Instrument — docs.rs, accessed 2026. The async-safe way to attach a span to a future.