Load Test & Tuning

Install oha

oha is an HTTP load generator with a live TUI showing latency percentiles as they update. Easier than vegeta for ad-hoc work; install via cargo:

cargo install oha
oha --version

If you'd rather use vegeta or k6, the patterns transfer — only the command lines change.

Baseline run

Run the gateway, the mock, and the observability stack from Chapter 05. From a fourth terminal:

oha -z 30s -c 100 -m POST \
  -H 'content-type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"hello"}],"stream":true}' \
  http://127.0.0.1:8080/v1/chat/completions

Flags: -z 30s (run for 30 seconds), -c 100 (100 concurrent workers). Watch oha's live histogram update. After 30s you'll see a summary like:

Summary:
  Success rate: 100.00%
  Total:        30.0202 secs
  Slowest:      0.8741 secs
  Fastest:      0.7610 secs
  Average:      0.8077 secs
  Requests/sec: 123.7

Response time histogram:
  0.761 [1]    |
  0.772 [125]  |■■■■
  0.784 [612]  |■■■■■■■■■■■■■■■■■■■
  ...
  0.874 [12]   |

Latency distribution:
  10%: 0.789 secs
  25%: 0.795 secs
  50%: 0.806 secs
  75%: 0.819 secs
  90%: 0.831 secs
  95%: 0.839 secs
  99%: 0.860 secs

That ~800ms is the mock's "response time" — 19 chunks throttled at 40ms each = 760ms minimum. Anything above that is your gateway's overhead plus jitter. If you're seeing < 100ms of extra overhead, you're doing fine.

Tune tokio worker threads

By default, #[tokio::main] uses one worker per CPU core. For a network-I/O service that's usually fine, but you can get measurable wins by under- or over-subscribing depending on workload shape.

Make the worker count configurable. Replace #[tokio::main] in infergw/src/main.rs with the manual builder form:

fn main() -> anyhow::Result<()> {
    let workers = std::env::var("INFERGW_WORKER_THREADS")
        .ok().and_then(|s| s.parse().ok())
        .unwrap_or_else(num_cpus::get);

    let runtime = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(workers)
        .enable_all()
        .thread_name("infergw-worker")
        .build()?;

    runtime.block_on(async_main())
}

async fn async_main() -> anyhow::Result<()> {
    // ... existing body of the old main()
    Ok(())
}

Add num_cpus = "1.16" to Cargo.toml.

Now run the baseline test with different worker counts:

# Single thread — surprisingly competitive for pure I/O
INFERGW_WORKER_THREADS=1 cargo run -p infergw --release

# Then in another terminal, run the same oha command and note the p99.
# Repeat with =2, =4, =8.

You'll find for this workload (mostly waiting on the mock) that 1–2 threads is plenty, and the marginal CPU gain past ~4 threads is approximately zero. The right answer for production: match worker count to expected CPU usage of the gateway itself, not to client concurrency. tokio happily multiplexes thousands of concurrent connections per thread.

Read the tail honestly

Latency averages lie. The real distribution looks like:

  count
   |
   |   ███████
   |   ███████
   |   ███████
   |   ████████
   |   █████████
   |   █████████████
   |   ████████████████████████   ← long tail
   +───────────────────────────────────▶  latency
       p50  p75  p95  p99  p99.9

Your average sits where the tall bars are. Your worst customer experience lives in the tail. Track p99 and p99.9, not mean. From Chapter 05:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.99, sum by (le) (rate(infergw_request_duration_seconds_bucket[1m])))' \
  | jq '.data.result[0].value[1]'

That's your live p99. Use this during the load test to see how it moves with concurrency.

Interpret the metrics during load

While the load test is running, open Grafana / Prometheus and have these four panels in front of you:

1. sum(rate(infergw_requests_total[1m])) by (status) — Are 5xx-class responses appearing? They shouldn't be unless you've broken something.
2. histogram_quantile(0.99, sum by (le) (rate(infergw_request_duration_seconds_bucket[1m]))) — Watch this number when you bump concurrency. The point at which it starts climbing nonlinearly is your knee.
3. infergw_inflight — How many requests are in flight at any moment. If this caps at some number while RPS plateaus, you've found a bottleneck — usually upstream connection pool or worker saturation.
4. rate(infergw_coalesce_followers_total[1m]) / rate(infergw_requests_total[1m]) — Coalescing ratio. If you're sending identical requests, this should approach (concurrency − 1) / concurrency.

The shape of the load curve as you increase concurrency from -c 10 to -c 1000 tells the story: throughput rises linearly, plateaus, then latency starts increasing while throughput stays flat. That plateau-and-tail is your real capacity. The number where latency starts climbing is what you'd report as "this gateway handles X concurrent requests at acceptable p99."

Prove the coalescer's win

Two test cases:

(A) Identical prompts — coalescing on:

oha -z 30s -c 200 -m POST \
  -H 'content-type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"identical"}],"stream":true}' \
  http://127.0.0.1:8080/v1/chat/completions

Count the upstream-side requests. The mock logs one INFO line per incoming request — docker logs if you ran it that way, or just count in the terminal:

# In the mock's terminal, after the run:
# (manually count, or)
# Adjust the mock to count in a static and print on every request.

(B) Distinct prompts — coalescing off in practice:

# oha can't easily randomize the body per-request out-of-the-box.
# Use a small bash loop with curl + xargs -P for parallelism:
seq 1 1000 | xargs -P 200 -I{} curl -sN -o /dev/null \
  -H 'content-type: application/json' \
  -d "{\"model\":\"x\",\"messages\":[{\"role\":\"user\",\"content\":\"prompt {}\"}],\"stream\":true}" \
  http://127.0.0.1:8080/v1/chat/completions

(A) should show ~38 upstream calls in 30 seconds (one in-flight at a time, ~800ms each). (B) should show ~1000 — one per distinct prompt. That ratio (200x or so under the right concurrency) is the coalescer doing its job.

Checkpoint

• You've driven sustained load through the gateway and watched real metrics move.
• You know how p99 changes with concurrency and where the knee is.
• Tokio worker thread count is configurable via env var; you've tried a few values.
• The coalescer's effectiveness is now measurable, not just plausible.

You're done with the build. One short chapter on where to take this next.

→ Chapter 08: Where to next