Load Test & Tuning

Install oha

oha is an HTTP load generator with a live TUI showing latency percentiles as they update. Easier than vegeta for ad-hoc work; install via cargo:

cargo install oha
oha --version

If you'd rather use vegeta or k6, the patterns transfer — only the command lines change.

Baseline run

Run the gateway, the mock, and the observability stack from Chapter 05. From a fourth terminal:

oha -z 30s -c 100 -m POST \
  -H 'content-type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"hello"}],"stream":true}' \
  http://127.0.0.1:8080/v1/chat/completions

Flags: -z 30s (run for 30 seconds), -c 100 (100 concurrent workers). Watch oha's live histogram update. After 30s you'll see a summary like:

Summary:
  Success rate: 100.00%
  Total:        30.0202 secs
  Slowest:      0.8741 secs
  Fastest:      0.7610 secs
  Average:      0.8077 secs
  Requests/sec: 123.7

Response time histogram:
  0.761 [1]    |
  0.772 [125]  |■■■■
  0.784 [612]  |■■■■■■■■■■■■■■■■■■■
  ...
  0.874 [12]   |

Latency distribution:
  10%: 0.789 secs
  25%: 0.795 secs
  50%: 0.806 secs
  75%: 0.819 secs
  90%: 0.831 secs
  95%: 0.839 secs
  99%: 0.860 secs

That ~800ms is the mock's "response time" — 19 chunks throttled at 40ms each = 760ms minimum. Anything above that is your gateway's overhead plus jitter. If you're seeing < 100ms of extra overhead, you're doing fine.

Tune tokio worker threads

By default, #[tokio::main] uses one worker per CPU core. For a network-I/O service that's usually fine, but you can get measurable wins by under- or over-subscribing depending on workload shape.

Make the worker count configurable. Replace #[tokio::main] in infergw/src/main.rs with the manual builder form:

fn main() -> anyhow::Result<()> {
    let workers = std::env::var("INFERGW_WORKER_THREADS")
        .ok().and_then(|s| s.parse().ok())
        .unwrap_or_else(num_cpus::get);

    let runtime = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(workers)
        .enable_all()
        .thread_name("infergw-worker")
        .build()?;

    runtime.block_on(async_main())
}

async fn async_main() -> anyhow::Result<()> {
    // ... existing body of the old main()
    Ok(())
}

Add num_cpus = "1.16" to Cargo.toml.

Now run the baseline test with different worker counts:

# Single thread — surprisingly competitive for pure I/O
INFERGW_WORKER_THREADS=1 cargo run -p infergw --release

# Then in another terminal, run the same oha command and note the p99.
# Repeat with =2, =4, =8.

You'll find for this workload (mostly waiting on the mock) that 1–2 threads is plenty, and the marginal CPU gain past ~4 threads is approximately zero. The right answer for production: match worker count to expected CPU usage of the gateway itself, not to client concurrency. tokio happily multiplexes thousands of concurrent connections per thread.

Read the tail honestly

Latency averages lie. The real distribution looks like:

  count
   |
   |   ███████
   |   ███████
   |   ███████
   |   ████████
   |   █████████
   |   █████████████
   |   ████████████████████████   ← long tail
   +───────────────────────────────────▶  latency
       p50  p75  p95  p99  p99.9

Your average sits where the tall bars are. Your worst customer experience lives in the tail. Track p99 and p99.9, not mean. From Chapter 05:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.99, sum by (le) (rate(infergw_request_duration_seconds_bucket[1m])))' \
  | jq '.data.result[0].value[1]'

That's your live p99. Use this during the load test to see how it moves with concurrency.

Interpret the metrics during load

While the load test is running, open Grafana / Prometheus and have these four panels in front of you:

1. sum(rate(infergw_requests_total[1m])) by (status) — Are 5xx-class responses appearing? They shouldn't be unless you've broken something.
2. histogram_quantile(0.99, sum by (le) (rate(infergw_request_duration_seconds_bucket[1m]))) — Watch this number when you bump concurrency. The point at which it starts climbing nonlinearly is your knee.
3. infergw_inflight — How many requests are in flight at any moment. If this caps at some number while RPS plateaus, you've found a bottleneck — usually upstream connection pool or worker saturation.
4. rate(infergw_coalesce_followers_total[1m]) / rate(infergw_requests_total[1m]) — Coalescing ratio. If you're sending identical requests, this should approach (concurrency − 1) / concurrency.

The shape of the load curve as you increase concurrency from -c 10 to -c 1000 tells the story: throughput rises linearly, plateaus, then latency starts increasing while throughput stays flat. That plateau-and-tail is your real capacity. The number where latency starts climbing is what you'd report as "this gateway handles X concurrent requests at acceptable p99."

Prove the coalescer's win

Two test cases:

(A) Identical prompts — coalescing on:

oha -z 30s -c 200 -m POST \
  -H 'content-type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"identical"}],"stream":true}' \
  http://127.0.0.1:8080/v1/chat/completions

Count the upstream-side requests. The mock logs one INFO line per incoming request — docker logs if you ran it that way, or just count in the terminal:

# In the mock's terminal, after the run:
# (manually count, or)
# Adjust the mock to count in a static and print on every request.

(B) Distinct prompts — coalescing off in practice:

# oha can't easily randomize the body per-request out-of-the-box.
# Use a small bash loop with curl + xargs -P for parallelism:
seq 1 1000 | xargs -P 200 -I{} curl -sN -o /dev/null \
  -H 'content-type: application/json' \
  -d "{\"model\":\"x\",\"messages\":[{\"role\":\"user\",\"content\":\"prompt {}\"}],\"stream\":true}" \
  http://127.0.0.1:8080/v1/chat/completions

(A) should show ~38 upstream calls in 30 seconds (one in-flight at a time, ~800ms each). (B) should show ~1000 — one per distinct prompt. That ratio (200x or so under the right concurrency) is the coalescer doing its job.

Checkpoint

• You've driven sustained load through the gateway and watched real metrics move.
• You know how p99 changes with concurrency and where the knee is.
• Tokio worker thread count is configurable via env var; you've tried a few values.
• The coalescer's effectiveness is now measurable, not just plausible.

You're done with the build. One short chapter on where to take this next.

→ Chapter 08: Where to next

✓ Check yourself

Answer from memory before opening each one.

1. Run the baseline: oha -z 30s -c 100 against the gateway with the mock behind it. What fastest latency and requests per second should you expect, and what does a result far from that mean?

   Show answer

   The mock throttles 19 chunks at 40 ms, so the floor is about 760 ms and a typical request takes about 800 ms. With 100 workers that gives roughly 100 / 0.8, about 125 requests a second, close to the 123.7 shown. A much lower rate with a much higher average means something serialises requests, such as a lock or a pool limit. See baseline run.

2. Run 30 seconds of identical requests with -c 200, then 1,000 distinct prompts. How many upstream calls should the mock see each time, and what should the follower counter do?

   Show answer

   Identical: about 38 calls, one per 0.8 second window, against roughly 7,500 client requests, so the follower counter climbs fast. Distinct: about 1,000 calls and the counter stays near zero. If the identical run reaches the mock thousands of times, the bodies differ. If the distinct run raises the counter, the shell substitution failed and every body is the same. See prove the coalescer's win.

3. One worker thread handles 100 concurrent streams fine. Why, and what change to the gateway would make you add threads?

   Show answer

   Each request is a task that is suspended at an await while waiting on I/O, so one thread can drive many of them. Threads exist to spend CPU. Add them when the gateway does real work per chunk, such as parsing SSE, counting tokens or hashing large bodies, and a profile or per-thread CPU shows workers saturating. See tune tokio worker threads.

4. A closed-loop run with 100 connections hits a 5-second upstream stall. Why does oha report a mild p99, and what do you change?

   Show answer

   Each worker waits for its stalled response and sends nothing meanwhile, so the requests that would have arrived during the stall are never made or timed. That is coordinated omission. Use a fixed rate with -q and --latency-correction, so latency is measured from the scheduled start time. See read the tail honestly.

5. Raising -c from 10 to 1,000, throughput levels off and p99 climbs. infergw_inflight stops growing at one fixed number. What does that tell you, and where do you look?

   Show answer

   A plateau in throughput with rising latency means you passed capacity, and an in-flight count that stops at a fixed value means some limit in the path caps concurrency. Check the upstream connection pool and the upstream's own concurrency limit, worker CPU, and the file descriptor limit on the load generator. Report the concurrency at the knee, not the peak. See interpret the metrics.

Takeaway

• Against a fixed-cost mock, a load test measures how little the gateway adds. The floor is about 760 ms and the gap between fastest and p99 is gateway jitter.
• At fixed concurrency, throughput is concurrency divided by latency, so 100 workers at 0.8 seconds give about 125 requests a second. A far lower number means something serialises.
• Tokio worker threads exist to use CPU, not to hold connections, so size them by the gateway's own work.
• Report p99 and p99.9 rather than the mean, and take the knee where latency rises while throughput is flat as the capacity.
• Closed-loop tools hide stalls. Test at a fixed rate with latency correction to see the true tail.
• The coalescer's win is checkable: about 38 upstream calls for 30 seconds of identical load against one per distinct prompt, confirmed by the follower counter.

You have built and measured the gateway. → Where to next

Further reading

• oha — GitHub, hatoo, accessed 2026. The load generator used here, including the rate and latency-correction flags.
• "How NOT to Measure Latency" — Gil Tene, Strange Loop talk, accessed 2026. The clearest account of coordinated omission and why averages mislead.
• tokio::runtime::Builder — docs.rs, accessed 2026. Worker thread and runtime options behind the tuning knob.
• Histograms and summaries — Prometheus docs, accessed 2026. What a histogram quantile can and cannot tell you.
• Query functions — Prometheus docs, accessed 2026. The reference for histogram_quantile and rate used in the panels.