Evaluation & Quality

What "quality" means for infrastructure

For a model team, quality means accuracy and safety. For this team, quality means: the system delivers expected performance, under expected load, within expected cost, with expected reliability. Each of those words is a metric. Each metric is something you measure, set targets for, and regress against.

Infrastructure quality has four dimensions:

• Performance — latency, throughput, TTFT.
• Efficiency — GPU utilization, memory utilization, cost per request.
• Reliability — availability, error rate, MTTR.
• Predictability — tail variance, behavior under load.

The core metrics — for an LLM serving role

TTFT — time-to-first-token

The single most user-visible metric for chat/streaming workloads. A 200ms TTFT feels responsive; 2 seconds feels broken.

TTFT is dominated by prefill — processing the input prompt through every layer to produce the first output logit. Prefill is compute-bound (a big matmul over the whole prompt) and parallelizable. Things that affect TTFT:

• Prompt length. Linear in tokens (with the matmul cost scaling).
• Concurrency. If 16 prefills are queued, the 16th waits behind them.
• Tensor parallel shape. More TP = faster prefill at the cost of communication.
• Prefix caching. Cached prompts skip prefill entirely for the cached portion.
• Chunked prefill. Interleave prefill with decode so big prompts don't starve other requests.

Throughput and "goodput"

Throughput is tokens/second per GPU at the system level. Goodput is throughput on requests that met their SLO. A system that does 5000 tok/s/GPU with p99 of 8 seconds (when the SLO is 2 seconds) has zero goodput.

Always report both. The conversion from raw throughput to goodput exposes overload before it shows up in error rates.

Latency percentiles — p50, p95, p99, p99.9

Means and medians lie. Always quote a percentile.

• p50 — the typical experience. Useful for capacity sizing.
• p95 — the "bad day" experience. Most SLOs sit here.
• p99 — the tail. Where queueing under load, GC pauses, NUMA cross-talk show up.
• p99.9 — the worst typical bucket. For high-volume services, the difference between p99 and p99.9 is "lots of users feel this."

Compute percentiles over a moving window (5 or 10 minutes typical). Use HDR histograms or Prometheus histogram buckets, not averages.

GPU utilization — three numbers, all called "utilization," all different

This single misunderstanding loses interviews. The naive question "what's your GPU utilization?" hides three distinct things.

Memory profiling

HBM is the scarce resource. Three categories share it:

• Model weights — fixed once loaded. 70B FP16 ≈ 140 GB; FP8 ≈ 70 GB; INT4 ≈ 35 GB.
• Activations — transient, per forward pass. Small in inference; massive in training without checkpointing.
• KV-cache — the headline variable. Grows with batch × context length.

Tools:

• nvidia-smi for coarse "used / total."
• DCGM DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE for time series.
• PyTorch torch.cuda.memory_summary() — fragmentation details inside a process.
• nsys profile + --cuda-memory-usage — allocations as a timeline.
• vLLM exports memory utilization & KV-cache block usage as metrics.

OOM debug rule: where in the timeline did allocation spike? First-step OOM usually means too-large weights or shape. Step-N OOM means activation/cache growth or fragmentation.

Cost per million tokens — the headline business metric

The number that matters to the CFO and lets you defend the in-house build vs an external API. The formula:

def cost_per_million_tokens(gpu_hourly_cost_usd: float,
                            tokens_per_second_per_gpu: float,
                            num_gpus: int = 1) -> float:
    """USD per million output tokens served."""
    tokens_per_hour = tokens_per_second_per_gpu * 3600
    fleet_cost_per_hour = gpu_hourly_cost_usd * num_gpus
    fleet_tokens_per_hour = tokens_per_hour * num_gpus
    return (fleet_cost_per_hour / fleet_tokens_per_hour) * 1_000_000

# Example: H100 at $3.50/hr, vLLM Llama-3.1-70B TP=4 at ~7000 tok/s aggregate
# fleet of 4 GPUs serving the tensor-parallel replica
cost = cost_per_million_tokens(3.50, 7000 / 4, num_gpus=4)
# ≈ $2.00 per million output tokens, vs ~$15 for hosted frontier APIs

Caveats interviewers will probe:

• Utilization assumption. 7000 tok/s only at saturation. Idle headroom raises effective cost.
• Reserved vs on-demand pricing. Use blended cost across the mix.
• Input tokens. Prefill costs compute too. Report input + output separately or assume a profile.
• Overheads. Control plane, gateway, observability, on-call all cost real money. Add 20-30% loaded overhead.

Benchmarking methodology

Bench numbers in a vacuum are dangerous. A defensible benchmark:

1. Pin everything. Driver version, CUDA version, runtime version, model checkpoint, GPU clocks (lock with nvidia-smi -lgc), batch sizes, concurrency.
2. Warm up. Discard the first N seconds. JIT, KV-cache allocator, NCCL — all need warm.
3. Realistic load shape. Open-loop (Poisson arrivals at target RPS) reveals tail; closed-loop (fixed concurrency) reveals max throughput. Run both, report both.
4. Real prompt distribution. Don't bench with all 100-token prompts when production is 1500-token mean. Use a sampled production trace if possible.
5. Long enough run. 60+ seconds at steady state. Five-second microbenches hide thermal throttling and cache warmup.
6. Report ranges. 3-5 runs; show median and spread.

The vllm bench serve command and Triton's perf_analyzer are starting points; for cross-stack comparison roll your own with the same harness in front of each.

Perf regression tests in CI

You will deploy new vLLM versions, new TensorRT engines, new drivers. Each can quietly slow you by 5-15%. Catch it before users do.

• Nightly benchmark on a held-out test rig, fixed model, fixed prompts. Compare to last week's median.
• PR-level smoke bench for serving stack changes — 30 seconds at one batch size, fail if >5% regression.
• Track quality alongside perf — a quantization change might gain 20% throughput and lose 4% accuracy. Both must show on the same dashboard.
• Alert on real-time drift — production p95 tokens/sec falling outside its baseline band is a page even before customers notice.

The dashboards that matter

For an internal review or interview design discussion, you should be able to list the boards you'd build.

• Fleet utilization — SM-active, memory used, power, per node, heatmapped.
• Per-service health — request rate, error rate, queue depth, p50/p95/p99 latency, TTFT, tok/s.
• Per-tenant cost — $ per day, $ per million tokens, broken out by namespace.
• Capacity pressure — pending pod count by priority class, time-to-bind p95, quota usage per team.
• Driver/hardware health — Xid counts, ECC error rates, fabric link state, temperature.
• KV-cache view — block allocation %, prefix cache hit rate, eviction rate.