Chapter 06 · ~20 minutes

Docker & Deploy

Builder → distroless multi-stage build, healthcheck wiring, and a Kubernetes Deployment YAML you could actually apply.

Multi-stage Dockerfile

The strategy: a fat builder image with the Rust toolchain produces a static binary. We then copy that binary into a tiny distroless image. The result is ~15 MB, no shell, no package manager, minimal attack surface.

Save this at infergw-workspace/Dockerfile:

# syntax=docker/dockerfile:1.7

# ---------- Stage 1: builder ----------
FROM rust:1.78-slim-bookworm AS builder

# System deps for openssl-sys etc. (rustls avoids most of these).
RUN apt-get update && apt-get install -y --no-install-recommends \
      pkg-config ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work

# Cache dependencies separately from source for fast incremental builds.
COPY Cargo.toml Cargo.lock ./
COPY infergw/Cargo.toml infergw/Cargo.toml
COPY mock-upstream/Cargo.toml mock-upstream/Cargo.toml

# Touch dummy mains so cargo can resolve and build deps.
RUN mkdir -p infergw/src mock-upstream/src \
 && echo 'fn main(){}' > infergw/src/main.rs \
 && echo 'fn main(){}' > mock-upstream/src/main.rs

ENV CARGO_NET_RETRY=10 \
    RUSTFLAGS="-C target-cpu=x86-64-v3"

RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/work/target \
    cargo build --release -p infergw \
 && rm -f infergw/src/main.rs mock-upstream/src/main.rs

# Now copy real sources and do the actual build.
COPY infergw infergw
COPY mock-upstream mock-upstream

RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/work/target \
    cargo build --release -p infergw \
 && cp /work/target/release/infergw /infergw

# ---------- Stage 2: runtime ----------
FROM gcr.io/distroless/cc-debian12:nonroot

COPY --from=builder /infergw /usr/local/bin/infergw

ENV INFERGW_BIND=0.0.0.0:8080 \
    INFERGW_LOG_JSON=1 \
    RUST_LOG=infergw=info,tower_http=info

EXPOSE 8080

# Distroless has no shell, so we exec the binary directly.
ENTRYPOINT ["/usr/local/bin/infergw"]
Three things that earn their keep here

1. distroless/cc-debian12:nonroot — has libc and libgcc (which our binary needs because of rustls-tls's ring dependency, and because we didn't target musl), runs as a non-root user, and is ~22 MB before our binary. :nonroot avoids the "running containers as root" CVE flag without extra config.

2. RUSTFLAGS="-C target-cpu=x86-64-v3" — generates code for ~2017+ x86 CPUs (Haswell or newer). Modest perf win on integer-heavy workloads. Skip this if you might run on older hardware.

3. The dummy-main trick — by building once with empty main.rs files, cargo compiles and caches all your dependencies. The second build only recompiles your code. This cuts incremental Docker builds from minutes to seconds.

Why the cache mounts matter

The two --mount=type=cache directives are BuildKit-specific and dramatically speed up rebuilds. They persist ~/.cargo/registry and the target/ directory across builds — between iterations, dependency compilation is essentially free.

If you're on an older Docker that doesn't support BuildKit by default, set:

export DOCKER_BUILDKIT=1

Build it

cd infergw-workspace
docker build -t infergw:0.1.0 .

First build takes 2–4 minutes (downloading and compiling the dependency tree). Subsequent rebuilds are 10–30 seconds. Check the final size:

docker images infergw:0.1.0
# REPOSITORY   TAG     IMAGE ID       CREATED         SIZE
# infergw      0.1.0   abc123def      1 minute ago    35MB

Around 30–40 MB depending on which features are compiled in. Most of that is the binary itself (Rust release builds aren't tiny — they include monomorphized generics for every async type), plus libc / libgcc / CA certs.

Want to go smaller?

Two options, in order of effort:

  • strip = true in [profile.release] — already done in Chapter 00. Strips debug symbols. ~30% smaller.
  • Target musl with x86_64-unknown-linux-musl and use distroless/static — gives you a single-file ~12 MB image with no libc dependency. Adds build complexity (musl rust toolchain), and some C dependencies (anything depending on OpenSSL) won't build cleanly. With pure-Rust deps (rustls, ring), it just works.

Run it

docker run --rm -p 8080:8080 \
  -e INFERGW_UPSTREAM=http://host.docker.internal:9000 \
  -e INFERGW_LOG_JSON=1 \
  infergw:0.1.0

(host.docker.internal is how a container on Mac / Windows reaches the host's mock-upstream. On Linux, pass --add-host=host.docker.internal:host-gateway or use --network host.)

From another terminal:

curl -i http://127.0.0.1:8080/healthz
# HTTP/1.1 200 OK ... ok

curl -N -H 'content-type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"hi"}],"stream":true}' \
  http://127.0.0.1:8080/v1/chat/completions

You should see the same SSE stream as before — now flowing through the containerized gateway to the mock running on your host.

A word on PGO

Profile-Guided Optimization can give Rust binaries another 5–15% throughput in CPU-bound code. The workflow:

  1. Build with -Cprofile-generate=<dir> instead of the normal release profile.
  2. Run a representative load (e.g. oha hitting your gateway for 60s).
  3. Merge profiles with llvm-profdata.
  4. Rebuild with -Cprofile-use=<merged-file>.

For a network-I/O-bound service like this gateway, PGO is mostly a marginal win — your bottleneck is usually upstream, not your CPU. Skip it for v1, revisit if a profile says CPU is actually pegged in the gateway's own code. The rustc PGO docs walk through the full flow when you're ready.

Kubernetes Deployment

A realistic Deployment + Service. Save as infergw-workspace/deploy/k8s.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: infergw
  labels: { app: infergw }
spec:
  replicas: 3
  selector:
    matchLabels: { app: infergw }
  template:
    metadata:
      labels: { app: infergw }
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: infergw
          image: infergw:0.1.0
          imagePullPolicy: IfNotPresent
          ports:
            - { containerPort: 8080, name: http }
          env:
            - { name: INFERGW_BIND, value: "0.0.0.0:8080" }
            - { name: INFERGW_UPSTREAM, value: "http://vllm:8000" }
            - { name: INFERGW_UPSTREAM_TIMEOUT_MS, value: "30000" }
            - { name: INFERGW_HEDGE_AFTER_MS, value: "1500" }
            - { name: INFERGW_SHUTDOWN_GRACE_MS, value: "20000" }
            - { name: INFERGW_LOG_JSON, value: "1" }
            - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector:4317" }
            - { name: RUST_LOG, value: "infergw=info" }
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            initialDelaySeconds: 2
            periodSeconds: 3
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 6
          resources:
            requests:
              cpu: "500m"
              memory: "128Mi"
            limits:
              cpu: "2"
              memory: "512Mi"
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
          lifecycle:
            preStop:
              exec:
                # Give the LB an extra second to update before we close.
                command: ["/bin/sleep", "2"]   # WON'T WORK on distroless
                # alternative for distroless: omit preStop and rely on
                # readyz-flips-on-SIGTERM, plus a long terminationGracePeriod.

---
apiVersion: v1
kind: Service
metadata:
  name: infergw
spec:
  selector: { app: infergw }
  ports:
    - { name: http, port: 80, targetPort: 8080 }
The preStop / distroless tension

The classic Kubernetes pattern is "preStop sleeps a few seconds so endpoints drain before the process gets SIGTERM." But distroless has no /bin/sleep. Three options: (a) bake a tiny sleep binary into your image; (b) switch to distroless/cc-debian12:debug which has busybox; (c) skip preStop entirely and rely on the readiness probe flipping to fail immediately when SIGTERM hits (which we implemented in Chapter 01). Option (c) is cleanest if your terminationGracePeriodSeconds is long enough — and it is here.

Comment out the preStop block and rely on the shutdown logic from Chapter 01. The flow is:

  1. K8s sends SIGTERM.
  2. Our shutdown handler flips readyz to 503 immediately.
  3. Endpoint controller observes readiness failure within ~3s (matches our probe period).
  4. Service load balancer stops routing new requests.
  5. In-flight requests drain over the next ~15s (grace period from INFERGW_SHUTDOWN_GRACE_MS).
  6. Process exits. K8s acknowledges within the 30s terminationGracePeriodSeconds.

Total cycle: ~18s from SIGTERM to clean exit, with zero new requests dropped after step 4.

Apply it to a real cluster

If you have a kind / minikube / k3d cluster running, push the image and apply:

# kind: load image directly into the cluster (no registry needed)
kind load docker-image infergw:0.1.0

kubectl apply -f deploy/k8s.yaml
kubectl rollout status deploy/infergw

# Port-forward to test:
kubectl port-forward svc/infergw 8080:80

Then curl as before. Pull the pods down with kubectl rollout restart deploy/infergw and watch the rolling update — readyz flipping correctly is what makes the rollout zero-downtime.

Checkpoint

  • docker build -t infergw:0.1.0 . succeeds in ~3 min the first time, ~30s on rebuild.
  • Image is ~35 MB or smaller and runs as non-root.
  • Container starts, serves /healthz, /readyz, /metrics, and /v1/chat/completions.
  • The Kubernetes manifest is realistic: probes, security context, env config, prometheus scrape annotations, shutdown handshake.

Next we drive load through it and watch the metrics tell the truth.

→ Chapter 07: Load test & tuning