Docker & Deploy

Multi-stage Dockerfile

The strategy: a fat builder image with the Rust toolchain produces a static binary. We then copy that binary into a tiny distroless image. The result is ~15 MB, no shell, no package manager, minimal attack surface.

Save this at infergw-workspace/Dockerfile:

# syntax=docker/dockerfile:1.7

# ---------- Stage 1: builder ----------
FROM rust:1.78-slim-bookworm AS builder

# System deps for openssl-sys etc. (rustls avoids most of these).
RUN apt-get update && apt-get install -y --no-install-recommends \
      pkg-config ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work

# Cache dependencies separately from source for fast incremental builds.
COPY Cargo.toml Cargo.lock ./
COPY infergw/Cargo.toml infergw/Cargo.toml
COPY mock-upstream/Cargo.toml mock-upstream/Cargo.toml

# Touch dummy mains so cargo can resolve and build deps.
RUN mkdir -p infergw/src mock-upstream/src \
 && echo 'fn main(){}' > infergw/src/main.rs \
 && echo 'fn main(){}' > mock-upstream/src/main.rs

ENV CARGO_NET_RETRY=10 \
    RUSTFLAGS="-C target-cpu=x86-64-v3"

RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/work/target \
    cargo build --release -p infergw \
 && rm -f infergw/src/main.rs mock-upstream/src/main.rs

# Now copy real sources and do the actual build.
COPY infergw infergw
COPY mock-upstream mock-upstream

RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/work/target \
    cargo build --release -p infergw \
 && cp /work/target/release/infergw /infergw

# ---------- Stage 2: runtime ----------
FROM gcr.io/distroless/cc-debian12:nonroot

COPY --from=builder /infergw /usr/local/bin/infergw

ENV INFERGW_BIND=0.0.0.0:8080 \
    INFERGW_LOG_JSON=1 \
    RUST_LOG=infergw=info,tower_http=info

EXPOSE 8080

# Distroless has no shell, so we exec the binary directly.
ENTRYPOINT ["/usr/local/bin/infergw"]

Why the cache mounts matter

The two --mount=type=cache directives are BuildKit-specific and dramatically speed up rebuilds. They persist ~/.cargo/registry and the target/ directory across builds — between iterations, dependency compilation is essentially free.

If you're on an older Docker that doesn't support BuildKit by default, set:

export DOCKER_BUILDKIT=1

Build it

cd infergw-workspace
docker build -t infergw:0.1.0 .

First build takes 2–4 minutes (downloading and compiling the dependency tree). Subsequent rebuilds are 10–30 seconds. Check the final size:

docker images infergw:0.1.0
# REPOSITORY   TAG     IMAGE ID       CREATED         SIZE
# infergw      0.1.0   abc123def      1 minute ago    35MB

Around 30–40 MB depending on which features are compiled in. Most of that is the binary itself (Rust release builds aren't tiny — they include monomorphized generics for every async type), plus libc / libgcc / CA certs.

Run it

docker run --rm -p 8080:8080 \
  -e INFERGW_UPSTREAM=http://host.docker.internal:9000 \
  -e INFERGW_LOG_JSON=1 \
  infergw:0.1.0

(host.docker.internal is how a container on Mac / Windows reaches the host's mock-upstream. On Linux, pass --add-host=host.docker.internal:host-gateway or use --network host.)

From another terminal:

curl -i http://127.0.0.1:8080/healthz
# HTTP/1.1 200 OK ... ok

curl -N -H 'content-type: application/json' \
  -d '{"model":"x","messages":[{"role":"user","content":"hi"}],"stream":true}' \
  http://127.0.0.1:8080/v1/chat/completions

You should see the same SSE stream as before — now flowing through the containerized gateway to the mock running on your host.

A word on PGO

Profile-Guided Optimization can give Rust binaries another 5–15% throughput in CPU-bound code. The workflow:

1. Build with -Cprofile-generate=<dir> instead of the normal release profile.
2. Run a representative load (e.g. oha hitting your gateway for 60s).
3. Merge profiles with llvm-profdata.
4. Rebuild with -Cprofile-use=<merged-file>.

For a network-I/O-bound service like this gateway, PGO is mostly a marginal win — your bottleneck is usually upstream, not your CPU. Skip it for v1, revisit if a profile says CPU is actually pegged in the gateway's own code. The rustc PGO docs walk through the full flow when you're ready.

Kubernetes Deployment

A realistic Deployment + Service. Save as infergw-workspace/deploy/k8s.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: infergw
  labels: { app: infergw }
spec:
  replicas: 3
  selector:
    matchLabels: { app: infergw }
  template:
    metadata:
      labels: { app: infergw }
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: infergw
          image: infergw:0.1.0
          imagePullPolicy: IfNotPresent
          ports:
            - { containerPort: 8080, name: http }
          env:
            - { name: INFERGW_BIND, value: "0.0.0.0:8080" }
            - { name: INFERGW_UPSTREAM, value: "http://vllm:8000" }
            - { name: INFERGW_UPSTREAM_TIMEOUT_MS, value: "30000" }
            - { name: INFERGW_HEDGE_AFTER_MS, value: "1500" }
            - { name: INFERGW_SHUTDOWN_GRACE_MS, value: "20000" }
            - { name: INFERGW_LOG_JSON, value: "1" }
            - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector:4317" }
            - { name: RUST_LOG, value: "infergw=info" }
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            initialDelaySeconds: 2
            periodSeconds: 3
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 6
          resources:
            requests:
              cpu: "500m"
              memory: "128Mi"
            limits:
              cpu: "2"
              memory: "512Mi"
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
          lifecycle:
            preStop:
              exec:
                # Give the LB an extra second to update before we close.
                command: ["/bin/sleep", "2"]   # WON'T WORK on distroless
                # alternative for distroless: omit preStop and rely on
                # readyz-flips-on-SIGTERM, plus a long terminationGracePeriod.

---
apiVersion: v1
kind: Service
metadata:
  name: infergw
spec:
  selector: { app: infergw }
  ports:
    - { name: http, port: 80, targetPort: 8080 }

Comment out the preStop block and rely on the shutdown logic from Chapter 01. The flow is:

1. K8s sends SIGTERM.
2. Our shutdown handler flips readyz to 503 immediately.
3. Endpoint controller observes readiness failure within 3 to 6 s (two failed probes at a 3 s period).
4. Service load balancer stops routing new requests.
5. In-flight requests drain over the next ~15s (grace period from INFERGW_SHUTDOWN_GRACE_MS).
6. Process exits. K8s acknowledges within the 30s terminationGracePeriodSeconds.

Total cycle: ~18s from SIGTERM to clean exit, with zero new requests dropped after step 4.

# kind: load image directly into the cluster (no registry needed)
kind load docker-image infergw:0.1.0

kubectl apply -f deploy/k8s.yaml
kubectl rollout status deploy/infergw

# Port-forward to test:
kubectl port-forward svc/infergw 8080:80

Checkpoint

• docker build -t infergw:0.1.0 . succeeds in ~3 min the first time, ~30s on rebuild.
• Image is ~35 MB or smaller and runs as non-root.
• Container starts, serves /healthz, /readyz, /metrics, and /v1/chat/completions.
• The Kubernetes manifest is realistic: probes, security context, env config, prometheus scrape annotations, shutdown handshake.

Next we drive load through it and watch the metrics tell the truth.

→ Chapter 07: Load test & tuning

✓ Check yourself

Answer from memory before opening each one.

1. Run the container without INFERGW_UPSTREAM and curl /healthz, then send a streaming request. What do you see, and why?

   Show answer

   /healthz returns 200 ok while the chat request fails. The default upstream is 127.0.0.1:9000, and inside the container that address is the container itself, where nothing listens. The liveness check does not touch the upstream, so it stays green. Set the variable to the host address, as in run it.

2. You change one line in proxy.rs and rebuild. Which Dockerfile steps run again? What changes if you add a dependency?

   Show answer

   Only the copy of the sources and the final cargo build, because the manifest layers and the dummy-main dependency build are cached. Adding a dependency changes Cargo.toml, which invalidates the dependency layer and rebuilds it, though the cache mounts keep downloads and artefacts. See multi-stage Dockerfile.

3. The binary links glibc and is built without musl. Why distroless/cc, and when would you move to distroless/static?

   Show answer

   The cc image ships glibc and libgcc, which a dynamically linked Rust binary needs. A musl build with the static image is smaller and has no libc to patch, but you take on the musl toolchain and any C dependency that will not build against it. Move when image size or attack surface outweighs that build cost. See build it.

4. The manifest sets a grace of 20 seconds and terminationGracePeriodSeconds: 30. A stream has 35 seconds left when SIGTERM arrives. What does the client see, and what is the rule?

   Show answer

   The handler sleeps 20 seconds before axum stops accepting, then waits for streams, but the kubelet sends SIGKILL at 30 seconds, so the client sees a truncated stream. The rule is that readiness delay plus grace plus your longest in-flight request must fit inside the termination period, or you shorten the streams or the grace. See Kubernetes Deployment.

5. Why is the Dockerfile entrypoint in exec form, and what do you give up by having no shell in the image?

   Show answer

   Exec form makes the binary process one, so it receives SIGTERM directly and the shutdown handler runs. A shell wrapper would hold that role and may not forward the signal. Without a shell you lose docker exec sh, a preStop that calls sleep, and a curl-based healthcheck, so you rely on Kubernetes probes and the debug image tag. See Kubernetes Deployment.

6. A colleague proposes profile-guided optimisation and target-cpu=x86-64-v3 to cut p99. How do you decide?

   Show answer

   Profile first. The gateway mostly waits on the network, so its latency is usually set by the upstream, and these flags only help if a CPU profile shows the gateway's own code is the bottleneck. The CPU flag also builds an image that fails on older x86 or ARM nodes. See a word on PGO.

Takeaway

• A multi-stage build keeps the toolchain out of the runtime image. The builder compiles, and the distroless stage copies in only the binary.
• Building dependencies against dummy mains first caches them, so a source edit rebuilds only your code.
• The cc distroless image suits a dynamically linked glibc binary. A static musl build is smaller but limits C dependencies.
• An exec-form entrypoint makes the gateway process one and the receiver of SIGTERM. With no shell, probes replace healthcheck scripts and a preStop sleep.
• The shutdown budget is readiness delay plus grace plus the longest in-flight request, and it must fit inside the termination period.
• Inside a container 127.0.0.1 is the container, so the upstream address comes from configuration.
• Reach for PGO and CPU-targeting flags only after a profile shows the gateway is CPU bound.

The image runs, so now measure its limits. → Load test & tuning

Further reading

• Multi-stage builds — Docker docs, accessed 2026. The mechanism behind the builder and runtime stages.
• Optimize cache usage in builds — Docker docs, accessed 2026. Layer ordering and cache mounts, which the dependency trick relies on.
• Distroless container images — Google Container Tools, accessed 2026. What each image variant contains, including the debug tag.
• Pod termination — Kubernetes docs, accessed 2026. The order of preStop, SIGTERM and SIGKILL.
• Configure liveness, readiness and startup probes — Kubernetes docs, accessed 2026. Probe timing fields that set how fast a draining pod leaves rotation.
• Profiles — The Cargo Book, accessed 2026. The release settings (lto, codegen-units, strip) from chapter 00.