Deployment & Ops

Multi-stage Docker builds

The standard pattern for Rust images: build in a heavy image, copy the binary into a minimal runtime image. Result: ~20MB images that boot in <1s.

# syntax=docker/dockerfile:1.6
FROM rust:1.78-slim AS build
WORKDIR /app
# Cache deps separately from source
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main(){}" > src/main.rs && \
    cargo build --release && rm -rf src
COPY src ./src
RUN cargo build --release --bin gateway

FROM gcr.io/distroless/cc-debian12 AS runtime
COPY --from=build /app/target/release/gateway /usr/local/bin/gateway
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/gateway"]

Things to mention in interviews:

• Layer caching — copying Cargo.toml/Cargo.lock first builds a deps-only layer that's reused across source changes. Huge CI win.
• Distroless base — no shell, no package manager, smaller attack surface. cc variant has glibc.
• Static binary alternative — build with musl (--target x86_64-unknown-linux-musl) for a fully static binary that runs on scratch. Worth doing for the smallest production image.
• cargo-chef — third-party tool that splits Rust builds for better Docker caching; worth knowing by name.

Release vs debug builds

Never deploy debug builds. Performance differences are 10-100x for compute-bound code. Always run perf tests against release.

[profile.release]
opt-level = 3
lto = "fat"            # whole-program optimization
codegen-units = 1      # single CGU enables more inlining; slower compile
panic = "abort"        # smaller binaries, can't unwind across FFI
strip = true           # strip symbols
debug = 1              # keep line-number debug info for backtraces in prod

Tradeoffs:

• lto = "fat" — big perf win; longer compile.
• codegen-units = 1 — better inlining; slower compile; you keep it on for prod releases, not dev.
• panic = "abort" — saves binary size, no unwinding overhead, but you lose catch_unwind. Most services don't need it.
• debug = 1 — keep just enough debug info for symbolic stack traces. Pair with a separate --strip=symbols step and external symbol upload (sentry, datadog).

PGO and LTO

Profile-Guided Optimization: compile instrumented binary, run representative load, recompile using the profile. Typical wins 5-15% on throughput. Tooling: cargo-pgo.

# 1. instrument
RUSTFLAGS="-Cprofile-generate=/tmp/pgo" cargo build --release

# 2. run representative workload
./target/release/gateway &
# ... load test for ~10 min ...
killall gateway

# 3. merge profile data
llvm-profdata merge -o /tmp/pgo.profdata /tmp/pgo

# 4. recompile using profile
RUSTFLAGS="-Cprofile-use=/tmp/pgo.profdata" cargo build --release

Worth doing for the inference gateway; not worth doing for everything. Like jemalloc: benchmark, then claim.

Blue/green deploys

Two production environments (blue, green). Traffic on blue. Deploy new version to green, smoke-test, flip the load balancer. Old blue stays warm; if issue, flip back.

• Zero-downtime cutover.
• Fast rollback (LB flip is seconds).
• Cost — 2x infra during the cutover period.
• State coordination — both environments writing to the same DB. Schema migrations must be compatible across both versions during the window.

For stateful services (orchestrators with active runs), blue/green is harder; canaries (next) tend to be better.

Canaries

Deploy the new version to a small fraction of capacity (e.g., 1% then 10% then 50% then 100%). Monitor SLOs at each step. Roll back on regression.

• Traffic split by header, by user-id hash, or by random sampling.
• Compare metrics between canary and baseline pods. Statistical comparison preferred (e.g., the canary's p99 vs baseline's p99 with confidence intervals).
• Tooling: Argo Rollouts, Flagger, Spinnaker. Or homegrown — a service mesh (Istio, Linkerd) can route by weight.
• Auto-rollback if canary metrics breach. Half-cooked canaries that need humans to notice problems aren't safer.

Feature flags

Decouple deploy from release. Ship code dark; flip flag to enable.

• Boolean flags — on/off per environment.
• Percentage rollout — enable for X% of users.
• Targeted rollout — enable for specific user IDs / tenants.
• Kill switches (covered in 09) are a subset.

Tooling: LaunchDarkly, Flagsmith, Unleash, or a homegrown service. In Rust, statsig-rust, launchdarkly-server-sdk, or your own config-watcher pulling from Postgres.

Rollback strategy

The bar for production: any change can be rolled back in <5 minutes without coordination.

• Code: Redeploy previous image. Pre-pull images on the cluster so rollback is fast.
• Feature flag: Flip the flag. Seconds.
• Schema changes: Always backward-compatible. Expand-then-contract: add new fields/columns, deploy code that handles both old and new, only later remove the old.
• Stateful migrations: Migrations are forward-only; rolling back code while the DB has the new schema must work.
• Drill rollback. Once a quarter, deliberately roll back something in staging. Find the broken assumption while you're not on fire.

Connection pooling

Three pools matter most:

• Database (Postgres): sqlx, deadpool-postgres, tokio-postgres. Size = expected concurrent DB ops per replica. 10-50 typical.
• Redis: fred, deadpool-redis. Size = concurrent Redis ops per replica. 20-100 typical for hot paths.
• HTTP (to backends): reqwest::Client reuses HTTP/2 connections. One Client per process; .pool_max_idle_per_host(N) tunes.

Anti-patterns:

• Creating a new client per request — TCP handshake / TLS per call. Disastrous for latency.
• Holding a DB connection across a long external HTTP call. Locks up the pool.
• Letting pool sizes exceed downstream capacity — your DB has a max-connections limit; sum of all replicas × pool size must fit.

Graceful shutdown and SIGTERM

k8s sends SIGTERM, then waits terminationGracePeriodSeconds (default 30s), then SIGKILL. Your service should:

1. preStop sleep — sleep 5-10s in the k8s preStop hook so the load balancer notices the pod is going away before SIGTERM arrives.
2. On SIGTERM: stop accepting new connections, mark unhealthy.
3. Drain in-flight requests, with a deadline.
4. For long-running runs (agents): persist state, mark resumable, exit.
5. Close resources: flush logs/traces, close DB connections, flush kafka producers.
6. Exit zero on clean shutdown.

See chapter 11 P10 for the Rust pattern.

Readiness vs liveness

Endpoints:

• /livez — returns 200 if the process can answer HTTP. Don't include DB/Redis checks; you don't want k8s killing pods because Postgres flickered.
• /readyz — returns 200 only if downstreams (DB, Redis, backends) are reachable and the service has loaded config.
• /startupz — returns 200 once init complete.

Kubernetes essentials for this role

You don't need to be a k8s expert, but you should be fluent in:

• Deployments, Pods, Services, Endpoints. The day-to-day primitives.
• HPA (Horizontal Pod Autoscaler) — scale on CPU, memory, or custom metrics (queue depth, p99 latency).
• PDBs (Pod Disruption Budgets) — keep N pods up during voluntary disruption.
• Resource requests/limits. Requests = scheduling guarantee; limits = OOMkill threshold. Set both.
• Node affinity / taints. Pin the inference gateway near GPU nodes if that matters; or pin away from noisy neighbors.
• Secrets, ConfigMaps. Where credentials/config live.
• Service mesh. Istio or Linkerd, for mTLS between services, traffic routing for canaries, retries-at-mesh.