Deployment & Ops

Multi-stage Docker builds

The standard pattern for Rust images: build in a heavy image, copy the binary into a minimal runtime image. Result: ~20MB images that boot in <1s.

# syntax=docker/dockerfile:1.6
FROM rust:1.78-slim AS build
WORKDIR /app
# Cache deps separately from source
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main(){}" > src/main.rs && \
    cargo build --release && rm -rf src
COPY src ./src
RUN cargo build --release --bin gateway

FROM gcr.io/distroless/cc-debian12 AS runtime
COPY --from=build /app/target/release/gateway /usr/local/bin/gateway
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/gateway"]

Things to mention in interviews:

• Layer caching — copying Cargo.toml/Cargo.lock first builds a deps-only layer that's reused across source changes. Huge CI win.
• Distroless base — no shell, no package manager, smaller attack surface. cc variant has glibc.
• Static binary alternative — build with musl (--target x86_64-unknown-linux-musl) for a fully static binary that runs on scratch. Worth doing for the smallest production image.
• cargo-chef — third-party tool that splits Rust builds for better Docker caching; worth knowing by name.

Release vs debug builds

Never deploy debug builds. Performance differences are 10-100x for compute-bound code. Always run perf tests against release.

[profile.release]
opt-level = 3
lto = "fat"            # whole-program optimization
codegen-units = 1      # single CGU enables more inlining; slower compile
panic = "abort"        # smaller binaries, can't unwind across FFI
strip = true           # strip symbols
debug = 1              # keep line-number debug info for backtraces in prod

Tradeoffs:

• lto = "fat" — big perf win; longer compile.
• codegen-units = 1 — better inlining; slower compile; you keep it on for prod releases, not dev.
• panic = "abort" — saves binary size, no unwinding overhead, but you lose catch_unwind. Most services don't need it.
• debug = 1 — keep just enough debug info for symbolic stack traces. Pair with a separate --strip=symbols step and external symbol upload (sentry, datadog).

PGO and LTO

Profile-Guided Optimization: compile instrumented binary, run representative load, recompile using the profile. Typical wins 5-15% on throughput. Tooling: cargo-pgo.

# 1. instrument
RUSTFLAGS="-Cprofile-generate=/tmp/pgo" cargo build --release

# 2. run representative workload
./target/release/gateway &
# ... load test for ~10 min ...
killall gateway

# 3. merge profile data
llvm-profdata merge -o /tmp/pgo.profdata /tmp/pgo

# 4. recompile using profile
RUSTFLAGS="-Cprofile-use=/tmp/pgo.profdata" cargo build --release

Worth doing for the inference gateway; not worth doing for everything. Like jemalloc: benchmark, then claim.

Blue/green deploys

Two production environments (blue, green). Traffic on blue. Deploy new version to green, smoke-test, flip the load balancer. Old blue stays warm; if issue, flip back.

• Zero-downtime cutover.
• Fast rollback (LB flip is seconds).
• Cost — 2x infra during the cutover period.
• State coordination — both environments writing to the same DB. Schema migrations must be compatible across both versions during the window.

For stateful services (orchestrators with active runs), blue/green is harder; canaries (next) tend to be better.

Canaries

Deploy the new version to a small fraction of capacity (e.g., 1% then 10% then 50% then 100%). Monitor SLOs at each step. Roll back on regression.

• Traffic split by header, by user-id hash, or by random sampling.
• Compare metrics between canary and baseline pods. Statistical comparison preferred (e.g., the canary's p99 vs baseline's p99 with confidence intervals).
• Tooling: Argo Rollouts, Flagger, Spinnaker. Or homegrown — a service mesh (Istio, Linkerd) can route by weight.
• Auto-rollback if canary metrics breach. Half-cooked canaries that need humans to notice problems aren't safer.

Feature flags

Decouple deploy from release. Ship code dark; flip flag to enable.

• Boolean flags — on/off per environment.
• Percentage rollout — enable for X% of users.
• Targeted rollout — enable for specific user IDs / tenants.
• Kill switches (covered in 09) are a subset.

Tooling: LaunchDarkly, Flagsmith, Unleash, or a homegrown service. In Rust, statsig-rust, launchdarkly-server-sdk, or your own config-watcher pulling from Postgres.

Rollback strategy

The bar for production: any change can be rolled back in <5 minutes without coordination.

• Code: Redeploy previous image. Pre-pull images on the cluster so rollback is fast.
• Feature flag: Flip the flag. Seconds.
• Schema changes: Always backward-compatible. Expand-then-contract: add new fields/columns, deploy code that handles both old and new, only later remove the old.
• Stateful migrations: Migrations are forward-only; rolling back code while the DB has the new schema must work.
• Drill rollback. Once a quarter, deliberately roll back something in staging. Find the broken assumption while you're not on fire.

Connection pooling

Three pools matter most:

• Database (Postgres): sqlx, deadpool-postgres, tokio-postgres. Size = expected concurrent DB ops per replica. 10-50 typical.
• Redis: fred, deadpool-redis. Size = concurrent Redis ops per replica. 20-100 typical for hot paths.
• HTTP (to backends): reqwest::Client reuses HTTP/2 connections. One Client per process; .pool_max_idle_per_host(N) tunes.

Anti-patterns:

• Creating a new client per request — TCP handshake / TLS per call. Disastrous for latency.
• Holding a DB connection across a long external HTTP call. Locks up the pool.
• Letting pool sizes exceed downstream capacity — your DB has a max-connections limit; sum of all replicas × pool size must fit.

Graceful shutdown and SIGTERM

k8s sends SIGTERM, then waits terminationGracePeriodSeconds (default 30s), then SIGKILL. Your service should:

1. preStop sleep — sleep 5-10s in the k8s preStop hook so the load balancer notices the pod is going away before SIGTERM arrives.
2. On SIGTERM: stop accepting new connections, mark unhealthy.
3. Drain in-flight requests, with a deadline.
4. For long-running runs (agents): persist state, mark resumable, exit.
5. Close resources: flush logs/traces, close DB connections, flush kafka producers.
6. Exit zero on clean shutdown.

See chapter 11 P10 for the Rust pattern.

Readiness vs liveness

Endpoints:

• /livez — returns 200 if the process can answer HTTP. Don't include DB/Redis checks; you don't want k8s killing pods because Postgres flickered.
• /readyz — returns 200 only if downstreams (DB, Redis, backends) are reachable and the service has loaded config.
• /startupz — returns 200 once init complete.

Kubernetes essentials for this role

You don't need to be a k8s expert, but you should be fluent in:

• Deployments, Pods, Services, Endpoints. The day-to-day primitives.
• HPA (Horizontal Pod Autoscaler) — scale on CPU, memory, or custom metrics (queue depth, p99 latency).
• PDBs (Pod Disruption Budgets) — keep N pods up during voluntary disruption.
• Resource requests/limits. Requests = scheduling guarantee; limits = OOMkill threshold. Set both.
• Node affinity / taints. Pin the inference gateway near GPU nodes if that matters; or pin away from noisy neighbors.
• Secrets, ConfigMaps. Where credentials/config live.
• Service mesh. Istio or Linkerd, for mTLS between services, traffic routing for canaries, retries-at-mesh.

✓ Check yourself

Answer from memory before opening each one.

1. CI recompiles every dependency on every commit, taking twenty minutes, though only gateway code changed. What does the Dockerfile do wrong, and what is the fix?

   Show answer

   It copies the source before building, so any code change invalidates the dependency layer. Copy the manifest and lock file first and build the dependencies in their own layer, which the cache reuses until they change. cargo-chef automates the split. See Multi-stage Docker builds.

2. You must roll out a new orchestrator version while thousands of agent runs are in flight. Blue/green or canary, and why?

   Show answer

   Canary. Blue/green runs two full environments against one database, which is harder when runs hold state and both versions must tolerate the same schema; it also doubles cost during the cutover. A canary moves a small share of capacity first and compares SLOs against the baseline. See Blue/green and Canaries.

3. A column that running code reads must be renamed. Lay out the deploys so that every step can be rolled back.

   Show answer

   Expand then contract. Add the new column, deploy code that handles both, backfill, switch reads to the new column, and drop the old one only in a later release. Each step stays backward compatible, so rolling back the code never meets a schema it cannot read. See Rollback strategy.

4. Postgres is unreachable for twenty seconds, and every gateway pod restarts, which makes recovery slower. What was wrong with the probes?

   Show answer

   The liveness probe checked dependencies, so a database blip told Kubernetes to kill healthy processes. Liveness should test only that the process responds. Dependency checks belong in readiness, which removes the pod from endpoints without restarting it. See Readiness vs liveness.

5. There are 40 replicas, each with a connection pool of 50, and Postgres allows 1,000 connections. What happens, and what do you change?

   Show answer

   The pools can ask for 2,000 connections, so under load the database refuses new ones or the pooler saturates. Size the sum of replicas times pool size to fit under the limit, for example by shrinking pools or using a pooler in front. See Connection pooling.

6. A pod gets SIGTERM with a 30 second grace period while agent runs are mid-flight, some needing minutes. What does the shutdown sequence do with them?

   Show answer

   A preStop sleep gives the load balancer time to drop the pod. On SIGTERM it stops accepting work, drains what can finish within the deadline, and persists the state of long runs as resumable so another instance continues them. Then it flushes telemetry and exits cleanly. See Graceful shutdown and SIGTERM.

Takeaway

• Build with multi-stage Docker, a dependency layer for caching and a minimal runtime image, and never deploy or benchmark a debug build.
• Release profile settings such as lto, codegen-units=1 and PGO are measured wins that trade longer builds for throughput.
• Prefer canaries for stateful services: shift a small share, compare SLOs with the baseline and roll back automatically.
• Feature flags separate deploy from release, and each flag needs a removal date.
• Any change should roll back in minutes without coordination, which requires expand-then-contract schema changes.
• Size connection pools so replicas times pool fits the downstream limit, and reuse one HTTP client per process.
• Liveness tests the process only, readiness tests dependencies, and shutdown is preStop sleep, stop accepting, drain, persist and exit.

With the operating story set, the next chapter collects the vocabulary around Rust services, networking and ML serving. → Domain context

Further reading

• Building best practices — Docker documentation. Multi-stage builds and layer caching.
• Profiles — The Cargo Book. The release settings for lto, codegen-units, panic and strip.
• Profile-guided Optimization — The rustc book. The exact flags and workflow for PGO.
• Canarying Releases — Google SRE Workbook, 2018. Canary design, sizing and evaluation.
• Liveness, Readiness and Startup Probes — Kubernetes documentation. What each probe does and how failures are handled.
• Pod Lifecycle — Kubernetes documentation. Termination order, grace periods and preStop hooks.
• Feature Toggles — Martin Fowler. Types of toggle and managing their lifetime.