Docker Deeper

Image vs container, in one breath

One distinction underpins everything in this chapter, so let's nail it. An image is a frozen, read-only template — a packaged filesystem plus the metadata that says "run this command when you start." A container is a running (or stopped) instance of that image — a live process with its own isolated filesystem layered on top.

docker run --rm python:3.12-slim python -c "print('hello from a container')"

Writing a Dockerfile — building your own image

A Dockerfile is a plain-text recipe: a list of instructions Docker follows, top to bottom, to assemble an image. Let's build one for a tiny Python "pipeline." First, the two files our image will contain.

cat > requirements.txt <<'EOF'
requests==2.32.3
EOF

cat > pipeline.py <<'EOF'
print("[pipeline] starting")
nums = [120, 80, 200, 95]
total = sum(nums)
print(f"[pipeline] processed {len(nums)} records, total = {total}")
print("[pipeline] done")
EOF

Now the Dockerfile. Each instruction is purposeful — read the comments:

# FROM: the base image we build on top of. "slim" = small Debian + Python.
FROM python:3.12-slim

# WORKDIR: set (and create) the working directory inside the image.
# Every later COPY/RUN/CMD runs relative to here.
WORKDIR /app

# COPY dependencies FIRST (before the app code) — see the next section for why.
COPY requirements.txt .

# RUN: execute a command at BUILD time and bake the result into the image.
RUN pip install --no-cache-dir -r requirements.txt

# Now copy the rest of the application code.
COPY pipeline.py .

# CMD: the default command run when a container STARTS (at run time).
CMD ["python", "pipeline.py"]

Build the image (the -t gives it a name/tag), then run a container from it:

docker build -t mypipeline:v1 .     # the trailing "." = build context (this folder)
docker run --rm mypipeline:v1

Layers & the build cache

Here's the idea that makes Docker fast once you understand it: every instruction in a Dockerfile creates a layer, and Docker caches each layer. On a rebuild, Docker reuses a cached layer as long as nothing it depends on changed. The moment one instruction's inputs change, that layer and every layer after it are rebuilt.

This is why we copied requirements.txt and ran pip install before copying the app code. Your code changes dozens of times a day; your dependencies rarely do. By ordering the cheap-but-frequent change (code) after the expensive-but-rare one (installing packages), most rebuilds skip the slow pip install entirely. Prove it to yourself:

# Change only the app code, not the dependencies
echo 'print("[pipeline] v2 — extra line")' >> pipeline.py

docker build -t mypipeline:v2 .

Two more habits keep images small and builds clean:

.git
__pycache__/
*.pyc
.env
data/
*.md

Multi-stage builds — ship a lean runtime

Sometimes building your app needs heavy tools (compilers, build dependencies) that the running app doesn't. Shipping those tools bloats the image and widens its attack surface. A multi-stage build uses one stage to build and a second, clean stage to run — copying only the finished artifacts across.

# ---- Stage 1: builder — has the tools to install dependencies ----
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
# Install packages into a self-contained folder we can copy out
RUN pip install --no-cache-dir --target=/deps -r requirements.txt

# ---- Stage 2: runtime — lean image, only what's needed to RUN ----
FROM python:3.12-slim
WORKDIR /app
# Copy ONLY the installed packages from the builder stage
COPY --from=builder /deps /usr/local/lib/python3.12/site-packages
COPY pipeline.py .
CMD ["python", "pipeline.py"]

The final image contains your code and your dependencies — but none of the build-time scratch work. For a Python script the savings are modest; for compiled languages or apps with build toolchains, multi-stage can shrink an image from gigabytes to tens of megabytes. The pattern is the same everywhere: build fat, ship thin.

Compose, deeper — wiring services together

A single container is rarely a whole system. A real data stack is several containers — a database, a script, maybe a scheduler — that must find and talk to each other. Docker Compose describes that whole stack in one YAML file and brings it up with one command. You met Compose in Course 2; now we use its connective tissue.

Named volume vs bind mount trips people up, so hold both in mind:

Image registries — push & pull

You built mypipeline:v1 locally — but how does a teammate or a production server get it? Through a registry: a server that stores and serves images. Docker Hub is the default public one; GHCR (GitHub Container Registry) is popular because it sits next to your code. docker pull python:3.12-slim already used a registry — you were pulling from Docker Hub.

You publish an image so the exact bytes you tested run unchanged elsewhere — no rebuilding, no "works on my machine." The flow is tag → log in → push:

# Tag the local image with a registry path: registry/namespace/name:tag
docker tag mypipeline:v1 ghcr.io/yourname/mypipeline:v1

docker login ghcr.io                       # authenticate once
docker push ghcr.io/yourname/mypipeline:v1 # upload it

# Anyone (with access) can now run the very same image:
docker pull ghcr.io/yourname/mypipeline:v1

Hands-on: a script that talks to Postgres

Now the payoff. We'll run a two-service stack — our Python pipeline and a Postgres database — on a shared network, and have the pipeline connect to Postgres by service name. This is a miniature of every data platform you'll build.

First, update the pipeline to actually connect to a database. Replace pipeline.py and add the Postgres driver:

cat > requirements.txt <<'EOF'
psycopg[binary]==3.2.1
EOF

cat > pipeline.py <<'EOF'
import os, time, psycopg

# "db" is the Compose SERVICE NAME — that's the hostname we connect to.
dsn = (
    f"host={os.environ['DB_HOST']} "
    f"dbname={os.environ['POSTGRES_DB']} "
    f"user={os.environ['POSTGRES_USER']} "
    f"password={os.environ['POSTGRES_PASSWORD']}"
)

print("[pipeline] connecting to Postgres...")
with psycopg.connect(dsn) as conn:
    with conn.cursor() as cur:
        cur.execute("CREATE TABLE IF NOT EXISTS sales (region TEXT, amount INT);")
        cur.execute("INSERT INTO sales VALUES ('us', 120), ('eu', 80), ('us', 200);")
        cur.execute("SELECT region, SUM(amount) FROM sales GROUP BY region ORDER BY region;")
        for region, total in cur.fetchall():
            print(f"[pipeline] {region} total = {total}")
print("[pipeline] done")
EOF

Now the Compose file. Note how depends_on waits for Postgres to be healthy, and how the pipeline reaches the database at host db — the service name:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    volumes:
      - pgdata:/var/lib/postgresql/data   # named volume → data persists
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d warehouse"]
      interval: 3s
      timeout: 3s
      retries: 10

  pipeline:
    build: .                              # build from our Dockerfile
    environment:
      DB_HOST: db                         # ← the service name above
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    depends_on:
      db:
        condition: service_healthy        # wait until Postgres is READY

volumes:
  pgdata:                                 # declare the named volume

Bring up the stack. --build rebuilds the pipeline image first:

docker compose up --build

# In another terminal (or after it exits), tidy up.
# down       = stop & remove containers/network (named volume survives)
# down -v    = also delete the named volume (wipes the database)
docker compose down

✓ Check yourself

Answer from memory before opening each one.

1. A teammate's Dockerfile reads FROM, WORKDIR /app, COPY . ., RUN pip install -r requirements.txt, CMD. Every one-line edit to pipeline.py costs them a full dependency install. Why, and what reordering fixes it?

   Show answer

   Each instruction is a cached layer, and when one layer's inputs change, that layer and every layer after it rebuild. COPY . . includes the code, so any edit invalidates it and the pip install below it. Copy requirements.txt and install first, then copy the code last — the slow layer now only rebuilds when dependencies change. See Layers & the build cache.

2. You run the hands-on stack with docker compose up, let it finish, run docker compose down, then docker compose up again. What totals does the pipeline print for eu and us the second time, why, and which command would have given you 80 and 320 again?

   Show answer

   eu total = 160 and us total = 640. down removes the containers and network, but the named volume pgdata survives, so the sales table still holds the first run's rows and the script inserts the same three again. docker compose down -v deletes the volume too and starts the database clean. See the hands-on and named volumes.

3. The hands-on compose.yaml contains no IP addresses and never declares a network. How does the pipeline find Postgres — and if you renamed the db service to warehouse-db, what else in the file would have to change?

   Show answer

   Compose creates one default network for the stack, joins every service to it, and makes each service reachable by its service name as a DNS hostname. Rename the service and the hostname changes with it, so DB_HOST must become warehouse-db (and the key under depends_on must match the new name). See Service name = hostname.

4. Someone simplifies the pipeline service to a plain depends_on: [db]. The stack now fails with "connection refused" on some runs and works on others. What is the race, and why did the original version not have it?

   Show answer

   A Postgres container counts as started within milliseconds but is not ready to accept connections for a second or two; plain depends_on only orders the starts, so the script sometimes connects too early. The original paired a pg_isready healthcheck with condition: service_healthy, which makes Compose wait until the database is actually usable. See Why the healthcheck matters.

5. A nightly job runs ghcr.io/yourname/mypipeline:latest. It worked yesterday, fails today, and nobody touched the job's configuration. What do you suspect first, and what change prevents a repeat?

   Show answer

   That :latest moved. A tag is a mutable label — whoever pushed last owns it — so the job may be running different bytes than yesterday even though "nothing changed". Deploy explicit version tags (:v1, a date) so the image can only change when someone changes the tag on purpose. See Image registries.

6. A colleague wants every image in the repo converted to a multi-stage build. For this chapter's pure-Python pipeline, is that worth doing? What decides it?

   Show answer

   What decides it is how much build-time tooling the running app does not need. Here the builder and runtime stages share the same slim base and the dependencies are copied across whole, so the saving is modest and the extra Dockerfile complexity is a fair objection. Multi-stage clearly wins when the build needs compilers or toolchains — that is where images shrink from gigabytes to tens of megabytes and the attack surface narrows. See Multi-stage builds.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d warehouse"]
      interval: 3s
      retries: 10

  app:
    build: .
    environment:
      DB_HOST: db
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    depends_on:
      db:
        condition: service_healthy

volumes:
  pgdata:

# requirements.txt -> psycopg[binary]==3.2.1
import os, psycopg
dsn = (f"host={os.environ['DB_HOST']} dbname={os.environ['POSTGRES_DB']} "
       f"user={os.environ['POSTGRES_USER']} password={os.environ['POSTGRES_PASSWORD']}")
with psycopg.connect(dsn) as conn, conn.cursor() as cur:
    cur.execute("SELECT 'connected to ' || current_database();")
    print(cur.fetchone()[0])

Takeaway

• An image is a frozen read-only template and a container is a disposable instance of it, so removing the container leaves the image cached.
• Every Dockerfile instruction is a cached layer, and a change invalidates that layer and every later one, so install dependencies before copying code that changes daily.
• CMD is a default that a command after docker run image replaces, and ENTRYPOINT is fixed with CMD supplying default arguments.
• A multi-stage build compiles in one stage and ships only the result in a clean final stage, which shrinks the image and its attack surface.
• Compose puts every service on one network where each is reachable by its service name as a hostname.
• A container that has started is not necessarily ready, so use a health check with condition: service_healthy or the pipeline races ahead and hits connection refused.
• :latest is a moving label, so tag with explicit versions when a deploy must be reproducible.

You can build images and join services by name, and the next chapter scales that up into a real local data platform. → Wiring the Mini-Platform

Further reading

• Docker: Building best practices — Docker docs, accessed 2026. Layer ordering, base images, .dockerignore and multi-stage builds in one place.
• Docker: Build cache — Docker docs, accessed 2026. How layer caching and invalidation work.
• Docker: Multi-stage builds — Docker docs, accessed 2026. Copying artifacts between stages.
• Docker: Networking in Compose — Docker docs, accessed 2026. The default network and service-name DNS.
• Docker: Control startup order in Compose — Docker docs, accessed 2026. Why depends_on alone does not wait for readiness, and how health checks fix it.