Docker Deeper

Image vs container, in one breath

One distinction underpins everything in this chapter, so let's nail it. An image is a frozen, read-only template — a packaged filesystem plus the metadata that says "run this command when you start." A container is a running (or stopped) instance of that image — a live process with its own isolated filesystem layered on top.

docker run --rm python:3.12-slim python -c "print('hello from a container')"

Writing a Dockerfile — building your own image

A Dockerfile is a plain-text recipe: a list of instructions Docker follows, top to bottom, to assemble an image. Let's build one for a tiny Python "pipeline." First, the two files our image will contain.

cat > requirements.txt <<'EOF'
requests==2.32.3
EOF

cat > pipeline.py <<'EOF'
print("[pipeline] starting")
nums = [120, 80, 200, 95]
total = sum(nums)
print(f"[pipeline] processed {len(nums)} records, total = {total}")
print("[pipeline] done")
EOF

Now the Dockerfile. Each instruction is purposeful — read the comments:

# FROM: the base image we build on top of. "slim" = small Debian + Python.
FROM python:3.12-slim

# WORKDIR: set (and create) the working directory inside the image.
# Every later COPY/RUN/CMD runs relative to here.
WORKDIR /app

# COPY dependencies FIRST (before the app code) — see the next section for why.
COPY requirements.txt .

# RUN: execute a command at BUILD time and bake the result into the image.
RUN pip install --no-cache-dir -r requirements.txt

# Now copy the rest of the application code.
COPY pipeline.py .

# CMD: the default command run when a container STARTS (at run time).
CMD ["python", "pipeline.py"]

Build the image (the -t gives it a name/tag), then run a container from it:

docker build -t mypipeline:v1 .     # the trailing "." = build context (this folder)
docker run --rm mypipeline:v1

Layers & the build cache

Here's the idea that makes Docker fast once you understand it: every instruction in a Dockerfile creates a layer, and Docker caches each layer. On a rebuild, Docker reuses a cached layer as long as nothing it depends on changed. The moment one instruction's inputs change, that layer and every layer after it are rebuilt.

This is why we copied requirements.txt and ran pip install before copying the app code. Your code changes dozens of times a day; your dependencies rarely do. By ordering the cheap-but-frequent change (code) after the expensive-but-rare one (installing packages), most rebuilds skip the slow pip install entirely. Prove it to yourself:

# Change only the app code, not the dependencies
echo 'print("[pipeline] v2 — extra line")' >> pipeline.py

docker build -t mypipeline:v2 .

Two more habits keep images small and builds clean:

.git
__pycache__/
*.pyc
.env
data/
*.md

Multi-stage builds — ship a lean runtime

Sometimes building your app needs heavy tools (compilers, build dependencies) that the running app doesn't. Shipping those tools bloats the image and widens its attack surface. A multi-stage build uses one stage to build and a second, clean stage to run — copying only the finished artifacts across.

# ---- Stage 1: builder — has the tools to install dependencies ----
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
# Install packages into a self-contained folder we can copy out
RUN pip install --no-cache-dir --target=/deps -r requirements.txt

# ---- Stage 2: runtime — lean image, only what's needed to RUN ----
FROM python:3.12-slim
WORKDIR /app
# Copy ONLY the installed packages from the builder stage
COPY --from=builder /deps /usr/local/lib/python3.12/site-packages
COPY pipeline.py .
CMD ["python", "pipeline.py"]

The final image contains your code and your dependencies — but none of the build-time scratch work. For a Python script the savings are modest; for compiled languages or apps with build toolchains, multi-stage can shrink an image from gigabytes to tens of megabytes. The pattern is the same everywhere: build fat, ship thin.

Compose, deeper — wiring services together

A single container is rarely a whole system. A real data stack is several containers — a database, a script, maybe a scheduler — that must find and talk to each other. Docker Compose describes that whole stack in one YAML file and brings it up with one command. You met Compose in Course 2; now we use its connective tissue.

Named volume vs bind mount trips people up, so hold both in mind:

Image registries — push & pull

You built mypipeline:v1 locally — but how does a teammate or a production server get it? Through a registry: a server that stores and serves images. Docker Hub is the default public one; GHCR (GitHub Container Registry) is popular because it sits next to your code. docker pull python:3.12-slim already used a registry — you were pulling from Docker Hub.

You publish an image so the exact bytes you tested run unchanged elsewhere — no rebuilding, no "works on my machine." The flow is tag → log in → push:

# Tag the local image with a registry path: registry/namespace/name:tag
docker tag mypipeline:v1 ghcr.io/yourname/mypipeline:v1

docker login ghcr.io                       # authenticate once
docker push ghcr.io/yourname/mypipeline:v1 # upload it

# Anyone (with access) can now run the very same image:
docker pull ghcr.io/yourname/mypipeline:v1

Hands-on: a script that talks to Postgres

Now the payoff. We'll run a two-service stack — our Python pipeline and a Postgres database — on a shared network, and have the pipeline connect to Postgres by service name. This is a miniature of every data platform you'll build.

First, update the pipeline to actually connect to a database. Replace pipeline.py and add the Postgres driver:

cat > requirements.txt <<'EOF'
psycopg[binary]==3.2.1
EOF

cat > pipeline.py <<'EOF'
import os, time, psycopg

# "db" is the Compose SERVICE NAME — that's the hostname we connect to.
dsn = (
    f"host={os.environ['DB_HOST']} "
    f"dbname={os.environ['POSTGRES_DB']} "
    f"user={os.environ['POSTGRES_USER']} "
    f"password={os.environ['POSTGRES_PASSWORD']}"
)

print("[pipeline] connecting to Postgres...")
with psycopg.connect(dsn) as conn:
    with conn.cursor() as cur:
        cur.execute("CREATE TABLE IF NOT EXISTS sales (region TEXT, amount INT);")
        cur.execute("INSERT INTO sales VALUES ('us', 120), ('eu', 80), ('us', 200);")
        cur.execute("SELECT region, SUM(amount) FROM sales GROUP BY region ORDER BY region;")
        for region, total in cur.fetchall():
            print(f"[pipeline] {region} total = {total}")
print("[pipeline] done")
EOF

Now the Compose file. Note how depends_on waits for Postgres to be healthy, and how the pipeline reaches the database at host db — the service name:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    volumes:
      - pgdata:/var/lib/postgresql/data   # named volume → data persists
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d warehouse"]
      interval: 3s
      timeout: 3s
      retries: 10

  pipeline:
    build: .                              # build from our Dockerfile
    environment:
      DB_HOST: db                         # ← the service name above
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    depends_on:
      db:
        condition: service_healthy        # wait until Postgres is READY

volumes:
  pgdata:                                 # declare the named volume

Bring up the stack. --build rebuilds the pipeline image first:

docker compose up --build

# In another terminal (or after it exits), tidy up.
# down       = stop & remove containers/network (named volume survives)
# down -v    = also delete the named volume (wipes the database)
docker compose down

✓ Check yourself

• Can you explain, in one sentence, the difference between an image and a container?
• Why do we COPY requirements.txt and install deps before copying the app code?
• How does one container reach another in a Compose stack — by IP, or by something else?
• What's the difference between a named volume and a bind mount, and which one would you use for a database?

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
CMD ["python", "app.py"]

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d warehouse"]
      interval: 3s
      retries: 10

  app:
    build: .
    environment:
      DB_HOST: db
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    depends_on:
      db:
        condition: service_healthy

volumes:
  pgdata:

# requirements.txt -> psycopg[binary]==3.2.1
import os, psycopg
dsn = (f"host={os.environ['DB_HOST']} dbname={os.environ['POSTGRES_DB']} "
       f"user={os.environ['POSTGRES_USER']} password={os.environ['POSTGRES_PASSWORD']}")
with psycopg.connect(dsn) as conn, conn.cursor() as cur:
    cur.execute("SELECT 'connected to ' || current_database();")
    print(cur.fetchone()[0])

Next

You can build images and compose services that find each other by name — now let's scale that up into a real local data platform with several cooperating services. → Wiring the Mini-Platform