Docker & Local Databases

Why containers

Imagine you write a pipeline on your laptop and it works perfectly. You hand it to a teammate, or push it to a server — and it breaks. A different Python version, a missing system library, a database configured slightly differently. The infamous "but it works on my machine." Containers exist to delete that sentence from your vocabulary.

A container ships your whole environment, not just your code — the operating system bits, the libraries, the exact versions, the config — all bundled together. Wherever Docker runs, your container runs the same way. You're not sending a recipe and hoping the other kitchen has the right ingredients; you're sending the whole sealed kitchen.

For a data engineer this is a superpower. Need Postgres 16? Redis? Kafka? A specific Spark version? You don't install and configure each one by hand on your laptop (and risk wrecking it). You pull a container and it just runs — the same on your machine, your teammate's, and the production server. That's the reproducibility principle from Chapter 00, made concrete.

Images vs containers

Two words you'll hear constantly, and beginners mix them up. The distinction is exactly like a class vs an instance in programming:

• An image is the blueprint — a frozen, read-only template (e.g. "Postgres 16"). It sits in a registry (Docker Hub by default) until you pull it.
• A container is a running instance of an image. You can start many containers from one image, just like many objects from one class.

The core verbs. Try them with a tiny database to feel the loop (we'll use the friendlier Compose approach for real work shortly):

docker pull postgres:16      # download the image from the registry
docker run -d --name demo \
  -e POSTGRES_PASSWORD=secret postgres:16   # start a container in the background (-d)
docker ps                    # list RUNNING containers (their id, name, status, ports)
docker logs demo             # see what the container printed
docker stop demo             # stop it
docker rm demo               # remove the stopped container

Docker Compose

Those docker run lines get long fast — environment variables, ports, volumes, names. And a real platform isn't one container; it's several that need to talk to each other. Typing all that by hand every time is error-prone and, worse, not written down anywhere.

Docker Compose fixes this. You declare your whole stack — every service, its image, its settings — in one file called docker-compose.yml. Then a single command brings the entire stack up. The file is the documentation, and it lives in git.

Run Postgres (the OLTP "source system")

Let's run a real database. Postgres is the classic open-source relational database — the kind of OLTP (Online Transaction Processing) system that powers the apps generating your data: orders, signups, rentals. In our world it stands in for the source system you extract data from.

Create a project folder and a Compose file:

mkdir -p ~/mini-griddp && cd ~/mini-griddp

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: griddp
      POSTGRES_PASSWORD: griddp_pw
      POSTGRES_DB: source
    ports:
      - "5432:5432"          # map host port 5432 -> container port 5432
    volumes:
      - pgdata:/var/lib/postgresql/data   # named volume keeps data across restarts

volumes:
  pgdata:

Bring it up and check it:

docker compose up -d        # start Postgres in the background
docker compose ps           # confirm it's running

Now open a SQL shell inside the container using docker compose exec, which runs a command in a running service. The built-in psql client is already there:

docker compose exec postgres psql -U griddp -d source

You're now at a source=# prompt — that's Postgres waiting for SQL. Type these:

CREATE TABLE customers (
    id    INT PRIMARY KEY,
    name  TEXT,
    city  TEXT
);

INSERT INTO customers (id, name, city) VALUES
    (1, 'Ada',  'Austin'),
    (2, 'Grace','Seattle'),
    (3, 'Linus','Helsinki');

SELECT * FROM customers;

 id | name  |   city
----+-------+----------
  1 | Ada   | Austin
  2 | Grace | Seattle
  3 | Linus | Helsinki
(3 rows)

DuckDB (the OLAP "warehouse")

Postgres runs as a server — a separate process you connect to. DuckDB works differently: it's in-process, meaning it runs inside your program (or CLI) with no server to start, no ports, no container. Think of it as "SQLite for analytics." It's built for OLAP (Online Analytical Processing) — scanning and aggregating lots of rows fast — so we treat it as your local data warehouse.

Its party trick: it queries CSV and Parquet files directly with SQL, no loading step. Install it (in your Chapter 04 project, or the standalone CLI):

uv add duckdb            # add the Python library to your project
# OR install the standalone CLI:
#   macOS/Linux:  brew install duckdb

Make a small CSV (the same kind of file you wrangled in Chapter 02) and query it straight from SQL:

cat > sales.csv <<'EOF'
date,region,amount
2026-06-01,us,120
2026-06-01,eu,80
2026-06-02,us,200
2026-06-02,eu,95
EOF

duckdb        # start the DuckDB CLI (or run python and import duckdb)

-- read the CSV directly — no import, no schema setup
SELECT region, SUM(amount) AS total
FROM 'sales.csv'
GROUP BY region
ORDER BY total DESC;

┌─────────┬───────┐
│ region  │ total │
├─────────┼───────┤
│ us      │   320 │
│ eu      │   175 │
└─────────┴───────┘

So when do you use which? They're complementary, not competitors:

The starter docker-compose.yml

Here's the clean, complete starter file for your mini-griddp project. It's just the Postgres service for now — but this file is the backbone of your platform, and you'll add services to it (a warehouse, an orchestrator, and more) as the curriculum grows.

# mini-griddp local stack
# Bring it up:   docker compose up -d
# Tear it down:  docker compose down
services:
  postgres:
    image: postgres:16
    container_name: griddp_postgres
    environment:
      POSTGRES_USER: griddp
      POSTGRES_PASSWORD: griddp_pw
      POSTGRES_DB: source
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:          # named volume: your database survives `down` and `up`

Volumes & data persistence

Containers are designed to be stateless and disposable — you can throw one away and start a fresh one anytime. That's great for the software, but terrible for your data: when a container is removed, everything written inside it vanishes with it. Run docker compose down without a volume and your tables are gone.

A named volume solves this. It's a storage area Docker manages outside any single container. Our file maps the volume pgdata to the folder where Postgres keeps its data (/var/lib/postgresql/data), so the data lives in the volume, not the container. Tear the container down, bring it back up — the volume reattaches and your tables are still there.

This split — disposable compute in containers, durable state in volumes (and later, cloud storage) — is one of the most important patterns in modern data engineering. Keep it in mind.

✓ Check yourself

• Can you explain the difference between an image and a container (hint: class vs instance)?
• Can you bring a Compose stack up in the background and check its status?
• Do you know why a named volume matters — what's lost without one?
• Can you say which database is OLTP and which is OLAP, and what each stands in for?

cd ~/mini-griddp
docker compose up -d
docker compose exec postgres psql -U griddp -d source

CREATE TABLE gpu_rentals (
    id        INT PRIMARY KEY,
    gpu_model TEXT,
    hours     NUMERIC
);

INSERT INTO gpu_rentals (id, gpu_model, hours) VALUES
    (1, 'H100', 12.5),
    (2, 'A100',  8.0),
    (3, 'H100',  4.5);

SELECT SUM(hours) AS total_hours FROM gpu_rentals;

Next

You have a real database running and an analytics engine ready. Now let's connect them: pull data out of Postgres, transform it, and land it in DuckDB — your first end-to-end pipeline. → Your First Pipeline