The Python Toolchain

Why a real toolchain, not just notebooks

Notebooks are wonderful for exploring — you'll use them forever. But production data work is code that runs unattended, on a schedule, on a machine that isn't yours. That code has to be reproducible: anyone who checks it out of git should be able to run it and get the same result. Notebooks alone don't give you that. A real project does.

The core problem a toolchain solves is dependency hell. Imagine two projects on your laptop:

If Python installs packages globally — one shared pile for the whole machine — these two projects fight. Installing pandas 2.2 for one breaks the other. You can only ever have one version, so something is always wrong.

The fix is to give each project its own private, isolated set of packages — a virtual environment. Each project carries its exact dependencies; they never collide:

Install uv

uv is a single, very fast tool that manages Python versions, virtual environments, and packages — replacing the old tangle of pip, venv, pyenv, and friends. We install it once, then let it install Python for us.

# macOS / Linux / WSL2 — one line
curl -LsSf https://astral.sh/uv/install.sh | sh

# restart your shell (or open a new terminal), then confirm
uv --version

# let uv install a modern Python for you
uv python install 3.12

Create a project

A "project" is just a folder with a few files that declare what it is and what it needs. uv init scaffolds one for you:

cd ~
uv init mini-griddp-pipeline
cd mini-griddp-pipeline
ls -a

You get a small, conventional layout. Here's what each piece is:

The heart of it is pyproject.toml — a plain text config file (in TOML format) that describes your project. Open it and you'll see something like:

[project]
name = "mini-griddp-pipeline"
version = "0.1.0"
description = "Add your description here"
requires-python = ">=3.12"
dependencies = []          # packages get listed here as you add them

Virtual environments, explained

A virtual environment (or "venv") is a private folder — .venv/ — that holds a copy of Python and only this project's packages. When a venv is active, python means "the Python in .venv" and its installed packages, nothing global. That's the isolation that makes the dependency-hell problem disappear.

The mental model: each project is a sealed box. What you install in one box never leaks into another. Two projects can hold different pandas versions because each has its own box.

# uv creates the venv for you (it also does this automatically on first 'uv add')
uv venv

# the magic command: 'uv run' runs anything INSIDE this project's venv,
# creating/syncing it first if needed — no manual "activate" step to remember
uv run python --version

Adding dependencies

To use a package, you uv add it. This installs it into the venv and records it in pyproject.toml so the project remembers it needs it. Let's add the core data tools:

uv add pandas duckdb polars

# look at what changed
cat pyproject.toml      # dependencies now lists pandas, duckdb, polars
ls                       # a new uv.lock file appeared

Two files now describe your dependencies, and the difference between them matters:

The lockfile is what makes installs reproducible across machines. When a teammate clones your repo and runs uv sync, uv reads uv.lock and installs the identical versions you had — not "whatever's newest today." Same inputs, same environment, every time. Commit pyproject.toml and uv.lock to git; never commit .venv/ (it's rebuildable and large).

A starter map of packages you'll meet constantly:

Write a real script

Time to do something useful. We'll read the data/sales.csv from Chapter 02, count the rows, and total the amount per region. First make sure the data is here:

# bring the CSV into this project (adjust the path if yours lives elsewhere)
mkdir -p data
cp ~/cli-practice/data/sales.csv data/sales.csv   # from Chapter 02

# if you skipped Chapter 02, recreate it:
cat > data/sales.csv <<'EOF'
date,region,amount
2026-06-01,us,120
2026-06-01,eu,80
2026-06-02,us,200
2026-06-02,eu,95
EOF

Now create summarize.py in the project folder with your editor (VS Code, nano, whatever you like):

import pandas as pd

# read the CSV into a dataframe (a table in memory)
df = pd.read_csv("data/sales.csv")

print(f"rows: {len(df)}")

# group by region, sum the amount column, and print each region's total
totals = df.groupby("region")["amount"].sum()
for region, total in totals.items():
    print(f"{region}: {total}")

Run it inside the project's environment:

uv run python summarize.py

rows: 4
eu: 175
us: 320

A taste of good practice

You don't need to master these yet — Courses 3 and 4 cover them fully — but a small taste now builds the right habits. Two cheap upgrades make code easier to read and trust: a docstring (what does this do?) and type hints (what goes in, what comes out?). Refactor the summary into a function:

import pandas as pd


def region_totals(path: str) -> pd.Series:
    """Read a sales CSV and return total amount per region."""
    df = pd.read_csv(path)
    return df.groupby("region")["amount"].sum()


if __name__ == "__main__":
    totals = region_totals("data/sales.csv")
    print(totals)

The -> pd.Series and path: str are type hints: they don't change how the code runs, but they document intent and let tools catch mistakes. Two tools you'll meet soon:

The --dev flag marks these as development dependencies — needed to work on the project, but not to run it in production. Keep it light for now; just know these exist and what they're for.

✓ Check yourself

Answer from memory before opening each one.

1. sales-pipeline needs pandas 1.5 and ml-features needs pandas 2.2. Why can't one global Python serve both, and what exactly does a virtual environment change?

   Show answer

   A global install is one shared pile of packages, and it can hold only one version of pandas — installing 2.2 for one project breaks the other. A virtual environment gives each project its own .venv/ with a private set of packages, so each "sealed box" holds its own version and nothing leaks between them. See Why a real toolchain and Virtual environments.

2. Inside mini-griddp-pipeline, uv add pandas succeeded. You then type python3 summarize.py and get ModuleNotFoundError: No module named 'pandas'. What went wrong, and what is the fix?

   Show answer

   Nothing is broken — pandas was installed into the project's .venv/, and plain python3 is the system Python, which can't see inside that box. Run uv run python summarize.py, which picks the project's environment automatically. The tempting fix is to install pandas globally, which brings back the problem environments exist to solve. See Virtual environments.

3. Six months from now a teammate clones the repo and runs uv sync. pyproject.toml just says pandas, and newer releases have shipped since. Which version do they get, and what would have happened if uv.lock had never been committed?

   Show answer

   They get the identical version you had, because uv sync installs from uv.lock, which records the exact version of pandas and of everything pandas depends on. Without the lockfile, only the loose request in pyproject.toml remains, so they'd get whatever is newest that day — a different environment from yours, and "works on my machine" is back. See Adding dependencies.

4. A teammate commits .venv/ "so nobody has to install anything." Why is that the wrong way to share an environment, and what does the job properly?

   Show answer

   .venv/ is large, machine-specific generated output — it is built for one operating system and one folder location, so it bloats the repo and may not even work elsewhere. The environment is already fully described by two small text files, pyproject.toml and uv.lock; commit those and anyone rebuilds the identical environment with uv sync in seconds. See Adding dependencies and the project layout.

5. In the first version of summarize.py, change groupby("region") to groupby("date") (and nothing else that matters). What does uv run python summarize.py print?

   Show answer

   rows: 4, then 2026-06-01: 200 and 2026-06-02: 295. The row count doesn't depend on the grouping; the sums are now per day — 120 + 80 and 200 + 95. If you got 515 on one line you summed without grouping. See Write a real script.

6. An analyst has a notebook that produces the weekly GPU revenue numbers and asks you to "just schedule it." When is a notebook the right home for that code, and what tells you it has outgrown one?

   Show answer

   A notebook is right for exploring — a person at the keyboard, looking at results, changing their mind. The deciding question is whether the code must run unattended, on a schedule, on a machine that isn't theirs: then anyone who checks it out of git has to get the same result, which needs a project with declared and locked dependencies and a script you can run with one command. Exploration stays in the notebook; the scheduled job becomes a project. See Why a real toolchain.

uv add polars        # already added earlier? uv just confirms it's present

import polars as pl

df = pl.read_csv("data/sales.csv")

# sort by amount descending, take the first row
top = df.sort("amount", descending=True).head(1)
print(top)

uv run python top_sale.py

Takeaway

• A virtual environment gives each project its own packages, so two projects can need different versions of pandas without breaking each other.
• pyproject.toml records the packages you asked for, and uv.lock records the exact versions installed, which is what makes a rebuild identical on another machine.
• Commit pyproject.toml and uv.lock, and never commit .venv, because it can be rebuilt from the lock.
• uv run executes inside the project's environment and syncs it first, so you do not activate anything by hand.
• Type hints and docstrings change nothing at runtime. They document intent and let tools like ruff and a type checker catch mistakes.
• Development tools such as pytest and ruff go in dev dependencies, because the project needs them to be worked on but not to run.

Your code now runs the same anywhere. Next you give it real databases to talk to. → Docker & Local Databases

Further reading

• Working on projects — uv documentation, Astral. init, add, lock, sync and run, the commands used in this chapter.
• Locking and syncing — uv documentation, Astral. What the lockfile pins and when uv re-resolves it.
• PEP 621: Storing project metadata in pyproject.toml — Python Software Foundation. The standard that defines the [project] table and its dependencies field.
• Virtual Environments and Packages — Python documentation. What a virtual environment is and why it isolates packages.
• Ruff documentation — Astral. The linter and formatter named in the quality section.