Git & GitHub
Every line of work a data engineer produces — pipelines, configs, infrastructure, even this site — lives in version control. Git is the time machine that records every change and lets you undo, branch, and collaborate without fear. This lab gets git installed, walks you through committing real work, and pushes your first repository to GitHub. Type every command; the muscle memory is the point.
You need two things. (1) Install git: macOS — run git --version and accept the developer-tools prompt, or brew install git; Linux/WSL2 — sudo apt update && sudo apt install git. (2) Create a free GitHub account at github.com/signup — you'll need it for the GitHub section. Keep this page beside your terminal and run each command as you read.
Why version control
Picture the folder of a project that grew without version control: analysis.py, analysis_v2.py, analysis_final.py, analysis_final_v3_REALLY.py. Nobody remembers what changed between them, two people overwrite each other's edits, and the version that actually worked got deleted last Tuesday. Version control removes all of this pain.
Git gives you three superpowers:
| Superpower | What it means for you |
|---|---|
| A time machine | Every saved change is a snapshot you can return to. Broke something? Roll back. Want to know what a file looked like last month? It's there. |
| Collaboration without chaos | Many people work on the same project on separate branches, then merge — no emailing zip files, no overwriting each other. |
| "Everything is code" | Pipelines, configs, and infrastructure all become text files in a repo. That's what makes a data platform reproducible — the theme from Chapter 00. |
Git is the tool that runs on your laptop and records history — it works fully offline. GitHub is a website that hosts git repositories so you can back them up, share them, and collaborate. You can use git with no GitHub at all; GitHub just gives your repos a home in the cloud. Keep the distinction clear and the rest follows.
Install & configure
First confirm git is installed, then tell it who you are — every snapshot you make is stamped with your name and email, so do this once before your first commit.
git --version # confirm git is installed
git config --global user.name "Your Name" # who you are (use your real name)
git config --global user.email "you@example.com" # use your GitHub email
git config --global init.defaultBranch main # name the first branch "main"
git config --list # review what's now setgit --version prints something like git version 2.43.0 (any 2.x is fine). After git config --list you should see your user.name and user.email in the list. If the name or email is wrong, just re-run the git config --global line — the last value wins.
Your first repo
Git has three areas, and understanding them is 80% of "getting" git. A change moves through them in order: you edit files in your working directory, you stage the ones you want to save, then you commit the staged set as a permanent snapshot.
Let's make a repository and put a file under version control. Type these in order:
cd ~
mkdir git-practice
cd git-practice
git init # turn this folder into a git repo (creates .git/)
echo "# Git Practice" > README.md # create a file in the working directory
git status # see what git noticesgit init prints "Initialized empty Git repository in …/git-practice/.git/". Then git status reports you are On branch main with README.md listed under Untracked files — git sees it but isn't tracking it yet. That's the working directory talking.
Now stage the file and commit it — moving it through all three areas:
git add README.md # stage it (working dir → staging area)
git status # README.md is now "Changes to be committed"
git commit -m "Add project README" # snapshot it (staging → history)After git add, git status shows README.md in green under "Changes to be committed." After git commit, git prints a line like [main (root-commit) a1b2c3d] Add project README — that short code is your commit's ID. Run git status again and it says "nothing to commit, working tree clean": everything on disk matches the last snapshot.
git add . stages every changed file in the current folder at once. Handy, but glance at git status first so you don't accidentally stage something you didn't mean to (like a giant data file — more on that in .gitignore).
Seeing history
The whole point of committing is that you can look back. Make one more change so there's a history to explore:
echo "A repo for practicing git." >> README.md # add a line
git diff # what changed since the last commit?
git add README.md
git commit -m "Describe the repo in the README"
git log --oneline # the history, one commit per linegit diff shows your new line prefixed with a green +. After the commit, git log --oneline lists two commits, newest first, each with its short ID and message. This is your time machine's index — every line is a point you could return to.
Four commands cover almost every day of git use:
| Command | Answers the question |
|---|---|
git status | What's changed, and what's staged right now? |
git diff | What exactly did I change (line by line) since the last commit? |
git add <file> | Stage this change for the next commit. |
git commit -m "…" | Snapshot the staged changes with a message. |
git log --oneline | What's the history of this project? |
Write it in the imperative mood — "Add login form," not "Added" or "Adds" (it reads as "this commit will… add login form"). Keep the first line under ~50 characters, and explain the why, not the what — the diff already shows what changed, so use the message for the reason: "Fix off-by-one so last row isn't dropped" beats "changed loop". Future-you reading git log at 2am will be grateful.
Branches & merging
A branch lets you work on something without touching the known-good version. You branch off main, experiment freely, and only merge back when it works. Mentally, a branch is just a movable pointer to a commit — making a new commit slides the pointer forward.
Walk through it: create a branch, make a change, commit it there, then merge it back into main.
git branch # list branches — the * marks where you are (main)
git switch -c add-license # create AND switch to a new branch "add-license"
echo "MIT License" > LICENSE # do some work on the branch
git add LICENSE
git commit -m "Add MIT license file"
git switch main # hop back to main — LICENSE disappears from disk!
git merge add-license # bring the branch's work into main
ls # LICENSE is back, now part of mainAfter git switch main, ls does not show LICENSE — it only exists on the branch, proof that branches are truly separate. After git merge add-license, git reports "Fast-forward" and LICENSE reappears. A fast-forward simply slides main's pointer up to the branch, because main hadn't moved on its own.
Once a branch is merged, you can delete it with git branch -d add-license — the commits stay in history; you're just removing the now-redundant label.
GitHub: putting your repo in the cloud
So far everything lives only on your laptop. GitHub is the remote — a copy of your repo in the cloud that you push to and pull from. Here's the loop you'll live in every day:
On GitHub, click New repository, name it git-practice, leave it empty (no README — you already have one), and create it. GitHub shows you a URL; connect your local repo to it:
# use the URL GitHub shows you (HTTPS shown here)
git remote add origin https://github.com/YOUR-USERNAME/git-practice.git
git remote -v # confirm the remote named "origin" is set
git push -u origin main # push main to GitHub; -u remembers it for next timegit prints upload progress and a line ending in main -> main. Refresh the GitHub page and your README.md and LICENSE are there, with your commit history under the commits tab. The -u flag means from now on you can just type git push and git pull with no extra arguments.
Over HTTPS, GitHub will prompt for credentials on your first push. Modern GitHub does not accept your account password here — use a Personal Access Token as the password (or install the gh CLI and run gh auth login, which handles it for you). Once authenticated, your machine remembers it.
To get an existing GitHub repo onto a new machine, you don't init — you clone it, which downloads the whole repo with its full history and remote already wired up:
git clone https://github.com/YOUR-USERNAME/git-practice.git.gitignore — what NOT to commit
Not everything belongs in version control. A .gitignore file lists patterns git should pretend don't exist — they'll never be staged or committed. Get this right and you avoid three classic disasters:
- Secrets — API keys and passwords in a
.envfile. Commit one once and it's in your history forever, even if you delete it later. - Data files — raw CSVs, databases, and exports bloat the repo and don't belong in code history.
- Virtualenvs & caches — generated folders like
.venv/and__pycache__/are rebuilt from your code; committing them is noise.
A committed password lives in your git history permanently — pushing it to a public GitHub repo means treating it as leaked, even after you delete the file. Add .env to .gitignore before your first commit, and never paste credentials directly into tracked files.
Create a .gitignore at the repo root. Here's a solid starter for the data work in this course:
# secrets — never commit these
.env
# python caches & virtual environments
__pycache__/
.venv/
# local data & databases — keep them out of git
data/
*.duckdbCreate a folder named data/ and a file .env in your repo, then run git status. Neither appears in the output — git is ignoring them exactly as intended. The .gitignore file itself should be committed, so everyone working on the repo shares the same rules.
Apply it: your mini-griddp repo
Time to use this for real. Open the mini-griddp project you started in Course 1, give it a .gitignore, commit it, and push the whole thing to GitHub.
cd ~/mini-griddp # your project from Course 1
git init # if it isn't a repo yet
# create the .gitignore (paste the starter from the section above)
cat > .gitignore <<'EOF'
.env
__pycache__/
.venv/
data/
*.duckdb
EOF
git add . # stage everything NOT ignored
git status # sanity-check: no .env, no data/, no .venv/
git commit -m "Add project files and .gitignore"
# create an empty "mini-griddp" repo on GitHub first, then:
git remote add origin https://github.com/YOUR-USERNAME/mini-griddp.git
git push -u origin mainThe git status before committing lists your real project files but not .env, data/, or .venv/. After the push, open github.com/YOUR-USERNAME/mini-griddp in your browser — your project is there, safely backed up, with no secrets or data files in sight. That's a real repository you could hand to a teammate.
✓ Check yourself
- Can you name the three areas and describe how a change moves through them?
- Can you explain the difference between git and GitHub?
- Do you know the daily loop:
pull → edit → add → commit → push? - Could you list three things that should go in a
.gitignoreand why?
Exercise — Branch, edit, commit, merge, push
In any repo you've made: create a branch, change the README on it, commit, merge back to main, and push to GitHub. Try it from memory before peeking.
git switch -c update-readme # 1) new branch
echo "Updated by me." >> README.md # 2) change the README
git add README.md
git commit -m "Add a note to the README" # 3) commit on the branch
git switch main # 4) back to main
git merge update-readme # bring the change in
git branch -d update-readme # tidy up the merged branch
git push # 5) push main to GitHubIf you reached for switch -c, committed on the branch, and merged without hesitating, the core git loop has landed. Everything else — rebasing, pull requests, resolving conflicts — is vocabulary you'll add as you need it.
Next
Your work is now tracked, reversible, and backed up in the cloud. Next we install the language every pipeline is written in and a modern tool to manage it cleanly. → The Python Toolchain