Course 2 · Hands-on

The Command Line

The shell is the cockpit of data engineering — every database, container, server, and pipeline tool is ultimately driven from a text prompt. This lab gets you comfortable moving around, manipulating files, and chaining small commands into something powerful. Type every command; the fluency comes from your fingers.

Setup — do this first

Open your terminal (on Windows, your WSL2 Ubuntu terminal from Chapter 00). That's all you need — the shell is already installed. Keep this page beside the terminal and run each command as you read. If a command isn't found on macOS, install Homebrew (brew.sh) and re-try; most tools here are built in.

What "the command line" actually is

When you open a terminal, a program called a shell is waiting to read what you type. The most common shell is bash (or its cousin zsh, the macOS default — they behave almost identically for our purposes). You type a command, press Enter, the shell runs it, prints output, and waits for the next one. That's the entire loop.

A command has three parts:

ls -l /tmp │ │ │ │ │ └── argument (what to act on) │ └─────── option/flag (how to behave; usually starts with - ) └──────────── command (the program to run)

Why the command line (when GUIs exist)

The shell gives you…Why it matters for data work
ReachRemote servers and containers have no GUI — only a shell. This is how you operate them.
RepeatabilityA command is text you can save, version, and re-run. Clicking is not reproducible.
ComposabilitySmall tools snap together with pipes to do things no single app does.
Speed at scaleRename 10,000 files or scan a million-line log in one line — no scrolling.

Making & moving files

Let's create a small workspace and practice. Type these in order:

terminal
cd ~                       # start at home
mkdir -p cli-practice/data # make nested folders (-p makes parents as needed)
cd cli-practice
touch notes.txt            # create an empty file
echo "hello data" > data/greeting.txt   # write text into a new file
cp data/greeting.txt data/greeting-backup.txt   # copy
mv notes.txt README.txt    # move/rename
ls -R                      # list recursively to see the whole tree
✓ You should see

ls -R shows README.txt and a data/ folder containing greeting.txt and greeting-backup.txt. If mkdir complained, you may have skipped -p — it creates intermediate folders for you.

⚠ rm has no undo

rm file deletes a file; rm -r folder deletes a folder and everything in it — permanently, with no recycle bin. Never run rm -rf on a path you haven't double-checked, and never on / or ~. Pause and read the path before pressing Enter. This habit will save you one day.

Looking inside files

Data engineers stare at files constantly — peeking at a CSV, tailing a log. The toolkit:

terminal
cat data/greeting.txt     # dump the whole file to the screen
head -n 5 bigfile.csv     # first 5 lines (great for peeking at headers)
tail -n 5 bigfile.csv     # last 5 lines
tail -f app.log           # FOLLOW a log live as it grows (Ctrl-C to stop)
wc -l bigfile.csv         # count lines (rows) — your quick "how big is this?"
less bigfile.csv          # scrollable viewer; press q to quit, / to search
Why head/tail matter

You'll often get a multi-gigabyte file you can't open in a spreadsheet. head shows you the columns and a few rows instantly without loading the whole thing — the first move when meeting any new dataset.

Pipes & redirection — where the shell gets powerful

This is the big idea. Two operators let small tools combine:

  • | (pipe) — sends the output of one command as the input of the next.
  • > and >> (redirect) — send output to a file (> overwrites, >> appends).
cat sales.csv │ grep "2026" │ wc -l > count.txt │ │ │ │ read file keep matching count them save result lines only to a file ───────────────────────────────────────────────────────▶ data flows left to right
terminal
# How many lines mention "error" in a log?
cat app.log | grep "error" | wc -l

# Save the first 100 rows of a big CSV to a sample file
head -n 100 big.csv > sample.csv

# Append a line to a file without erasing it
echo "2026-06-21 run ok" >> run.log
The Unix philosophy

Each tool does one thing well; pipes compose them into pipelines. This is the same idea — small stages, data flowing through them — that underlies every data pipeline you'll build later. You're already thinking like a data engineer.

A taste of data wrangling

Let's make a tiny CSV and answer questions about it with only the shell — no spreadsheet, no Python yet.

terminal
# create a small CSV
cat > data/sales.csv <<'EOF'
date,region,amount
2026-06-01,us,120
2026-06-01,eu,80
2026-06-02,us,200
2026-06-02,eu,95
EOF

wc -l data/sales.csv                 # how many lines (incl. header)?
head -n 1 data/sales.csv             # the header row
grep ",us," data/sales.csv           # only US rows
grep ",us," data/sales.csv | wc -l   # how many US rows?
cut -d, -f3 data/sales.csv | tail -n +2   # just the amount column, skip header
✓ You should see

wc -l → 5 (4 data rows + 1 header). The grep ",us," lines show the two US rows; piping to wc -l → 2. cut prints 120 / 200 / 80 / 95's amounts. You just filtered and projected columns — the shell equivalent of a SQL WHERE and SELECT.

Survival kit

The handful of extras that make daily life smooth:

Command / keyDoes
TabAuto-complete file/command names — use it constantly
↑ / Ctrl-RPrevious command / search command history
Ctrl-CStop the running command
Ctrl-L / clearClear the screen
man ls / ls --helpRead a command's manual / quick help
which pythonShow which program runs for a name
grep -r "TODO" .Recursively search text in files under .
find . -name "*.csv"Find files by name pattern

✓ Check yourself

  • Can you create a nested folder, add a file, rename it, and delete it — without looking it up?
  • Can you explain the difference between an absolute and a relative path?
  • Can you read what a cmd1 | cmd2 > file line does, left to right?
  • Do you know how to peek at the first few rows of a huge file?
Exercise — Answer three questions about a file using only the shell

Using the data/sales.csv you created: (1) how many data rows are there (excluding the header)? (2) How many are from the eu region? (3) Save just the EU rows to data/eu.csv.

solution
# 1) data rows = total lines minus the header
tail -n +2 data/sales.csv | wc -l        # -> 4

# 2) count EU rows
grep ",eu," data/sales.csv | wc -l        # -> 2

# 3) save EU rows to a new file
grep ",eu," data/sales.csv > data/eu.csv
cat data/eu.csv                            # confirm

If you reached for grep and a pipe instinctively, the core skill has landed. Everything else is vocabulary you'll pick up as you need it.

Next

You can drive the machine. Now let's make sure every change you make is tracked, reversible, and shareable — by putting your work under version control. → Git & GitHub