Observability & Monitoring

You can't operate what you can't see

There's a tempting moment, right after you ship a pipeline, when everything seems fine and you walk away. Then three weeks later someone asks, "why is the sales dashboard showing yesterday's numbers as zero?" — and you have no idea. Did the job fail? Did it succeed but read an empty source? Did a schema change silently drop a column? You're flying blind, and the only way to find out is to dig, manually, while people wait.

Monitoring answers a fixed question you decided to ask in advance: "did the nightly job finish?" Observability is the broader property of a system that lets you answer questions you didn't anticipate — to start from a symptom ("the dashboard is wrong") and trace your way to a cause without shipping new code first. Monitoring tells you that something is wrong; observability helps you understand why.

The three pillars: logs, metrics, traces

Observability is conventionally built from three kinds of signal. They overlap, but each answers a different question, and a healthy system emits all three.

Data observability — a special case that matters

Here's the twist that's unique to our field. In ordinary software, "healthy" usually means "the service is up and responding." For a data platform, the code can run perfectly, exit zero, log no errors — and still produce wrong data. The job succeeded; the output is garbage. Generic observability won't catch that, because nothing technically failed.

Data observability (you met the idea back in Course 4, Ch.11 on data quality) extends monitoring from "is the code running?" to "is the data correct and on time?". It watches the data itself across four dimensions:

Dedicated platforms (Monte Carlo and Bigeye are well-known examples) automate this: they profile your tables, learn what "normal" freshness and volume look like, and alert when reality drifts — often before any human notices. You don't need a vendor to start, though. dbt tests, freshness checks, and Dagster asset checks already give you a solid first layer; a managed tool simply scales it across hundreds of tables.

Alerting: turning signals into action

Signals are useless if nobody acts on them. An alert is a signal that crosses a threshold and reaches a human (or another system) to do something about it. The art is entirely in restraint.

The single biggest failure mode is noise. An alert that fires for things that don't need action trains everyone to ignore it — and the one time it matters, it's lost in the flood. This is "alert fatigue," and it's how real outages slip through.

To decide what's "truly bad" you need a shared definition of acceptable. That's what SLAs, SLOs, and error budgets give you, in plain language:

• SLO (Service Level Objective) — your internal target. "The orders table is fresh within 2 hours, 99% of the time." It's a number you commit to and measure against.
• SLA (Service Level Agreement) — a promise to someone else (a customer, another team), often with consequences if broken. SLAs are usually a little looser than your SLO, so you have margin.
• Error budget — the flip side of an SLO. If you aim for 99% freshness, you've granted yourself 1% of "stale is OK." That 1% is a budget: spend it on risk and shipping fast, and when it's gone, slow down and stabilize. It turns "never fail" (impossible) into "fail within a known, agreed amount."

Dashboards for the platform itself

You've built business dashboards — revenue, signups, churn. A data platform needs a different kind: an operational dashboard that watches the machinery, not the business. Same tools, opposite audience. The business dashboard answers "how is the company doing?"; the operational dashboard answers "is the system that produces those numbers healthy?"

The encouraging part: your orchestrator gives you much of this for free. Dagster's UI already shows run history, status, durations, asset freshness, and logs out of the box — a real operational dashboard you didn't have to build. Tools like Grafana add custom panels on top of your metrics when you outgrow the built-in views.

On-call & blameless postmortems

Alerts have to reach a person, and that's what being on-call means: for a defined window — usually a week at a time, rotating around the team — you're the one who responds when something pages. Done well, on-call is fair and humane; done badly, it's burnout. The practices below are what make the difference.

• Rotation — responsibility cycles through the team so no single person carries it. Everyone shares both the load and the learning.
• Runbooks — a short written guide for each known alert: what it means, how to check, common fixes, who to escalate to. A good runbook lets a tired engineer at 3am act without first understanding the whole system. Write the runbook the first time you solve a problem.
• Incident response — a calm, repeatable sequence: acknowledge the alert, assess impact, mitigate (stop the bleeding — even a manual rerun counts), communicate to stakeholders, then resolve the root cause. Mitigate first; perfect fixes come after the fire is out.
• Postmortem — after a notable incident, write up what happened, the timeline, the root cause, and the concrete changes that prevent a repeat.

Hands-on: structured logging & a metric

Let's make a pipeline function observable. Two upgrades: structured logging (logs as JSON, so a machine can search and filter them — not just prose for humans) and a simple metric (row count and duration emitted at the end of a run). Create observe_demo.py and run it.

import json
import logging
import time

# --- structured logging: emit each log line as a JSON object ---
class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "ts": self.formatTime(record),
            "level": record.levelname,
            "event": record.getMessage(),
        }
        # attach any extra fields passed via logger.info(..., extra={...})
        for key, value in getattr(record, "context", {}).items():
            entry[key] = value
        return json.dumps(entry)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("pipeline")
log.setLevel(logging.INFO)
log.addHandler(handler)


def run_pipeline(source_rows):
    """A stand-in for a real ingest/transform step."""
    start = time.perf_counter()
    log.info("run_started", extra={"context": {"job": "daily_orders"}})

    processed = [r for r in source_rows if r is not None]   # pretend transform
    time.sleep(0.2)                                          # pretend work

    duration_s = round(time.perf_counter() - start, 3)
    row_count = len(processed)

    # --- a metric: structured, machine-readable, easy to chart or alert on ---
    log.info("run_finished", extra={"context": {
        "job": "daily_orders",
        "metric.row_count": row_count,
        "metric.duration_s": duration_s,
        "status": "success",
    }})
    return row_count, duration_s


if __name__ == "__main__":
    run_pipeline([1, 2, None, 4, 5])

python observe_demo.py

{"ts": "2026-06-21 14:02:11,330", "level": "INFO", "event": "run_started", "job": "daily_orders"}
{"ts": "2026-06-21 14:02:11,533", "level": "INFO", "event": "run_finished", "job": "daily_orders", "metric.row_count": 4, "metric.duration_s": 0.203, "status": "success"}

Wrapping up Tooling

Step back and look at what Course 5 has turned your work into. You started with a folder of scripts that ran on one laptop. You now have a platform that is:

That list is, almost word for word, what a senior data engineering job description asks for. You haven't just learned tools — you've learned how professionals operate a system other people depend on.

✓ Check yourself

• Can you explain the difference between monitoring and observability — and between logs, metrics, and traces?
• Why does data need its own observability beyond "is the code running?" Name the four dimensions.
• What makes an alert good rather than noisy, and how do SLOs and error budgets feed into that?
• Why is a postmortem "blameless," and what does that get you?

import json, logging, time

log = logging.getLogger("pipeline")

def with_observability(fn):
    """Wrap a pipeline step so it emits a structured run metric."""
    def wrapped(*args, **kwargs):
        start = time.perf_counter()
        try:
            result = fn(*args, **kwargs)
            status = "success"
            return result
        except Exception:
            status = "failed"
            raise
        finally:
            log.info(json.dumps({
                "event": "run_finished",
                "job": fn.__name__,
                "metric.duration_s": round(time.perf_counter() - start, 3),
                "status": status,
            }))
    return wrapped

SLO:   The orders table is fresh within 2 hours, 99% of the time
       (measured monthly).
Alert: Fire to the #data-oncall channel when
       now() - max(loaded_at) > 2 hours.
Budget: 1% of the month (~7 hours of staleness) is acceptable
        before we pause feature work to stabilize.

Next

🎉 That's a wrap on Course 5 — Tooling & the Modern Stack. You've taken work that ran once on a laptop and made it collaborative, reproducible, cloud-aware, automated, and observable. That's the full arc from "demo" to "production system," and it's a genuinely hard milestone — take a moment to appreciate it.

What's next is the fun part: putting all of it together. Course 6 — Capstone Labs hands you end-to-end builds where you assemble everything from Courses 1–5 into one real platform — model, ingest, transform, test, containerize, deploy, and observe — start to finish. (After that, Course 7 turns to career: portfolio, interviews, and landing the role.) It's ready now → start the Capstone Labs.