Observability & Monitoring

You can't operate what you can't see

There's a tempting moment, right after you ship a pipeline, when everything seems fine and you walk away. Then three weeks later someone asks, "why is the sales dashboard showing yesterday's numbers as zero?" — and you have no idea. Did the job fail? Did it succeed but read an empty source? Did a schema change silently drop a column? You're flying blind, and the only way to find out is to dig, manually, while people wait.

Monitoring answers a fixed question you decided to ask in advance: "did the nightly job finish?" Observability is the broader property of a system that lets you answer questions you didn't anticipate — to start from a symptom ("the dashboard is wrong") and trace your way to a cause without shipping new code first. Monitoring tells you that something is wrong; observability helps you understand why.

The three pillars: logs, metrics, traces

Observability is conventionally built from three kinds of signal. They overlap, but each answers a different question, and a healthy system emits all three.

Data observability — a special case that matters

Here's the twist that's unique to our field. In ordinary software, "healthy" usually means "the service is up and responding." For a data platform, the code can run perfectly, exit zero, log no errors — and still produce wrong data. The job succeeded; the output is garbage. Generic observability won't catch that, because nothing technically failed.

Data observability (you met the idea back in Course 4, Ch.11 on data quality) extends monitoring from "is the code running?" to "is the data correct and on time?". It watches the data itself across four dimensions:

Dedicated platforms (Monte Carlo and Bigeye are well-known examples) automate this: they profile your tables, learn what "normal" freshness and volume look like, and alert when reality drifts — often before any human notices. You don't need a vendor to start, though. dbt tests, freshness checks, and Dagster asset checks already give you a solid first layer; a managed tool simply scales it across hundreds of tables.

Alerting: turning signals into action

Signals are useless if nobody acts on them. An alert is a signal that crosses a threshold and reaches a human (or another system) to do something about it. The art is entirely in restraint.

The single biggest failure mode is noise. An alert that fires for things that don't need action trains everyone to ignore it — and the one time it matters, it's lost in the flood. This is "alert fatigue," and it's how real outages slip through.

To decide what's "truly bad" you need a shared definition of acceptable. That's what SLAs, SLOs, and error budgets give you, in plain language:

• SLO (Service Level Objective) — your internal target. "The orders table is fresh within 2 hours, 99% of the time." It's a number you commit to and measure against.
• SLA (Service Level Agreement) — a promise to someone else (a customer, another team), often with consequences if broken. SLAs are usually a little looser than your SLO, so you have margin.
• Error budget — the flip side of an SLO. If you aim for 99% freshness, you've granted yourself 1% of "stale is OK." That 1% is a budget: spend it on risk and shipping fast, and when it's gone, slow down and stabilize. It turns "never fail" (impossible) into "fail within a known, agreed amount."

Dashboards for the platform itself

You've built business dashboards — revenue, signups, churn. A data platform needs a different kind: an operational dashboard that watches the machinery, not the business. Same tools, opposite audience. The business dashboard answers "how is the company doing?"; the operational dashboard answers "is the system that produces those numbers healthy?"

The encouraging part: your orchestrator gives you much of this for free. Dagster's UI already shows run history, status, durations, asset freshness, and logs out of the box — a real operational dashboard you didn't have to build. Tools like Grafana add custom panels on top of your metrics when you outgrow the built-in views.

On-call & blameless postmortems

Alerts have to reach a person, and that's what being on-call means: for a defined window — usually a week at a time, rotating around the team — you're the one who responds when something pages. Done well, on-call is fair and humane; done badly, it's burnout. The practices below are what make the difference.

• Rotation — responsibility cycles through the team so no single person carries it. Everyone shares both the load and the learning.
• Runbooks — a short written guide for each known alert: what it means, how to check, common fixes, who to escalate to. A good runbook lets a tired engineer at 3am act without first understanding the whole system. Write the runbook the first time you solve a problem.
• Incident response — a calm, repeatable sequence: acknowledge the alert, assess impact, mitigate (stop the bleeding — even a manual rerun counts), communicate to stakeholders, then resolve the root cause. Mitigate first; perfect fixes come after the fire is out.
• Postmortem — after a notable incident, write up what happened, the timeline, the root cause, and the concrete changes that prevent a repeat.

Hands-on: structured logging & a metric

Let's make a pipeline function observable. Two upgrades: structured logging (logs as JSON, so a machine can search and filter them — not just prose for humans) and a simple metric (row count and duration emitted at the end of a run). Create observe_demo.py and run it.

import json
import logging
import time

# --- structured logging: emit each log line as a JSON object ---
class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "ts": self.formatTime(record),
            "level": record.levelname,
            "event": record.getMessage(),
        }
        # attach any extra fields passed via logger.info(..., extra={...})
        for key, value in getattr(record, "context", {}).items():
            entry[key] = value
        return json.dumps(entry)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("pipeline")
log.setLevel(logging.INFO)
log.addHandler(handler)


def run_pipeline(source_rows):
    """A stand-in for a real ingest/transform step."""
    start = time.perf_counter()
    log.info("run_started", extra={"context": {"job": "daily_orders"}})

    processed = [r for r in source_rows if r is not None]   # pretend transform
    time.sleep(0.2)                                          # pretend work

    duration_s = round(time.perf_counter() - start, 3)
    row_count = len(processed)

    # --- a metric: structured, machine-readable, easy to chart or alert on ---
    log.info("run_finished", extra={"context": {
        "job": "daily_orders",
        "metric.row_count": row_count,
        "metric.duration_s": duration_s,
        "status": "success",
    }})
    return row_count, duration_s


if __name__ == "__main__":
    run_pipeline([1, 2, None, 4, 5])

python observe_demo.py

{"ts": "2026-06-21 14:02:11,330", "level": "INFO", "event": "run_started", "job": "daily_orders"}
{"ts": "2026-06-21 14:02:11,533", "level": "INFO", "event": "run_finished", "job": "daily_orders", "metric.row_count": 4, "metric.duration_s": 0.203, "status": "success"}

Wrapping up Tooling

Step back and look at what Course 5 has turned your work into. You started with a folder of scripts that ran on one laptop. You now have a platform that is:

That list is, almost word for word, what a senior data engineering job description asks for. You haven't just learned tools — you've learned how professionals operate a system other people depend on.

✓ Check yourself

Answer from memory before opening each one.

1. Last night's job exited zero, logged no errors, and shows green in the orchestrator. This morning the dashboard reports yesterday's total as zero, because the upstream source was empty when the job read it. Why did process monitoring see nothing, and which data-observability dimension would have caught it?

   Show answer

   Nothing technically failed: the code ran correctly on empty input, and "is the code running?" is all process monitoring asks. The tempting belief is that a green run means good data; a pipeline that loads wrong numbers throws nothing. A volume check — rows arrived versus the normal range — catches a load that came in empty or short. See Data observability.

2. You run observe_demo.py unchanged, with its input [1, 2, None, 4, 5]. How many lines does it print, what is metric.row_count on the last one, and what could a log system do with that line that it could not do with print("done")?

   Show answer

   Two JSON lines — run_started and run_finished — and metric.row_count is 4, because the pretend transform drops the None. Since each line is a JSON object with named fields, a log system can filter on event, chart metric.duration_s over time, or alert when status is not success; prose output supports none of that. See the hands-on.

3. The nightly run has crept from 10 minutes to 40 over a month, and it spans ingest, transform and publish across several services. Which kind of signal tells you the creep is happening, which tells you where the time goes, and which do you read once you know the step?

   Show answer

   A metric (run duration over time) shows the trend — but metrics say that something changed, rarely why. A trace follows one run hop by hop with timing, so it shows which step is slow. Logs then give the ordered detail of what that step was doing. See The three pillars.

4. The team channel gets "CPU is at 71%" most nights; all twelve engineers are tagged, and most have muted it. Name what is wrong with this alert, the danger it creates, and how you would arrive at a replacement rather than guessing a better number.

   Show answer

   It is not actionable (71% implies no fix), not rare, not targeted, and it interrupts people for something that is at most a ticket. The danger is alert fatigue: a muted channel is where the one real outage gets lost. Start from an SLO — "the table is fresh within 2 hours, 99% of the time" — and let it write the alert: fire when freshness exceeds 2 hours, routed to whoever is on call. See Alerting.

5. Your SLO is "fresh within 2 hours, 99% of the time, measured monthly". By the 15th the table has already been stale for nine hours in total, and product wants a risky pipeline migration shipped this week. What does the error budget say, and what would have made shipping it the right call?

   Show answer

   One percent of a month is roughly seven hours, so the budget is already overspent: slow down and stabilize before adding risk. Had most of the budget been left, shipping would be exactly what it is for — the budget turns "never fail", which is impossible, into "fail within an agreed amount", and spending it on speed is legitimate. What decides it is the measured budget, not anyone's nerve. See SLOs and error budgets.

6. A postmortem draft gives the root cause as "the on-call engineer ran the backfill against prod instead of staging by mistake" and the action item as "be more careful". What is wrong with both lines, how would you rewrite them, and what does the team lose if they stay?

   Show answer

   Both blame a person and change nothing about the system, so the next tired engineer can make the same slip. A blameless version asks what about the system let it happen — why was pointing a backfill at prod that easy, and what makes it safe — and produces concrete changes that prevent a repeat. Keep the blame and people learn to hide mistakes, so problems surface late, as disasters rather than early reports. See On-call & blameless postmortems.

import json, logging, time

# Without a handler and a level, INFO lines are silently dropped.
logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("pipeline")

def with_observability(fn):
    """Wrap a pipeline step so it emits a structured run metric."""
    def wrapped(*args, **kwargs):
        start = time.perf_counter()
        status, row_count = "failed", None
        try:
            result = fn(*args, **kwargs)
            status = "success"
            row_count = len(result) if hasattr(result, "__len__") else None
            return result
        finally:
            log.info(json.dumps({
                "event": "run_finished",
                "job": fn.__name__,
                "status": status,
                "metric.row_count": row_count,
                "metric.duration_s": round(time.perf_counter() - start, 3),
            }))
    return wrapped

@with_observability
def load_rentals():
    return [{"rental_id": 1}, {"rental_id": 2}, {"rental_id": 3}]

load_rentals()

SLO:   The orders table is fresh within 2 hours, 99% of the time
       (measured monthly).
Alert: Fire to the #data-oncall channel when
       now() - max(loaded_at) > 2 hours.
Budget: 1% of the month (~7 hours of staleness) is acceptable
        before we pause feature work to stabilize.

Takeaway

• Monitoring answers questions you chose in advance, and observability lets you start from a symptom and ask questions you did not anticipate.
• Logs say what happened and in what order, metrics say whether it is normal and getting worse, and traces show where one run spent its time across services.
• A data job can exit zero and still load wrong data, so data observability watches the data itself, such as its volume and freshness, and not only whether the code ran.
• Alert only on what needs a human, because noise causes alert fatigue and the one real page gets lost; an operational dashboard that watches the machinery gives the context.
• Define the SLO first and make the alert fire when it is breached; the SLA is the looser promise made to others, and the error budget is the room the SLO leaves you.
• On-call works with a rotation, a runbook per alert, mitigate-then-fix incident steps and a blameless postmortem that asks what let it happen.
• Emit structured JSON logs and a run metric, because print("done") cannot be filtered, charted or alerted on.

That completes the tooling course, and the next course puts every piece together into one platform you build end to end. → Capstone Labs — Build Mini-GridDP

Further reading

• Google SRE book: Monitoring Distributed Systems — Beyer et al., Site Reliability Engineering, O'Reilly, 2016. The four golden signals and what makes an alert worth a page.
• Google SRE book: Service Level Objectives — Beyer et al., Site Reliability Engineering, O'Reilly, 2016. SLIs, SLOs and SLAs, and how to choose targets.
• Google SRE book: Postmortem Culture: Learning from Failure — Beyer et al., Site Reliability Engineering, O'Reilly, 2016. Blameless postmortems and what they should contain.
• OpenTelemetry: Signals — OpenTelemetry docs, accessed 2026. The vendor-neutral definitions of traces, metrics and logs.
• Python: Logging HOWTO — Python docs, accessed 2026. Levels, handlers and formatters, the basis of the structured logging in the lab.