Section E · Production / Cloud

Ops & Rollout

Embedded ops hygiene — staging environments at the customer, change management, runbooks, rollbacks, and the on-call posture when the customer's data team is your first responder.

Why ops is FDE work

In a normal SRE setup, platform engineering owns production. In an FDE deployment, the customer's data team owns their environment. You're operating across the boundary — your pipeline runs in a place where you don't fully control the change-management or incident-response process.

The shape that works:

  • Treat the customer's data team as a peer ops partner, not as users.
  • Adopt their change-management process. Don't invent your own.
  • Write runbooks they can run without you.
  • Hand off on-call responsibility gradually as the deployment matures.

Staging at the customer

Every customer deployment should have at least two environments:

  • Staging — where you ship changes first. Customer's data team validates. Output lands in a sandbox warehouse schema or a feature-flagged dashboard.
  • Production — where the customer's business users see output. Changes promoted only after staging validation.

Sandbox data

The customer's data team needs a realistic data subset to validate against. Patterns:

  • Subset of recent documents — fast, less realistic.
  • Anonymized full-volume — slower setup, more realistic.
  • Shadow-mode in production — run the change in production but write to a separate schema; compare.

Schema separation

Use distinct warehouse schemas (e.g., contracts_prod, contracts_staging) rather than separate databases. Lets analysts query either side with familiar tools.

Change management

Every customer has a process. Find it and follow it. Common shapes:

  • Change Advisory Board (CAB) — formal review meeting, weekly or biweekly. Required for production changes. Common in regulated industries.
  • Ticket-based approval — file a change ticket, get sign-off from the customer's release manager, deploy in a defined window.
  • Freeze windows — most customers freeze around quarter-end, year-end, key business events.
  • Maintenance windows — when changes are allowed (often nights or weekends).

What goes in a change request

  • What's changing — concrete, not abstract.
  • Why — the business reason, not the technical one.
  • Risk — what could go wrong, how likely, what the blast radius is.
  • Validation — what tests run pre-deploy, what monitoring catches regressions post-deploy.
  • Rollback — exact steps to revert if it goes wrong.
  • Communication plan — who gets notified before, during, after.

Runbooks

The artifact that matters most for handoff. A runbook is "when X happens, do Y." Written for the customer's data team — not for you.

What to write runbooks for

  • Pipeline failure — DAG didn't run, ingestion step errored, extraction service unreachable.
  • Quality alert — extraction accuracy dropped on a class.
  • Schema drift — upstream source added or removed a field.
  • Backfill — how to re-run extraction on a historical batch.
  • New document class — how to onboard a new document type without your involvement.
  • HITL queue overflow — what to do when reviewer capacity is exceeded.

The structure of a good runbook

  1. Symptom — what the alert or report looks like.
  2. Severity — what to escalate, what to handle inline.
  3. Triage steps — first 10 minutes of debugging.
  4. Likely causes — in order of frequency.
  5. Resolution paths — what to do for each cause.
  6. Escalation — when to page the platform team or your account team.
The runbook quality test

Hand a runbook to a customer's data analyst who wasn't involved in your deployment. Can they follow it? If not, it's not finished. Senior FDE habit: write runbooks early, test them with a real customer analyst, iterate.

Rollbacks

Every change needs a defined rollback path. Two patterns:

Versioned outputs

Because extractions are versioned by model (chapter 6), rollback to a prior model is "switch the active version pointer and re-promote prior data." Idempotent pipelines make this safe.

Feature flags

Wrap new behaviors in flags scoped per customer (or per document class). Toggle on for staging, then production, then off if needed.

What to NOT rollback

Schema additions (new columns) — additive changes don't roll back; they get ignored. Customer-data ingestion — you don't delete the customer's data because you broke something downstream. The data is preserved; only the consuming logic rolls back.

On-call

Embedded ops on-call is unusual. Patterns:

Stage 1: FDE-primary

During active deployment (months 1–3), the FDE is primary on-call for issues affecting the customer. You handle pages, you write the runbooks, you escalate to platform.

Stage 2: Joint on-call

As the deployment matures (months 3–6), you train the customer's data team to be primary for first-tier triage. They follow runbooks; they escalate to you only when runbooks don't resolve.

Stage 3: Customer-primary

Post-handoff. Customer's team owns first responder; you're available for platform escalations only. Your weekly cadence drops to monthly check-ins.

What to monitor

  • Service health — pipeline DAG success rate, latency, throughput.
  • Data quality — row counts in expected bands, schema-drift alerts, confidence-distribution drift.
  • SLA metrics — extraction accuracy by class, HITL queue depth, review latency.
  • Business signals — number of customer-flagged corrections, support tickets, dashboard usage.

The business signals are the leading indicator that a deployment is going wrong, before any monitoring metric fires.

Interview probes

Show probe 1: "Walk me through change management for a new extraction model version."

(1) Spec the change — what behavior is changing, what's the expected impact. (2) Run in shadow against production for 1–2 weeks; compare outputs against the current version. (3) Promote to staging schema; customer's data team validates against their sample. (4) File CAB ticket (if customer requires) with risk, validation, rollback, communication plan. (5) Deploy in maintenance window. (6) Monitor SLA metrics for the first week; revert if any class regresses. (7) Document in customer-facing notes for their analysts. The senior signal: shadow mode before staging, written rollback plan before deploy.

Show probe 2: "Customer's analyst pages you at 3 AM. What does a good runbook look like?"

Six sections, on one page: symptom (what the page looks like), severity (page or wait until morning), triage (first 10 minutes), likely causes (in order of frequency), resolution paths (what to do for each), escalation (when to page platform). Written for the analyst, not you — vocabulary they know, links to dashboards they have access to, no internal jargon. If they can't resolve a top-3 cause without paging you, the runbook needs work.

Show probe 3: "How do you hand off on-call to the customer?"

Three stages over months 3–6. Stage 1 (you primary): you write runbooks, you take pages, you triage. Stage 2 (joint): customer's team is first responder, follows your runbooks, escalates only when runbooks don't resolve. Stage 3 (customer-primary): they own first response; you're available for platform-team escalations only. The transition criterion is "their team can resolve the top-5 most common issues without me." Test it with intentional pages or shadow drills.

Show probe 4: "What's the most common reason a deployment slips?"

Identity and IT, usually. Auth tickets sit in customer queues for two weeks; production access requires CAB approval that meets every other week; the customer's IT team had a competing priority. The technical work is usually on track; the customer-coordination work isn't. Senior FDE habit: file every identity/access ticket in week one, even if you don't need it until week six. The lead time on people-coordination is longer than the lead time on engineering.

Show probe 5: "What goes in a status update to customer leadership?"

Five sections on one screen (see 04 §cadence for the Friday status doc structure): what shipped, what's open, what I need from you, what I'm worried about, next week's plan. The third and fourth sections are the differentiator — most candidates write status as a one-way report; great FDEs write it as a working document that prompts customer action. Send weekly, on Friday morning, regardless of progress. Consistency beats completeness.

✓ Check yourself

Answer from memory before opening each one.

  1. Why keep staging and production as separate schemas rather than separate databases, and what does shadow mode catch that a sampled staging copy does not?

    Show answer

    Analysts can query either side with the same tools and credentials. Shadow mode runs the change on real production inputs and writes elsewhere, so it exposes volume and data surprises that a small sample hides. See staging at the customer.

  2. A change board rejects your request for lacking a rollback. The change adds a column and switches the active extraction model. Write the rollback for each.

    Show answer

    For the column there is nothing to roll back, because additive changes are ignored by consumers. For the model, switch the active version pointer back and re-promote the earlier outputs, which versioned, idempotent pipelines allow. Never delete customer data to undo a downstream bug. See rollbacks.

  3. A runbook lists likely causes in order of frequency and says to test it on someone who was not on the project. Why both?

    Show answer

    The first sends a tired responder to the commonest cause first. The second shows whether the document works without your memory, since it is written for the customer's team, not you. See runbooks.

  4. In month four, the customer's analysts want to be paged first. What should be true before you agree, and what would hold you back?

    Show answer

    They should be able to resolve common incidents from runbooks without your help, and escalations to you should be falling. If runbooks fail often or the same incidents return, stay primary and fix the runbooks first. See on-call.

  5. Dashboards are green, but three analysts have reported wrong values this week. How do you read that?

    Show answer

    As a leading indicator that something is wrong before monitoring notices. Business signals such as corrections, tickets and dashboard usage often move first, so investigate and add the missing check. See on-call.

  6. The customer freezes changes around quarter-end, and you want to roll out a new model. How do you sequence it?

    Show answer

    Plan around the freeze: shadow before it starts, promote to staging, request approval through their process, and schedule production inside an allowed window. If the window is missed, wait rather than bypass it. See change management.

Takeaway

  • You operate across the customer's boundary, so adopt their change process and treat their data team as operations peers.
  • Keep staging and production as separate schemas, and validate with sampled, anonymized or shadow-mode data.
  • A change request states what, why, risk, validation, rollback and who is told.
  • Write runbooks for the customer's analysts, in a fixed shape, and test one on a person who was not on the project.
  • Roll back logic with versioned outputs and flags; do not roll back additive schema changes or the customer's data.
  • On-call moves from FDE-primary to joint to customer-primary, and business signals lead metrics.

Next is the business vocabulary the customer will use with you. → Domain Context

Further reading

  • On-Call — Google SRE workbook, 2018. Structuring rotations and handoffs between teams.
  • Being On-Call — Google SRE book, 2016. What good paging practice looks like, and its human cost.
  • Deployment environments — dbt documentation, accessed 2026. Separating development, staging and production for transformations.
  • Work with table history — Databricks documentation, accessed 2026. Time travel and restore, a concrete rollback mechanism for versioned data.
  • Feature Toggles (aka Feature Flags) — Martin Fowler, 2017. Kinds of flag and how to manage them, relevant to per-customer rollout.