Ops & Rollout

Why ops is FDE work

In a normal SRE setup, platform engineering owns production. In an FDE deployment, the customer's data team owns their environment. You're operating across the boundary — your pipeline runs in a place where you don't fully control the change-management or incident-response process.

The shape that works:

• Treat the customer's data team as a peer ops partner, not as users.
• Adopt their change-management process. Don't invent your own.
• Write runbooks they can run without you.
• Hand off on-call responsibility gradually as the deployment matures.

Staging at the customer

Every customer deployment should have at least two environments:

• Staging — where you ship changes first. Customer's data team validates. Output lands in a sandbox warehouse schema or a feature-flagged dashboard.
• Production — where the customer's business users see output. Changes promoted only after staging validation.

Sandbox data

The customer's data team needs a realistic data subset to validate against. Patterns:

• Subset of recent documents — fast, less realistic.
• Anonymized full-volume — slower setup, more realistic.
• Shadow-mode in production — run the change in production but write to a separate schema; compare.

Schema separation

Use distinct warehouse schemas (e.g., contracts_prod, contracts_staging) rather than separate databases. Lets analysts query either side with familiar tools.

Change management

Every customer has a process. Find it and follow it. Common shapes:

• Change Advisory Board (CAB) — formal review meeting, weekly or biweekly. Required for production changes. Common in regulated industries.
• Ticket-based approval — file a change ticket, get sign-off from the customer's release manager, deploy in a defined window.
• Freeze windows — most customers freeze around quarter-end, year-end, key business events.
• Maintenance windows — when changes are allowed (often nights or weekends).

What goes in a change request

• What's changing — concrete, not abstract.
• Why — the business reason, not the technical one.
• Risk — what could go wrong, how likely, what the blast radius is.
• Validation — what tests run pre-deploy, what monitoring catches regressions post-deploy.
• Rollback — exact steps to revert if it goes wrong.
• Communication plan — who gets notified before, during, after.

Runbooks

The artifact that matters most for handoff. A runbook is "when X happens, do Y." Written for the customer's data team — not for you.

What to write runbooks for

• Pipeline failure — DAG didn't run, ingestion step errored, extraction service unreachable.
• Quality alert — extraction accuracy dropped on a class.
• Schema drift — upstream source added or removed a field.
• Backfill — how to re-run extraction on a historical batch.
• New document class — how to onboard a new document type without your involvement.
• HITL queue overflow — what to do when reviewer capacity is exceeded.

The structure of a good runbook

1. Symptom — what the alert or report looks like.
2. Severity — what to escalate, what to handle inline.
3. Triage steps — first 10 minutes of debugging.
4. Likely causes — in order of frequency.
5. Resolution paths — what to do for each cause.
6. Escalation — when to page the platform team or your account team.

Rollbacks

Every change needs a defined rollback path. Two patterns:

Versioned outputs

Because extractions are versioned by model (chapter 6), rollback to a prior model is "switch the active version pointer and re-promote prior data." Idempotent pipelines make this safe.

Feature flags

Wrap new behaviors in flags scoped per customer (or per document class). Toggle on for staging, then production, then off if needed.

What to NOT rollback

Schema additions (new columns) — additive changes don't roll back; they get ignored. Customer-data ingestion — you don't delete the customer's data because you broke something downstream. The data is preserved; only the consuming logic rolls back.

On-call

Embedded ops on-call is unusual. Patterns:

Stage 1: FDE-primary

During active deployment (months 1–3), the FDE is primary on-call for issues affecting the customer. You handle pages, you write the runbooks, you escalate to platform.

Stage 2: Joint on-call

As the deployment matures (months 3–6), you train the customer's data team to be primary for first-tier triage. They follow runbooks; they escalate to you only when runbooks don't resolve.

Stage 3: Customer-primary

Post-handoff. Customer's team owns first responder; you're available for platform escalations only. Your weekly cadence drops to monthly check-ins.

What to monitor

• Service health — pipeline DAG success rate, latency, throughput.
• Data quality — row counts in expected bands, schema-drift alerts, confidence-distribution drift.
• SLA metrics — extraction accuracy by class, HITL queue depth, review latency.
• Business signals — number of customer-flagged corrections, support tickets, dashboard usage.

The business signals are the leading indicator that a deployment is going wrong, before any monitoring metric fires.

Interview probes