Section C · The hard parts

Runbook Templates

Six pre-written runbooks for the most common failure modes. Each is the starting point — customize per deployment with the customer's specific stack, alert channels, and on-call contacts.

Why runbooks matter

Runbooks are the artifact that lets you hand off on-call. Without them, every incident pages the FDE; with them, the customer's data team can handle the top-5 most-common incidents independently. The single biggest determinant of whether you can actually exit a deployment is whether the runbooks are good.

Write them for the customer's analyst, not for you. Vocabulary they know. Links they have access to. No internal jargon. If a customer-side analyst can follow a runbook end-to-end without paging you, it's working.

Standard structure

Every runbook in this set uses six sections. Consistency makes them readable; consistency makes them maintainable.

Symptom — what the alert or report looks like.
Severity — page vs morning vs file-and-forget.
Triage — first 10 minutes of debugging. Specific commands, specific dashboards.
Likely causes — in order of frequency. The 80/20.
Resolution paths — what to do for each likely cause.
Escalation — when to page the platform team or the FDE.

Runbook 1: Pipeline failure

runbook_pipeline_failure.md

# Runbook: Pipeline Failure
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- Datadog alert "ingestion DAG failed" or
- Friday status mentions "extraction did not run for last N days" or
- Customer reports stale data in dashboards

## Severity
- **Page immediately** if last successful run was > 24 hours ago
  AND business hours are within the next 4 hours.
- **Next business morning** if last successful run was > 24 hours ago
  outside business hours.
- **Note & investigate** if < 24 hours.

## Triage (first 10 minutes)
1. Open the orchestration UI ([link to Airflow / Dagster instance])
2. Look at the failed task — what's the error message?
3. Look at upstream task — did its data arrive?
4. Check the source system status page (CLM, ERP, SharePoint)
5. Note the last successful run timestamp

## Likely causes (order of frequency)
1. **Source system unavailable.** CLM had an outage; SharePoint
   permissions changed; API token expired.
2. **Network / auth.** Service account password rotated; VPN
   gateway issue; IP allowlist change.
3. **Upstream schema change.** Source added or removed a field;
   our parser doesn't handle it.
4. **Warehouse storage / quota.** Snowflake / BigQuery storage
   limit hit; load failed.
5. **Code regression.** Recent deploy broke something.

## Resolution paths
- Source unavailable → wait + verify when restored. Backfill when stable.
- Auth → rotate token / update service-account credentials.
  See "Auth refresh" section of [link to ops doc].
- Schema change → file ticket with FDE on-call; deploy parser
  update (link to repo).
- Warehouse quota → increase quota with [warehouse admin] or
  archive old partitions per [retention policy doc].
- Regression → revert the last release; file bug in [tracking system].

## Escalation
- If unresolved in 2 hours and a customer-visible impact:
  page [FDE on-call: name, contact].
- If platform-team work is needed:
  file [internal ticket template] tagged "FDE-customer-blocker."

Runbook 2: Quality alert

runbook_quality_alert.md

# Runbook: Quality Alert
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- SLA dashboard shows accuracy below threshold on a document class.
- Customer reports specific extractions are wrong.
- Confidence-distribution alert (sudden shift in confidence pattern).

## Severity
- **Page immediately** if > 5pp below SLA and impacts customer-facing
  decisions (e.g., renewal alerts going to wrong agreements).
- **Same business day** if < 5pp below SLA but trending downward.
- **Investigate this week** if isolated incident, not trending.

## Triage (first 15 minutes)
1. Open the SLA dashboard. Confirm the regression is real
   (not a small-sample artifact).
2. Identify which document class is below SLA.
3. Sample 20 recent failures by eye. Pattern?
4. Check confidence-distribution dashboard for that class.
5. Check whether a recent model deploy preceded the drop.

## Likely causes
1. **Customer data drift.** New documents in the corpus have
   different layout / vocabulary than the trained distribution.
2. **Model regression.** Recent platform model deploy degraded
   on this class.
3. **HITL miscalibration.** Reviewers' bar shifted; what they
   accepted last month, they reject this month.
4. **Confidence drift.** Model is over-confident now; auto-approval
   threshold needs retuning.
5. **Ground-truth issue.** The gold set is wrong; the model is fine.

## Resolution paths
- Customer data drift → expand prompt / templates to cover the
  variance. Add to HITL queue temporarily while platform iterates.
- Model regression → revert to prior model version (link to
  versioned deploys). File platform bug.
- HITL miscalibration → recalibrate reviewers against gold set.
  Run inter-rater agreement check on calibration sample.
- Confidence drift → recompute reliability diagram (link to
  notebook). Tighten threshold per the new calibration.
- Gold-truth issue → audit the gold set. Update if confirmed.

## Communicate to customer
- If SLA is at risk: notify the customer data lead within 4 hours.
  Use this template:

> "Heads up — we noticed extraction quality on [class] trending
> toward the SLA line ([current pct] vs SLA of [threshold pct]).
> We've already done [containment]. We'll have a root-cause
> understanding by [date]. Won't impact your Monday review."

## Escalation
- Multi-day below SLA → page FDE on-call.
- Platform bug → file with customer evidence in [tracking system].

Runbook 3: Schema drift

runbook_schema_drift.md

# Runbook: Schema Drift
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- Pipeline runs but downstream rows are missing fields.
- dbt test failure on uniqueness or not-null.
- New / removed field detected in source-system payload.

## Severity
- **Same business day** if pipeline is silently dropping data.
- **Next business day** if pipeline is failing loudly (failure
  is the safer mode; you have time).

## Triage
1. Find the dbt test that failed, or the row-count anomaly that
   surfaced the drift.
2. Sample 5 source records before vs after the drift point.
3. Identify the specific field(s) that changed.
4. Determine: added, removed, type-changed, or renamed?

## Likely causes
1. **Source-system upgrade** — vendor pushed new schema (ERP,
   CLM, procurement suite).
2. **Customer-side configuration change** — customer's admin
   added/removed a custom field.
3. **Authentication scope change** — a field we used to see is
   now restricted to a different role.

## Resolution paths
- Added field → update parser to read it; add to JSON sidecar
  if you're not sure whether to promote yet.
- Removed field → check if downstream depends on it.
  If yes, file customer-side ticket. If no, deprecate gracefully.
- Type change → cast at the parser layer; alert if cast fails.
- Renamed → update the field-mapping config (link to the file).

## Escalation
- Customer-side scope change → escalate to customer's data lead;
  cc IT / system admin.
- Vendor-driven upgrade → file with platform PM (this affects
  every customer using that vendor).

Runbook 4: Backfill

runbook_backfill.md

# Runbook: Backfill
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## When you need this
- New model version released; reprocess historical corpus.
- New document class onboarded; backfill for the new class.
- Bug fixed that affected a date range; re-extract.
- Customer adds new historical documents (e.g., post-acquisition).

## Pre-flight checklist
- [ ] Determine scope: which documents, which date range, which
      document classes?
- [ ] Estimate cost: rows × model cost × HITL volume.
- [ ] Plan HITL capacity: backfill HITL volume can swamp the queue.
- [ ] Communicate to customer: their analysts may see changes.
- [ ] Confirm idempotency: re-running won't duplicate data.

## Execution
1. Identify the run scope (SQL query against `documents` table).
2. Update model_version pin if applicable.
3. Trigger backfill via [orchestrator UI / CLI command].
4. Monitor HITL queue depth and reviewer throughput.
5. If queue overflows, throttle backfill rate.

## Validation
- Spot-check 20 random reprocessed documents.
- Run dbt tests on the affected schema.
- Compare aggregates against pre-backfill snapshot.

## Customer communication
- Backfills change historical data. Tell the customer when one runs
  (especially on dashboards they actively use).
- Template:
  > "We're reprocessing [N] documents from [date range] with our
  > latest extraction model. You may see updated values in your
  > [dashboard] starting [date]. We expect [accuracy improvement]
  > on average; flag anything that looks unexpected."

## Escalation
- HITL queue can't absorb the volume → contact platform PM
  for additional reviewer capacity.
- Validation fails → pause backfill; escalate to FDE.

Runbook 5: Onboarding a new document class

runbook_new_doc_class.md

# Runbook: New Document Class
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## When you need this
- Customer brings a new document type into scope (e.g., DPAs,
  NDAs, supplier code-of-conduct attachments).

## Pre-flight
- [ ] Confirm the class is in scope per working agreement.
      If not, scope discussion first.
- [ ] Gather 20+ representative samples from the customer.
- [ ] Identify the fields to extract per class.
- [ ] Define the SLA threshold for this class.

## Configuration
1. Add the document-class taxonomy entry: [link to config file]
2. Define field schema (which fields, types, tolerances): [link]
3. Configure extraction prompt / template: [link]
4. Set initial confidence threshold (default 0.85; calibrate after
   first 100 documents).

## Pilot
- Run on 100-document sample.
- Have customer-side analysts validate the output by eye.
- Measure field-level accuracy against their validation.
- Tune threshold; expand HITL coverage if needed.

## Production
- Roll out gradually: 10% → 50% → 100% over a week.
- Watch confidence distribution and HITL queue depth.
- Add the class to the SLA dashboard.
- Update working agreement to reflect the added class.

## Escalation
- Accuracy on pilot is materially below SLA target → file with
  platform team for extraction-model expansion.
- Customer wants the class in scope before pilot validates →
  escalate to AE (commercial conversation).

Runbook 6: HITL queue overflow

runbook_hitl_overflow.md

# Runbook: HITL Queue Overflow
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- HITL queue depth alert (queue grows faster than reviewers process).
- Customer-visible delay between ingest and final output.
- Reviewer-throughput metric below capacity.

## Severity
- **Same business day** if delay exceeds 24 hours (SLA likely
  affected).
- **Investigate next day** if delay 12–24 hours.

## Triage
1. Check queue depth trend over last 7 days.
2. Check reviewer throughput. Down? Up but volume up more?
3. Check confidence-distribution. Are more documents routing to
   HITL than expected?

## Likely causes
1. **Volume spike.** Customer ingested a backlog; HITL volume
   spikes proportionally.
2. **Confidence drift.** Model has shifted toward more uncertain
   predictions; more docs route to HITL than before.
3. **Reviewer capacity drop.** Reviewer team is short-staffed
   (vacation, training, attrition).
4. **Threshold misconfiguration.** Recent threshold tightening
   pushed too many docs to HITL.

## Resolution paths
- Volume spike → throttle ingest if customer can tolerate; add
  reviewer capacity if not.
- Confidence drift → recalibrate model; recompute thresholds.
- Reviewer capacity drop → escalate to reviewer-pool ops; consider
  contractor support if multi-week.
- Threshold misconfiguration → adjust threshold back; document why.

## Don't do this
- Don't relax HITL just to clear the queue. Below-threshold
  extractions hit customer-visible output; quality drops; SLA
  is now at risk.
- Don't promise customer faster turnaround without confirming
  with reviewer ops first.

## Escalation
- Persistent > 3 days → page FDE; consider temporary scope
  reduction conversation with customer.

✓ Check yourself

Answer from memory before opening each one.

You hand a finished runbook to the customer's analyst. They stall at the second triage step and call you. What does that tell you, and what do you change?

Show answer
The runbook failed its only test: it must work without paging you. The triage step is probably not specific enough, so it needs the exact dashboard or command, a link they can open and no internal vocabulary. Fix it and have them run it again. See why runbooks matter.
The ingestion DAG's last good run was 30 hours ago and you notice at 2 a.m. on a Saturday. What severity does the runbook assign?

Show answer
Next business morning. Page immediately only when the last success is over 24 hours old and business hours begin within four hours. Between 12 and 24 hours you investigate; under 24 hours you note it. See the pipeline failure runbook.
Two incidents arrive on Monday: a pipeline that crashes on a renamed source field, and one that finishes green but drops a field. Which do you take first?

Show answer
The silent one. A loud failure is the safer mode because nothing wrong reaches the customer, so it can wait until the next day. A pipeline that quietly drops data is same-day. See the schema drift runbook.
The dashboard shows accuracy on one class down four points. Name the first checks before you revert the model.

Show answer
Confirm it is not a small-sample artifact, identify the class, read about 20 recent failures by eye, look at the confidence distribution and check whether a model deploy came first. The gold set itself may be wrong, so check that before blaming the model. See the quality alert runbook.
A new model version needs 200,000 historical documents reprocessed. About 15% will route to review, and the queue clears 500 a day. What happens, and what do you do?

Show answer
That is 30,000 reviews, about 60 days of queue, and it would bury the live traffic. Throttle the backfill, run it in batches, confirm idempotency so reruns do not duplicate data, ask the platform PM for reviewer capacity and tell the customer before the history changes. See the backfill runbook.
The customer's sponsor asks you to lower the confidence threshold so the review queue is empty before Monday's review. What do you say?

Show answer
No. Below-threshold extractions would reach customer-visible output, quality would drop and the SLA would come under threat. Offer other options: throttle ingest, add reviewer capacity, or set the expectation about turnaround. See the HITL overflow runbook.

Takeaway

Runbooks are what let you leave, so write them for the customer's analyst, and test them by seeing whether the analyst can finish one without paging you.
Every runbook has the same six parts: symptom, severity, triage, likely causes by frequency, resolution paths and escalation.
Severity depends on how stale the data is and when business hours fall, and a quiet failure outranks a loud one.
On a quality alert, confirm the drop is real, find the class, sample failures and check for a recent model deploy before acting.
A backfill rewrites history, so check idempotency, cost and review capacity, and tell the customer first.
Do not loosen the review threshold to clear the queue; that trades a visible delay for hidden errors.
Roll a new document class out in steps from 10% to 50% to 100%, and add it to the SLA dashboard.

Runbooks cover what you can predict; the next chapter covers what to do when the problem sits outside the runbook or outside your control. → Escalation Playbook