Runbook Templates

# Runbook: Pipeline Failure
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- Datadog alert "ingestion DAG failed" or
- Friday status mentions "extraction did not run for last N days" or
- Customer reports stale data in dashboards

## Severity
- **Page immediately** if last successful run was > 24 hours ago
  AND business hours are within the next 4 hours.
- **Next business morning** if last successful run was > 24 hours ago
  outside business hours.
- **Note & investigate** if < 24 hours.

## Triage (first 10 minutes)
1. Open the orchestration UI ([link to Airflow / Dagster instance])
2. Look at the failed task — what's the error message?
3. Look at upstream task — did its data arrive?
4. Check the source system status page (CLM, ERP, SharePoint)
5. Note the last successful run timestamp

## Likely causes (order of frequency)
1. **Source system unavailable.** CLM had an outage; SharePoint
   permissions changed; API token expired.
2. **Network / auth.** Service account password rotated; VPN
   gateway issue; IP allowlist change.
3. **Upstream schema change.** Source added or removed a field;
   our parser doesn't handle it.
4. **Warehouse storage / quota.** Snowflake / BigQuery storage
   limit hit; load failed.
5. **Code regression.** Recent deploy broke something.

## Resolution paths
- Source unavailable → wait + verify when restored. Backfill when stable.
- Auth → rotate token / update service-account credentials.
  See "Auth refresh" section of [link to ops doc].
- Schema change → file ticket with FDE on-call; deploy parser
  update (link to repo).
- Warehouse quota → increase quota with [warehouse admin] or
  archive old partitions per [retention policy doc].
- Regression → revert the last release; file bug in [tracking system].

## Escalation
- If unresolved in 2 hours and a customer-visible impact:
  page [FDE on-call: name, contact].
- If platform-team work is needed:
  file [internal ticket template] tagged "FDE-customer-blocker."

# Runbook: Quality Alert
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- SLA dashboard shows accuracy below threshold on a document class.
- Customer reports specific extractions are wrong.
- Confidence-distribution alert (sudden shift in confidence pattern).

## Severity
- **Page immediately** if > 5pp below SLA and impacts customer-facing
  decisions (e.g., renewal alerts going to wrong agreements).
- **Same business day** if < 5pp below SLA but trending downward.
- **Investigate this week** if isolated incident, not trending.

## Triage (first 15 minutes)
1. Open the SLA dashboard. Confirm the regression is real
   (not a small-sample artifact).
2. Identify which document class is below SLA.
3. Sample 20 recent failures by eye. Pattern?
4. Check confidence-distribution dashboard for that class.
5. Check whether a recent model deploy preceded the drop.

## Likely causes
1. **Customer data drift.** New documents in the corpus have
   different layout / vocabulary than the trained distribution.
2. **Model regression.** Recent platform model deploy degraded
   on this class.
3. **HITL miscalibration.** Reviewers' bar shifted; what they
   accepted last month, they reject this month.
4. **Confidence drift.** Model is over-confident now; auto-approval
   threshold needs retuning.
5. **Ground-truth issue.** The gold set is wrong; the model is fine.

## Resolution paths
- Customer data drift → expand prompt / templates to cover the
  variance. Add to HITL queue temporarily while platform iterates.
- Model regression → revert to prior model version (link to
  versioned deploys). File platform bug.
- HITL miscalibration → recalibrate reviewers against gold set.
  Run inter-rater agreement check on calibration sample.
- Confidence drift → recompute reliability diagram (link to
  notebook). Tighten threshold per the new calibration.
- Gold-truth issue → audit the gold set. Update if confirmed.

## Communicate to customer
- If SLA is at risk: notify the customer data lead within 4 hours.
  Use this template:

> "Heads up — we noticed extraction quality on [class] trending
> toward the SLA line ([current pct] vs SLA of [threshold pct]).
> We've already done [containment]. We'll have a root-cause
> understanding by [date]. Won't impact your Monday review."

## Escalation
- Multi-day below SLA → page FDE on-call.
- Platform bug → file with customer evidence in [tracking system].

# Runbook: Schema Drift
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- Pipeline runs but downstream rows are missing fields.
- dbt test failure on uniqueness or not-null.
- New / removed field detected in source-system payload.

## Severity
- **Same business day** if pipeline is silently dropping data.
- **Next business day** if pipeline is failing loudly (failure
  is the safer mode; you have time).

## Triage
1. Find the dbt test that failed, or the row-count anomaly that
   surfaced the drift.
2. Sample 5 source records before vs after the drift point.
3. Identify the specific field(s) that changed.
4. Determine: added, removed, type-changed, or renamed?

## Likely causes
1. **Source-system upgrade** — vendor pushed new schema (ERP,
   CLM, procurement suite).
2. **Customer-side configuration change** — customer's admin
   added/removed a custom field.
3. **Authentication scope change** — a field we used to see is
   now restricted to a different role.

## Resolution paths
- Added field → update parser to read it; add to JSON sidecar
  if you're not sure whether to promote yet.
- Removed field → check if downstream depends on it.
  If yes, file customer-side ticket. If no, deprecate gracefully.
- Type change → cast at the parser layer; alert if cast fails.
- Renamed → update the field-mapping config (link to the file).

## Escalation
- Customer-side scope change → escalate to customer's data lead;
  cc IT / system admin.
- Vendor-driven upgrade → file with platform PM (this affects
  every customer using that vendor).

# Runbook: Backfill
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## When you need this
- New model version released; reprocess historical corpus.
- New document class onboarded; backfill for the new class.
- Bug fixed that affected a date range; re-extract.
- Customer adds new historical documents (e.g., post-acquisition).

## Pre-flight checklist
- [ ] Determine scope: which documents, which date range, which
      document classes?
- [ ] Estimate cost: rows × model cost × HITL volume.
- [ ] Plan HITL capacity: backfill HITL volume can swamp the queue.
- [ ] Communicate to customer: their analysts may see changes.
- [ ] Confirm idempotency: re-running won't duplicate data.

## Execution
1. Identify the run scope (SQL query against `documents` table).
2. Update model_version pin if applicable.
3. Trigger backfill via [orchestrator UI / CLI command].
4. Monitor HITL queue depth and reviewer throughput.
5. If queue overflows, throttle backfill rate.

## Validation
- Spot-check 20 random reprocessed documents.
- Run dbt tests on the affected schema.
- Compare aggregates against pre-backfill snapshot.

## Customer communication
- Backfills change historical data. Tell the customer when one runs
  (especially on dashboards they actively use).
- Template:
  > "We're reprocessing [N] documents from [date range] with our
  > latest extraction model. You may see updated values in your
  > [dashboard] starting [date]. We expect [accuracy improvement]
  > on average; flag anything that looks unexpected."

## Escalation
- HITL queue can't absorb the volume → contact platform PM
  for additional reviewer capacity.
- Validation fails → pause backfill; escalate to FDE.

# Runbook: New Document Class
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## When you need this
- Customer brings a new document type into scope (e.g., DPAs,
  NDAs, supplier code-of-conduct attachments).

## Pre-flight
- [ ] Confirm the class is in scope per working agreement.
      If not, scope discussion first.
- [ ] Gather 20+ representative samples from the customer.
- [ ] Identify the fields to extract per class.
- [ ] Define the SLA threshold for this class.

## Configuration
1. Add the document-class taxonomy entry: [link to config file]
2. Define field schema (which fields, types, tolerances): [link]
3. Configure extraction prompt / template: [link]
4. Set initial confidence threshold (default 0.85; calibrate after
   first 100 documents).

## Pilot
- Run on 100-document sample.
- Have customer-side analysts validate the output by eye.
- Measure field-level accuracy against their validation.
- Tune threshold; expand HITL coverage if needed.

## Production
- Roll out gradually: 10% → 50% → 100% over a week.
- Watch confidence distribution and HITL queue depth.
- Add the class to the SLA dashboard.
- Update working agreement to reflect the added class.

## Escalation
- Accuracy on pilot is materially below SLA target → file with
  platform team for extraction-model expansion.
- Customer wants the class in scope before pilot validates →
  escalate to AE (commercial conversation).

# Runbook: HITL Queue Overflow
For [Customer] · Maintained by [name] · Last updated YYYY-MM-DD

## Symptom
- HITL queue depth alert (queue grows faster than reviewers process).
- Customer-visible delay between ingest and final output.
- Reviewer-throughput metric below capacity.

## Severity
- **Same business day** if delay exceeds 24 hours (SLA likely
  affected).
- **Investigate next day** if delay 12–24 hours.

## Triage
1. Check queue depth trend over last 7 days.
2. Check reviewer throughput. Down? Up but volume up more?
3. Check confidence-distribution. Are more documents routing to
   HITL than expected?

## Likely causes
1. **Volume spike.** Customer ingested a backlog; HITL volume
   spikes proportionally.
2. **Confidence drift.** Model has shifted toward more uncertain
   predictions; more docs route to HITL than before.
3. **Reviewer capacity drop.** Reviewer team is short-staffed
   (vacation, training, attrition).
4. **Threshold misconfiguration.** Recent threshold tightening
   pushed too many docs to HITL.

## Resolution paths
- Volume spike → throttle ingest if customer can tolerate; add
  reviewer capacity if not.
- Confidence drift → recalibrate model; recompute thresholds.
- Reviewer capacity drop → escalate to reviewer-pool ops; consider
  contractor support if multi-week.
- Threshold misconfiguration → adjust threshold back; document why.

## Don't do this
- Don't relax HITL just to clear the queue. Below-threshold
  extractions hit customer-visible output; quality drops; SLA
  is now at risk.
- Don't promise customer faster turnaround without confirming
  with reviewer ops first.

## Escalation
- Persistent > 3 days → page FDE; consider temporary scope
  reduction conversation with customer.