Pipelines Under Deployment

What makes deployed pipelines different

Pipelines you build in an FDE deployment have constraints that internal pipelines don't:

• The data shape changes during deployment. You'll get new document classes in week 4 that nobody mentioned in week 1. Your pipeline must absorb new inputs without rewrites.
• Production lives at the customer. Sometimes your pipeline runs in your cloud feeding theirs; sometimes the opposite; sometimes both. Multi-region, multi-tenant considerations.
• You'll reprocess. A lot. A new extraction-model version drops, the customer wants the historical corpus reprocessed. Idempotency is mandatory.
• The customer's data team will read your code. Their analysts will run your queries, debug your DAGs, and eventually own them. Readability is a deliverable.
• SLAs apply. A pipeline failure isn't just an internal alert — it can be a contract breach.

Ingestion patterns

Three patterns you'll meet often:

1. Customer drop bucket

Customer drops files into an S3 / GCS / Azure Blob bucket the platform watches. Cheapest, easiest to set up, awkward when there are millions of docs. Used for initial historical loads and one-shot batches.

2. Connector / API pull

Pipeline pulls from the customer's source system — DocuSign, Concur, SAP Ariba, custom CLM. More work to set up (auth, paging, rate limits, schemas) but durable. Used for ongoing flow.

3. Event push

Customer's system fires a webhook when a new document is signed/uploaded. Real-time. Requires the customer to do dev work on their side. Used for time-sensitive flows (renewal triggers).

Pattern selection

• Historical backfill → drop bucket.
• Ongoing flow at moderate volume → connector pull on a schedule (daily, hourly).
• Real-time decisioning → event push.
• Most deployments end up with two of three.

Idempotency & reprocessing

Every stage of the pipeline should be idempotent — running it twice on the same input produces the same output. This sounds obvious; in practice most pipelines aren't built this way the first time, and fixing it later is painful.

The discipline

• Stable IDs. Every document, every extraction, every output row has a deterministic ID derived from inputs. document_id = hash of file contents + customer_id. extraction_id = document_id + model_version.
• Upserts, not appends. Re-running the same input updates rather than duplicates.
• Versioned outputs. Output rows carry the model version that produced them. New model version produces new rows; old rows stay for audit.
• Partition by ingest date AND processing date. Lets you reprocess "all documents ingested before March 1 with the new model" without touching the original ingest record.

-- Re-run extraction on all contracts where current extraction is below threshold
-- and re-extract with the new model version. Old rows preserved for audit.
INSERT INTO extractions (
  document_id, model_version, extracted_at, fields, confidence
)
SELECT
  d.document_id,
  'extract-v5.0.0' AS model_version,
  NOW() AS extracted_at,
  -- ... extraction columns
  -- (in practice, this would be a pipeline call, not pure SQL)
FROM documents d
LEFT JOIN extractions e
  ON e.document_id = d.document_id
 AND e.model_version = 'extract-v5.0.0'
WHERE e.document_id IS NULL  -- not yet processed with v5
  AND EXISTS (
    SELECT 1 FROM extractions e2
    WHERE e2.document_id = d.document_id
      AND e2.confidence < 0.85
  );

Versioning

FDE pipelines version everything:

• Pipeline version — Git SHA of the pipeline code that ran.
• Model version — extraction model identifier.
• Schema version — output table schema can evolve; row carries which schema applied at write time.
• Customer config version — per-customer extraction template / mapping rules. Treat as code, version, deploy.

The forensic value: when something looks wrong, you can reconstruct exactly which combination of pipeline + model + schema + config produced the suspect row.

Schema evolution

The customer's data shape will change. Your schema needs to absorb it without rewrites.

Patterns

• Strict typed columns + flexible JSON sidecar. Common pattern. Required fields are typed columns; everything else lives in a raw JSON column. New fields appear in the sidecar first, get promoted to typed columns once stable.
• Wide-table schema with nullable columns. Works for small variance. Breaks at scale.
• Per-document-class tables. Used when document classes have fundamentally different shapes (a master service agreement and an invoice don't have the same columns).

Migrations

Schema migrations in deployed pipelines need to be:

• Backward-compatible — old queries still work.
• Forward-deployable — you can deploy the migration before the code that uses it.
• Reversible — if the migration is wrong, you can roll back.

This means no DROP COLUMN in the same release as the code change. Migrate in two stages: add new column → backfill → deploy code → deprecate old column → drop in a later release.

Orchestration

The dominant tools you'll meet:

• Airflow — the incumbent. Verbose but battle-tested. Customers know it.
• dbt — for the SQL-transformation layer specifically. Standard for the warehouse step.
• Dagster / Prefect — newer, more flexible, Python-native. Less common in enterprise customer stacks but more pleasant to build with.
• Internal platform DAGs — many document-AI platforms have their own internal orchestration. As FDE-DE you'll often write pipelines that consume from the platform-owned DAG and produce into the customer's chosen tool (often Airflow).

The FDE-DE move

Don't impose your favorite orchestration tool on the customer. Use whatever they already have. If they have nothing, default to the simplest thing that works (often a scheduled Python script + dbt for the SQL piece) and only graduate to Airflow if scale demands it.

Observability

Three layers you must monitor:

1. Pipeline-level

Are DAGs running on schedule? Are runs succeeding? How long do they take? Standard data-platform metrics — push to whatever the customer's ops team uses (Datadog, PagerDuty, their internal Prometheus stack).

2. Data-quality-level

Are output row counts in expected bands? Are confidence scores stable? Are HITL queue depths within capacity? These are FDE-specific; standard pipeline observability won't surface them.

3. SLA-level

Are extraction-quality metrics above the contracted threshold per document class? A dashboard the customer sees, showing pass-rate by class, with anomalies flagged. The customer sees this dashboard regularly — that's a feature, not a problem.

Interview probes