Pipelines Under Deployment

What makes deployed pipelines different

Pipelines you build in an FDE deployment have constraints that internal pipelines don't:

• The data shape changes during deployment. You'll get new document classes in week 4 that nobody mentioned in week 1. Your pipeline must absorb new inputs without rewrites.
• Production lives at the customer. Sometimes your pipeline runs in your cloud feeding theirs; sometimes the opposite; sometimes both. Multi-region, multi-tenant considerations.
• You'll reprocess. A lot. A new extraction-model version drops, the customer wants the historical corpus reprocessed. Idempotency is mandatory.
• The customer's data team will read your code. Their analysts will run your queries, debug your DAGs, and eventually own them. Readability is a deliverable.
• SLAs apply. A pipeline failure isn't just an internal alert — it can be a contract breach.

Ingestion patterns

Three patterns you'll meet often:

1. Customer drop bucket

Customer drops files into an S3 / GCS / Azure Blob bucket the platform watches. Cheapest, easiest to set up, awkward when there are millions of docs. Used for initial historical loads and one-shot batches.

2. Connector / API pull

Pipeline pulls from the customer's source system — DocuSign, Concur, SAP Ariba, custom CLM. More work to set up (auth, paging, rate limits, schemas) but durable. Used for ongoing flow.

3. Event push

Customer's system fires a webhook when a new document is signed/uploaded. Real-time. Requires the customer to do dev work on their side. Used for time-sensitive flows (renewal triggers).

Pattern selection

• Historical backfill → drop bucket.
• Ongoing flow at moderate volume → connector pull on a schedule (daily, hourly).
• Real-time decisioning → event push.
• Most deployments end up with two of three.

Idempotency & reprocessing

Every stage of the pipeline should be idempotent — running it twice on the same input produces the same output. This sounds obvious; in practice most pipelines aren't built this way the first time, and fixing it later is painful.

The discipline

• Stable IDs. Every document, every extraction, every output row has a deterministic ID derived from inputs. document_id = hash of file contents + customer_id. extraction_id = document_id + model_version.
• Upserts, not appends. Re-running the same input updates rather than duplicates.
• Versioned outputs. Output rows carry the model version that produced them. New model version produces new rows; old rows stay for audit.
• Partition by ingest date AND processing date. Lets you reprocess "all documents ingested before March 1 with the new model" without touching the original ingest record.

-- Re-run extraction on all contracts where current extraction is below threshold
-- and re-extract with the new model version. Old rows preserved for audit.
INSERT INTO extractions (
  document_id, model_version, extracted_at, fields, confidence
)
SELECT
  d.document_id,
  'extract-v5.0.0' AS model_version,
  NOW() AS extracted_at,
  -- ... extraction columns
  -- (in practice, this would be a pipeline call, not pure SQL)
FROM documents d
LEFT JOIN extractions e
  ON e.document_id = d.document_id
 AND e.model_version = 'extract-v5.0.0'
WHERE e.document_id IS NULL  -- not yet processed with v5
  AND EXISTS (
    SELECT 1 FROM extractions e2
    WHERE e2.document_id = d.document_id
      AND e2.confidence < 0.85
  );

Versioning

FDE pipelines version everything:

• Pipeline version — Git SHA of the pipeline code that ran.
• Model version — extraction model identifier.
• Schema version — output table schema can evolve; row carries which schema applied at write time.
• Customer config version — per-customer extraction template / mapping rules. Treat as code, version, deploy.

The forensic value: when something looks wrong, you can reconstruct exactly which combination of pipeline + model + schema + config produced the suspect row.

Schema evolution

The customer's data shape will change. Your schema needs to absorb it without rewrites.

Patterns

• Strict typed columns + flexible JSON sidecar. Common pattern. Required fields are typed columns; everything else lives in a raw JSON column. New fields appear in the sidecar first, get promoted to typed columns once stable.
• Wide-table schema with nullable columns. Works for small variance. Breaks at scale.
• Per-document-class tables. Used when document classes have fundamentally different shapes (a master service agreement and an invoice don't have the same columns).

Migrations

Schema migrations in deployed pipelines need to be:

• Backward-compatible — old queries still work.
• Forward-deployable — you can deploy the migration before the code that uses it.
• Reversible — if the migration is wrong, you can roll back.

This means no DROP COLUMN in the same release as the code change. Migrate in two stages: add new column → backfill → deploy code → deprecate old column → drop in a later release.

Orchestration

The dominant tools you'll meet:

• Airflow — the incumbent. Verbose but battle-tested. Customers know it.
• dbt — for the SQL-transformation layer specifically. Standard for the warehouse step.
• Dagster / Prefect — newer, more flexible, Python-native. Less common in enterprise customer stacks but more pleasant to build with.
• Internal platform DAGs — many document-AI platforms have their own internal orchestration. As FDE-DE you'll often write pipelines that consume from the platform-owned DAG and produce into the customer's chosen tool (often Airflow).

The FDE-DE move

Don't impose your favorite orchestration tool on the customer. Use whatever they already have. If they have nothing, default to the simplest thing that works (often a scheduled Python script + dbt for the SQL piece) and only graduate to Airflow if scale demands it.

Observability

Three layers you must monitor:

1. Pipeline-level

Are DAGs running on schedule? Are runs succeeding? How long do they take? Standard data-platform metrics — push to whatever the customer's ops team uses (Datadog, PagerDuty, their internal Prometheus stack).

2. Data-quality-level

Are output row counts in expected bands? Are confidence scores stable? Are HITL queue depths within capacity? These are FDE-specific; standard pipeline observability won't surface them.

3. SLA-level

Are extraction-quality metrics above the contracted threshold per document class? A dashboard the customer sees, showing pass-rate by class, with anomalies flagged. The customer sees this dashboard regularly — that's a feature, not a problem.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. A reprocess job fails at 60% and you rerun it. What in the design makes that safe, and what choice of identifier would break it?

   Show answer

   Identifiers derived from inputs, such as a file hash with the customer, and a model version in the key, mean the rerun skips finished work and upserts the rest. Identifiers built from timestamps or random values make every rerun a new row and create duplicates. See idempotency and reprocessing.

2. In week four the customer sends a new document class with fields nobody mentioned. Where do the new fields land first, when are they promoted, and why not widen the table?

   Show answer

   They land in the flexible JSON column next to the typed core, and become typed columns once they prove stable. A wide table with many nullable columns works for small variance and fails at scale, and a separate table per class suits shapes that really differ. See schema evolution.

3. You must rename a column that customer dashboards read. Sequence the release.

   Show answer

   Add the new column, backfill it, deploy the code that uses it, deprecate the old one, and drop it in a later release. Each step stays backward compatible, forward deployable and reversible, so no single deploy can break a dashboard. See schema evolution.

4. The customer runs Airflow and you prefer Dagster. Who decides, and what evidence would change the answer?

   Show answer

   The customer decides, because their team must run it after handoff. Only an incompatibility with the workload, shown with evidence such as a scheduling limit you actually hit, justifies pushing back. Preference alone earns no trust. See orchestration.

5. A pipeline shows green runs for a month while its output is wrong. Which monitoring layers would have caught it, and why would the first not?

   Show answer

   Pipeline-level health only says tasks ran. Data-quality checks on row counts, schema drift and confidence distribution would catch it, as would an SLA view of accuracy by class. A green run proves execution, not correctness. See observability. What to track for the SLA layer is in monitoring.

6. The customer has four million historical files, a daily trickle of new ones, and wants renewal triggers within minutes of signing. Match each to an ingestion pattern.

   Show answer

   Historical files use a drop bucket, the daily flow uses a scheduled connector pull, and renewal triggers need event push from the customer's system. Most deployments end up with two of the three. See ingestion patterns.

Takeaway

• Deployed pipelines must absorb new inputs mid-deployment, survive frequent reprocessing, and be readable by the customer's team.
• Choose ingestion by need: drop bucket for backfill, connector pull for ongoing flow, event push for real time; most deployments use two.
• Idempotency comes from stable input-derived identifiers, upserts and versioned outputs, which turn reprocessing into routine work.
• Version the pipeline code, the model, the schema and the customer configuration so you can rebuild what produced any row.
• Keep a typed core with a flexible JSON column, and migrate in stages; never drop a column in the same release as the code change.
• Use the customer's orchestrator, and watch three layers: pipeline health, data quality and SLA metrics. A green run does not mean good data.

Next is the data model and SQL those pipelines produce and serve. → SQL & Contract Data Modeling

Further reading

• Best Practices — Apache Airflow documentation, accessed 2026. Writing idempotent tasks and DAGs that are safe to rerun.
• Configure incremental models — dbt documentation, accessed 2026. Upsert strategies for large tables and their trade-offs.
• MERGE — Snowflake documentation, accessed 2026. Upserts in the warehouse the customer is most likely to run.
• Upsert into a Delta Lake table using merge — Databricks documentation, accessed 2026. The same pattern on a lakehouse, including schema evolution options.
• Parallel Change — Martin Fowler, 2014. The expand, migrate, contract sequence behind staged schema changes.