Document Extraction

The extraction stack

Document-AI platforms typically run a layered pipeline:

1. Pre-processing — file ingestion, format detection (PDF, DOCX, image), de-skewing, page splitting.
2. OCR — for image-based PDFs and scans. Modern stacks use Tesseract, AWS Textract, Google Document AI, Azure Form Recognizer, or a homegrown model.
3. Layout analysis — segment the document into regions (tables, paragraphs, headers, signature blocks). Layout-aware models (LayoutLM family, DocFormer, LiLT, or custom) consume both text and spatial features.
4. Field extraction — pull specific values: party names, effective date, renewal terms, payment terms, jurisdiction, signatures. Either rule-based (for templated docs) or model-based (LLM + structured-output, or task-specific encoders).
5. Normalization — map extracted strings to canonical entities (supplier dedup, currency normalization, date parsing).
6. Validation + HITL — confidence scoring, threshold-based human review, feedback into next training round.
7. Persistence — write the structured output to a database (Postgres, Snowflake, etc.) tied to the source document.

As FDE-DE, you don't typically train these models. You consume their output, surface failure modes back to platform, and wire results into the customer's stack. But you need to be able to reason about each layer.

OCR layer

The vocabulary worth knowing:

• Native PDF: text is stored as actual characters. Parse with pdfminer / pdfplumber — no OCR needed.
• Image-based PDF / scan: text is rendered as pixels. Needs OCR. Quality varies wildly with scan resolution and skew.
• Mixed PDF: native text + scanned attachments. Process page-by-page.
• Word docs / RTF: easy to parse but layout fidelity isn't perfect.

OCR failure modes

• Multi-column layouts read in the wrong order — text comes out as gibberish.
• Tables flatten — column structure lost.
• Handwritten signatures or annotations confuse the model.
• Rotated pages produce garbage.
• Watermarks bleed into text.

Layout-aware models

Modern document extraction uses models that jointly process text and spatial position — where words appear on the page, not just what they say. LayoutLM (Microsoft), LayoutLMv3, DocFormer, LiLT are the well-known family.

Why it matters: in a contract, the word "30 days" means different things if it's in the "payment terms" section vs the "termination notice" section. Pure text models can't distinguish; layout-aware models can.

For an FDE, the practical implication is: document classes that are visually consistent (same layout across many docs) extract well; visually idiosyncratic docs do not. The first thing to know about a new customer is how visually varied their corpus is.

Field extraction

Two patterns:

Rule-based extraction

Regex + pattern templates. Fast, deterministic, easy to debug. Works on templated documents. Brittle when layouts shift.

Model-based extraction

Either a task-specific encoder (fine-tuned for "extract effective date") or a general LLM with structured-output prompting. More robust to variance, slower, harder to debug. The norm for non-templated corpora.

The hybrid play

Most production stacks use both: rules for high-confidence, easy-pattern fields (signed date often appears in a predictable location); models for harder fields (renewal terms, indemnification clauses); HITL when neither is confident enough.

Output schema

The extraction output should be:

• Structured — JSON or row-level records with explicit field names.
• Provenance-stamped — every extracted value carries which document, which page, which coordinates it came from. Auditability matters.
• Confidence-scored — every field has a model-confidence score that downstream pipelines can threshold on.
• Versioned — which model version produced this output, in case you need to reprocess.

{
  "document_id": "msa-acme-2024-038.pdf",
  "doc_class": "master_service_agreement",
  "extracted_at": "2026-05-26T14:32:01Z",
  "model_version": "extract-v4.2.1",
  "fields": {
    "parties": {
      "value": ["Acme Corp", "Customer Inc"],
      "confidence": 0.97,
      "provenance": {"page": 1, "bbox": [120, 240, 480, 280]}
    },
    "effective_date": {
      "value": "2024-03-15",
      "confidence": 0.94,
      "provenance": {"page": 1, "bbox": [120, 310, 290, 330]}
    },
    "renewal_term": {
      "value": "auto-renew, 12 months, 60 days notice",
      "confidence": 0.62,
      "provenance": {"page": 8, "bbox": [80, 540, 520, 615]},
      "review_status": "pending_human"
    }
  }
}

Human-in-the-loop

The differentiator for SLA-backed platforms. Confidence-thresholded extractions go to human reviewers (often labeled "analysts") who either confirm, correct, or flag.

What HITL solves

• Low-confidence model outputs become high-confidence labeled data.
• Customer-visible accuracy stays above SLA even when the model is wrong.
• Reviewer corrections become training signal for the next model.

What FDE-DE needs to know

• The HITL queue is a real piece of infrastructure with capacity constraints. Pushing too many docs to review breaks the queue.
• Inter-rater agreement on borderline fields can be 70–85%. Calibrate the customer's expectations.
• Some customers can't tolerate HITL — they have data sensitivity that prohibits external review. The platform either has on-prem reviewers, customer-side reviewers, or doesn't sell to them.

Failure modes you'll diagnose

• OCR quality below threshold — customer's scans are poor; extraction is downstream-noisy. Diagnose with a quality sample; talk to customer about re-scanning or relax SLA.
• Layout class regression — extraction quality drops on a specific document class because the platform model was updated. File platform bug; deploy temporary HITL workaround.
• Schema variance — customer's documents from 2008 use different terminology than 2024 documents. Talk to customer's procurement team; expand the prompt/model coverage.
• Confidence-score drift — model is more confident than it should be on a certain field type. Recalibrate threshold; surface to platform.
• Provenance mismatch — extracted value's bounding box doesn't match the source text on the page. Usually an OCR-segmentation issue. File bug; check if rotated pages or multi-column layout.

Where the FDE fits in the extraction stack

You don't train models. You're not building the OCR. You don't ship LayoutLM v2.

What you do own:

• Decide which customer document classes get extraction and in what order.
• Configure extraction templates / prompts / rules for the customer's specific patterns.
• Wire extraction output into the customer's data layer.
• Operate the HITL queue against the customer's deadline.
• Surface failure modes to platform with customer-evidence write-ups.
• Set expectations with the customer about what's achievable when.

The interview implication: you should be able to narrate the stack credibly without claiming you'd train the model yourself. The bar is "I understand how this works, I know what's hard, I can debug a failure, I know what to escalate vs fix locally."

Interview probes