Document AI Extraction — Deep Dive
A working reference on the extraction stack that turns unstructured contracts, invoices, and POs into structured data — OCR, layout-aware models, field extraction patterns, calibration, HITL economics, evaluation methodology, drift detection. Aimed at practitioners who design, debug, or operate production extraction systems backed by accuracy SLAs.
Who this is for
You're an engineer, ML practitioner, or technical PM working at or with a document-AI platform — building the extraction stack, integrating with a customer who depends on it, or debugging when accuracy slips. This guide is the working reference you return to when "the extraction is wrong, why?"
Section A · The extraction stack
00Start Here
Master index, how to read this guide, the framings worth carrying into every chapter.
01The Extraction Stack
Seven layers from raw file to structured row. What each layer does, where errors compound, who owns each piece in practice.
02OCR Layer in Depth
Native vs scanned PDFs, modern OCR engines (Textract, Document AI, Form Recognizer, open-source), failure modes, the pre-flight check every deployment needs.
03Layout-Aware Models
LayoutLM family, DocFormer, LiLT, DocLLM. Why spatial features matter, when to use which, what they're bad at.
Section B · Field extraction
04Field Extraction Patterns
Rule-based vs model-based vs hybrid; LLMs with structured output (JSON-mode, function-calling, Pydantic schemas); task-specific encoders.
05Output Schema & Provenance
Structured output design, provenance stamping (page + bbox), confidence per field, model versioning. The audit-ready output shape.
Section C · Quality & eval
06Confidence & Calibration
Reliability diagrams, isotonic and Platt scaling, why calibration matters for HITL economics, what miscalibration costs.
07HITL Economics
Queue design, reviewer capacity, inter-rater agreement, threshold tuning, the cost-per-extraction math.
08Eval Methodology
Ground truth sourcing, sample sizing for SLA measurement, per-field vs per-document accuracy, precision/recall under skew.
Section D · Operating it
09Drift Detection & Monitoring
Input drift (document distributions), output drift (confidence + extraction patterns), outcome drift (accuracy on labeled samples). What to alert on, what to surface to customers.
10Failure Modes & Debugging
The catalog of how extraction goes wrong in production — rotated pages, multi-column reflow, watermarks, schema variance — and the debugging protocol for each.