Extensions & Next Steps

What you've built

• Idempotent ingestion: PDFs in a folder become rows in documents.
• Native + OCR text extraction.
• LLM-based field extraction with Pydantic-validated structured output.
• Persistence to a realistic contract data model — documents, suppliers, agreements, extractions, links.
• An evaluation harness with a small gold set and a Markdown report.
• A FastAPI service exposing the data.

It's ~1500 lines of Python that demonstrate the full loop. Not production. Useful as a foundation.

Quality extensions

Add citations & provenance

The schema already supports citations; the extraction prompt is set up to produce them. Wire the citations through to persistence (store as part of fields JSONB on the extraction record). Surface provenance in the API response. Validate citations by matching against OCR text — failed matches signal hallucination.

Confidence calibration

Self-reported LLM confidence isn't calibrated. Use the gold set to fit a calibration mapping (isotonic regression on raw vs actual accuracy). Apply at extraction time. See Doc AI §06.

Per-field threshold tuning

Different fields have different reliability profiles. Tune the threshold per (doc_class, field) instead of a single global threshold. Use the gold set to set initial thresholds; refit as you accumulate HITL data.

Layout-aware extraction

For dense or non-templated documents, native pdfplumber text loses layout. Swap the OCR step for a cloud service that returns layout regions (AWS Textract Analyze Document, Google Document AI Layout). Feed structured regions to the LLM instead of flat text. See Doc AI §03.

Scale extensions

Parallel processing

The current pipeline processes documents serially. Wrap process with a worker pool (asyncio + httpx for parallel LLM calls; rate-limit-aware). 10× throughput typical.

Move to Snowflake / BigQuery

Postgres works for development. For customer-deployment scale, deliver into the customer's warehouse via dbt models. See Doc AI §08 and the Data Model Reference §09.

Incremental processing

Schedule the pipeline (Airflow / Dagster). Process only new documents (where extracted_at IS NULL for the current model_version). The pattern is in the Pipelines Under Deployment chapter of the Doc AI deep dive.

Caching

OCR + LLM calls on the same document repeatedly should hit cache. Use the file_hash as cache key for OCR results; (file_hash, model_version) for extraction results.

Multi-class extraction

You built an MSA extractor. Add others:

• SOW extractor: deliverables, term, value, parent MSA reference.
• Amendment extractor: what changed (term extension, price change, scope add). Link to parent MSA.
• Invoice extractor: vendor, invoice number, line items, total, PO reference. Populate the spend table.
• NDA extractor: parties, term, jurisdiction.

Each gets its own Pydantic schema + prompt + persistence path. Reuse the OCR + LLM infrastructure; only the schema and persistence logic differ per class.

HITL workflow

The biggest functional gap. To approach an SLA-backed product:

1. HITL queue: rows where confidence < threshold get a review_status = pending flag.
2. Reviewer UI: a simple internal app showing the source document, the extracted value, the proposed correction. Could be FastAPI + a minimal React frontend, or a Streamlit / Gradio quick build.
3. Correction logging: reviewer corrections become labeled data. Append to the eval gold set; refit calibration.
4. SLA dashboard: a customer-facing view of accuracy by class with the SLA threshold drawn as a line.

See Doc AI §07 for the economics and design.

Operations

Multi-tenant

Add row-level security on every table (customer_id is already in the schema). Test cross-tenant isolation explicitly.

Auth

API has no auth today. Add OAuth2 / JWT with FastAPI's OAuth2PasswordBearer. For an embedded-deployment scenario, integrate with the customer's IdP (Okta SAML, etc.).

Observability

Structured logging with correlation IDs. Metrics: pipeline latency, extraction error rate, confidence distribution, HITL queue depth. Push to Datadog / Prometheus.

Error handling & retries

LLM calls fail intermittently — rate limits, transient network. Wrap with exponential backoff. Use tenacity or a similar library.

Schema migrations

Alembic is set up. Use it. Two-phase migrations for column renames; backward-compatible additions.

Pointers to the deep dive

For each extension, the relevant deep-dive chapter:

• Extraction quality, calibration, HITL: Document AI Extraction — Deep Dive.
• Data modeling decisions, SQL patterns, schema evolution: Contract Data Model — Reference.
• Customer-deployment operating model: The FDE-DE Playbook.
• Enterprise system integration: Enterprise Integration Atlas.

The build-along is the smallest end-to-end loop. The other guides are where you grow each piece toward production.

✓ Check yourself

Answer from memory before opening each one.

1. Your eval shows renewal_terms at 71%, and the misses cluster on contracts where the renewal clause sits in a table. You can do only one of confidence calibration or layout-aware extraction first. Which, and why?

   Show answer

   Layout-aware extraction. Calibration maps stated confidence onto observed accuracy, which improves which items you route to review, but it leaves the answers unchanged. Flattened table text is an input problem, so fix it upstream. Calibrate after the inputs are sound. See Quality extensions.

2. A colleague fits an isotonic calibration on the 7-document gold set, about 35 confidence and correctness pairs. Do you accept the result?

   Show answer

   No. Isotonic regression is a step function with many free values, and 35 points will fit noise. Gather several hundred labelled fields, ideally corrections from the review queue, before you fit, and fit per field only when each field has enough points. See Quality extensions.

3. The caching advice keys OCR results by file_hash and extraction results by (file_hash, model_version). You edit the prompt and keep the same model. What does the extraction cache return?

   Show answer

   The old result, because nothing in the key changed. OCR output depends only on the bytes, but an extraction also depends on the prompt, schema and pipeline version. Put a pipeline or prompt version in the key. See Scale extensions.

4. You add a worker pool to process and raise concurrency from 1 to 32. The run gets slower in places and some documents fail. What is the likely limit, and what do you add?

   Show answer

   The provider's rate limits. Past the allowed requests or tokens per minute, calls return 429 errors, and retries add load. Cap concurrency to the limit, retry only transient errors such as 429 and 5xx with backoff and jitter, and never retry a validation failure unchanged. See Scale extensions.

5. You enable row-level security on every table and add a policy on customer_id, yet the API, connected as the compose file's ci user, still returns all tenants. Why?

   Show answer

   The image creates ci as a superuser and it owns the tables, and superusers bypass row security, as do owners unless the table forces it. Connect as a separate role that does not own the tables and has no bypass, and set the tenant per transaction. Test with a second tenant. See Operations.

Takeaway

• Fix the failure the eval shows: layout problems are an input fix, and calibration only changes which answers get reviewed.
• Calibration needs hundreds of labelled fields; a 7-document gold set is enough to measure but not to fit.
• A cache key must include everything that changes the output, so a prompt edit needs a new version in the key.
• Parallel workers are bounded by provider rate limits, and retries should cover only transient errors.
• Row-level security protects nothing when the application connects as an owner or superuser, so test isolation with a second tenant.
• The review queue is the missing piece for a service-level promise, and its corrections feed the gold set and the calibration.

Each extension above has a full treatment in the deep dive on extraction, calibration, review economics and evaluation. → Document AI Extraction — Deep Dive

Further reading

• On Calibration of Modern Neural Networks — Guo et al., 2017. Why model confidence drifts from accuracy and how to correct it.
• Probability calibration — scikit-learn docs, accessed 2026. Isotonic and sigmoid calibration, with reliability curves.
• Row Security Policies — PostgreSQL 16 docs. Policies, and which roles bypass them.
• Rate limits — Anthropic docs, accessed 2026. How limits are measured and what a 429 means.
• Errors — Anthropic docs, accessed 2026. Which errors are transient and which are your request's fault.
• Tenacity — Tenacity docs, accessed 2026. Retry, backoff and jitter as decorators.