Where to grow from here

Extensions & Next Steps

What you've built so far is a minimal end-to-end pipeline. This chapter is the roadmap of where to extend it toward production — and which deep-dive chapters to read along the way.

What you've built

  • Idempotent ingestion: PDFs in a folder become rows in documents.
  • Native + OCR text extraction.
  • LLM-based field extraction with Pydantic-validated structured output.
  • Persistence to a realistic contract data model — documents, suppliers, agreements, extractions, links.
  • An evaluation harness with a small gold set and a Markdown report.
  • A FastAPI service exposing the data.

It's ~1500 lines of Python that demonstrate the full loop. Not production. Useful as a foundation.

Quality extensions

Add citations & provenance

The schema already supports citations; the extraction prompt is set up to produce them. Wire the citations through to persistence (store as part of fields JSONB on the extraction record). Surface provenance in the API response. Validate citations by matching against OCR text — failed matches signal hallucination.

Confidence calibration

Self-reported LLM confidence isn't calibrated. Use the gold set to fit a calibration mapping (isotonic regression on raw vs actual accuracy). Apply at extraction time. See Doc AI §06.

Per-field threshold tuning

Different fields have different reliability profiles. Tune the threshold per (doc_class, field) instead of a single global threshold. Use the gold set to set initial thresholds; refit as you accumulate HITL data.

Layout-aware extraction

For dense or non-templated documents, native pdfplumber text loses layout. Swap the OCR step for a cloud service that returns layout regions (AWS Textract Analyze Document, Google Document AI Layout). Feed structured regions to the LLM instead of flat text. See Doc AI §03.

Scale extensions

Parallel processing

The current pipeline processes documents serially. Wrap process with a worker pool (asyncio + httpx for parallel LLM calls; rate-limit-aware). 10× throughput typical.

Move to Snowflake / BigQuery

Postgres works for development. For customer-deployment scale, deliver into the customer's warehouse via dbt models. See Doc AI §08 and the Data Model Reference §09.

Incremental processing

Schedule the pipeline (Airflow / Dagster). Process only new documents (where extracted_at IS NULL for the current model_version). The pattern is in the Pipelines Under Deployment chapter of the Doc AI deep dive.

Caching

OCR + LLM calls on the same document repeatedly should hit cache. Use the file_hash as cache key for OCR results; (file_hash, model_version) for extraction results.

Multi-class extraction

You built an MSA extractor. Add others:

  • SOW extractor: deliverables, term, value, parent MSA reference.
  • Amendment extractor: what changed (term extension, price change, scope add). Link to parent MSA.
  • Invoice extractor: vendor, invoice number, line items, total, PO reference. Populate the spend table.
  • NDA extractor: parties, term, jurisdiction.

Each gets its own Pydantic schema + prompt + persistence path. Reuse the OCR + LLM infrastructure; only the schema and persistence logic differ per class.

HITL workflow

The biggest functional gap. To approach an SLA-backed product:

  1. HITL queue: rows where confidence < threshold get a review_status = pending flag.
  2. Reviewer UI: a simple internal app showing the source document, the extracted value, the proposed correction. Could be FastAPI + a minimal React frontend, or a Streamlit / Gradio quick build.
  3. Correction logging: reviewer corrections become labeled data. Append to the eval gold set; refit calibration.
  4. SLA dashboard: a customer-facing view of accuracy by class with the SLA threshold drawn as a line.

See Doc AI §07 for the economics and design.

Operations

Multi-tenant

Add row-level security on every table (customer_id is already in the schema). Test cross-tenant isolation explicitly.

Auth

API has no auth today. Add OAuth2 / JWT with FastAPI's OAuth2PasswordBearer. For an embedded-deployment scenario, integrate with the customer's IdP (Okta SAML, etc.).

Observability

Structured logging with correlation IDs. Metrics: pipeline latency, extraction error rate, confidence distribution, HITL queue depth. Push to Datadog / Prometheus.

Error handling & retries

LLM calls fail intermittently — rate limits, transient network. Wrap with exponential backoff. Use tenacity or a similar library.

Schema migrations

Alembic is set up. Use it. Two-phase migrations for column renames; backward-compatible additions.

Pointers to the deep dive

For each extension, the relevant deep-dive chapter:

The build-along is the smallest end-to-end loop. The other guides are where you grow each piece toward production.