Section A · The stack

OCR Layer in Depth

Native vs scanned PDFs, the modern OCR engines, what they output and what they break on, and the preflight check every customer deployment should run before promising an extraction SLA.

Native vs scanned PDFs

The first decision in OCR is whether you need OCR at all. PDFs come in three flavors:

Native PDF (text-layer present)

Text is stored as actual character data — generated by Word, Adobe, or another authoring tool. Parse directly with pdfplumber, pdfminer.six, or PyMuPDF. No OCR needed. Output is fast, accurate, and includes coordinates for free.

Image-based PDF (scan or photo)

Text is rendered as pixels. Requires OCR. Quality depends on scan resolution, skew, and image cleanliness.

Mixed PDF

Some pages native, some scanned. Common for contracts with scanned signature pages appended to native documents. Process page-by-page: detect format per page, route appropriately.

How to detect

detect_pdf_type.py

import pdfplumber

def page_is_native(page) -> bool:
    """Native if the page has embedded text characters."""
    # pdfplumber returns the text layer; empty/whitespace = no native text
    text = page.extract_text() or ""
    return len(text.strip()) > 20  # threshold avoids false positives on stubs

def classify_pdf(path: str) -> str:
    """Returns 'native', 'image', or 'mixed'."""
    with pdfplumber.open(path) as pdf:
        flags = [page_is_native(p) for p in pdf.pages]
    if all(flags): return "native"
    if not any(flags): return "image"
    return "mixed"

Threshold of 20 characters per page avoids classifying near-empty pages (cover sheets, blank dividers) as native; tune for your corpus.

OCR engines

The major options grouped by cost and capability:

Cloud OCR (recommended default)

AWS Textract — strong general OCR; "Analyze Document" variant returns layout + tables + key-value pairs. Pricing scales per-page; volume discounts at scale. Often the default for AWS-shop customers.
Google Document AI — formerly Document Understanding AI. Strong on form-heavy documents. Custom processor training available for specific document types.
Azure AI Document Intelligence — formerly Form Recognizer. Strong on layout-aware extraction; competitive with Textract on general OCR.

Open-source

Tesseract — the workhorse. 30+ years old, fully open. Reasonable on clean, single-column documents; struggles on complex layouts. Run with --psm mode tuned per document type.
PaddleOCR — Baidu-developed; strong on Asian-language documents; includes layout-aware variant (PP-Structure).
EasyOCR — newer Python-native; good for prototyping; less battle-tested at scale.

Specialty / legacy

ABBYY FineReader — legacy strong-print quality; common in legal-tech and government document workflows.
Adobe Document Cloud — Acrobat-flavored OCR; common when customers already license Adobe enterprise.

Modern alternatives

Vision-capable LLMs (Claude with vision, GPT-4o, Gemini) can perform OCR + extraction in a single step on small documents. Faster prototyping; cost and latency are not competitive with dedicated OCR at corpus scale.
Nougat, Donut — research-grade end-to-end document-understanding models. Useful in narrow domains (academic papers, receipts); not general-purpose yet.

The choice that usually wins

For a contract-intelligence platform, the default stack is: native-PDF parsing via pdfplumber + OCR on scanned pages via Textract or Document AI. Open-source OCR only when cost or data-residency forces it. The cloud services' quality advantage on real-world document variance is large enough to justify the per-page cost for most customers.

Output shape

OCR output should always include three things per recognized text:

Text string — what the engine recognized.
Bounding box — (x, y, width, height) in page coordinates. Required for layout analysis downstream.
Confidence — per-word or per-token confidence from the OCR engine.

Cloud services produce hierarchical output (page → block → line → word) with bounding boxes at each level. Use the line level for most extraction logic; the word level when you need precise spans.

example: Textract Block output (abbreviated)

{
  "BlockType": "LINE",
  "Text": "This Master Service Agreement (the \"Agreement\") is entered into",
  "Confidence": 99.4,
  "Geometry": {
    "BoundingBox": {"Width": 0.521, "Height": 0.014, "Left": 0.131, "Top": 0.231},
    "Polygon": [...]
  },
  "Page": 1,
  "Relationships": [{"Type": "CHILD", "Ids": ["word-id-1", "word-id-2", ...]}]
}

BoundingBox coordinates are normalized (0.0–1.0) on AWS; absolute pixel coordinates on others. Normalize before further use.

Failure modes

The catalog of how OCR breaks. Most appear in real customer corpora; the question is which.

Multi-column reading order

Documents laid out in columns can be read row-by-row (correct) or top-to-bottom column-by-column (wrong). Different OCR engines have different defaults; some let you configure. Tesseract --psm 1 attempts automatic page-segmentation; --psm 4 assumes single column.

Table flattening

Tables can come out as cleanly-structured cells (cloud services with layout) or as flat text where row/column relationships are lost (basic OCR). For contract tables (pricing schedules, deliverable lists), the difference is critical.

Rotation

Even 1–2° rotation degrades OCR quality. 90°/180°/270° rotation usually fails entirely without explicit rotation handling. Preprocessing should detect and correct.

Watermarks & stamps

"DRAFT" / "CONFIDENTIAL" / company seals overlap with text; OCR may include them as content or partially overlap with the real text below. Mitigate with image-preprocessing (HSV filtering on common watermark colors) or post-process to remove known watermark phrases.

Handwriting

Annotations, signatures, notes in margins. Most OCR engines have a separate "handwriting" mode (Textract, Document AI); enable explicitly when expected.

Low resolution

Below ~200 DPI, OCR quality degrades. Upsampling can help marginally; the right fix is requesting better scans.

Compressed PDFs / artifacts

Some PDFs render fine in Acrobat but compress text into image blocks. Detect by checking native-text presence per page; route to OCR.

Languages not configured

Tesseract requires language packs; cloud services support most major languages but require explicit language hints. Multi-language documents (English contract with German appendix) need per-page or per-region detection.

The OCR preflight

Before signing an SLA with a new customer, run an OCR quality preflight on a representative corpus sample. The 30-minute investment in discovery prevents weeks of remediation later.

ocr_preflight.py

"""
Sample 50 documents from the customer's corpus, run OCR, eyeball results.
"""
import random
from pathlib import Path

def run_preflight(corpus_dir: Path, sample_size: int = 50):
    pdfs = list(corpus_dir.glob("**/*.pdf"))
    sample = random.sample(pdfs, min(sample_size, len(pdfs)))

    classifications = {"native": [], "image": [], "mixed": []}
    for pdf_path in sample:
        kind = classify_pdf(pdf_path)
        classifications[kind].append(pdf_path)

    print(f"Native PDFs:  {len(classifications['native'])}")
    print(f"Image PDFs:   {len(classifications['image'])}")
    print(f"Mixed PDFs:   {len(classifications['mixed'])}")

    # For image PDFs, run OCR and produce a quality report
    for pdf_path in classifications["image"] + classifications["mixed"]:
        ocr_result = run_textract(pdf_path)
        avg_confidence = sum(b["Confidence"] for b in ocr_result["Blocks"]
                             if b["BlockType"] == "LINE") / max(1, sum(1 for b in ocr_result["Blocks"] if b["BlockType"] == "LINE"))
        write_report(pdf_path, ocr_result, avg_confidence)

    # Spot-check by eye: open the first 5 reports, compare to source
    print("Open data/preflight/report-*.html in the browser and compare side-by-side.")

What to look for

What fraction of the corpus is image-based? Sets the OCR cost.
Average OCR confidence per page? Below 90% means trouble.
How often does multi-column reading order break?
Tables — do they extract as structured cells or flat text?
Languages other than English?
Watermarks, stamps, handwritten annotations — frequency?

The report becomes evidence in the working-agreement conversation. If 30% of the corpus is below OCR confidence threshold, the SLA needs a carve-out for that subset.

Quality measurement

OCR quality has two flavors:

Engine-reported confidence

Every modern OCR engine produces per-word confidence. Aggregate (mean, p95) per page. Useful for triage and HITL routing.

Ground-truth comparison

Label a sample of pages by hand or via dual annotation. Score OCR output against the ground truth. Standard metrics:

CER (Character Error Rate): edit distance between OCR text and ground truth, normalized by length. Lower is better.
WER (Word Error Rate): same but word-level.
Field-level accuracy: did OCR get the specific values you care about right? This matters more than raw CER/WER for downstream extraction.

For contract intelligence, field-level matters most: a 5% CER might be fine if the parties, dates, and key terms are all readable. A 2% CER is unacceptable if the 2% concentrates in dollar amounts.

Cost & throughput

OCR is a meaningful line item at scale.

Cloud OCR cost: roughly $1.50–$15 per 1000 pages depending on service and "analyze" tier. Layout-aware variants are 3–10× the base OCR rate.
Throughput: cloud services are roughly 1–5 pages/second per request; parallelize across requests for higher throughput. Batch endpoints (Textract async, Document AI batch) for cost optimization on large historical backfills.
Caching: same document re-OCRed should hit cache. Use file-content hash as cache key.

For a customer with a 50,000-document corpus and 90% image-based PDFs at ~10 pages each = 450,000 pages. At $5/1000 pages, the OCR bill for historical backfill is $2,250. Ongoing flow at 500 new docs/month with similar shape: ~$22/month. Most customers don't blink at OCR cost; the cost discussion is usually about the LLM-extraction step downstream.

✓ Check yourself

Answer from memory before opening each one.

A 40-page contract PDF returns text on 38 pages and nothing on the two appended signature pages. How should the pipeline handle it?

Show answer
Classify the file as mixed and decide per page: parse the 38 native pages directly and send only the two scanned pages to OCR. Treating the whole file as one kind either wastes OCR spend or loses the signature pages. See native versus scanned PDFs.
A scanned page carries a native-text footer stamp, "ACME-000123 CONFIDENTIAL", that sits over the scan. The 20-character test calls the page native. What happens, and what extra check catches it?

Show answer
The page is parsed for its footer only, so the body is never OCRed and the extractor sees an almost empty page. Add a check that compares the text with the page area, or detects a page-sized image under the text, and send such pages to OCR as well. See native versus scanned PDFs.
Boxes from Textract and from another vendor go onto the same highlight overlay. The other vendor's boxes land correctly and all of Textract's sit in the top-left corner. Why?

Show answer
Textract returns boxes as ratios from 0 to 1 of the page, while the other engine returns absolute units. Multiply the ratios by the page width and height, or normalise everything to one system as the first step after OCR. See output shape.
An OCR run shows 2% character error, and the customer still says the dollar amounts are wrong. How can both be true, and what do you measure instead?

Show answer
Character error averages over all text, so a few digit errors inside amounts hide among thousands of correct words. Measure field-level accuracy on the values the downstream step needs. A higher character error rate that spares the amounts beats a lower one that does not. See quality measurement.
A customer has 120,000 documents, 80% image-based at 8 pages each. Estimate the backfill OCR bill at $5 per 1,000 pages, and say whether that number should decide the engine.

Show answer
That is 96,000 scanned documents, 768,000 pages, so about $3,840 at the base rate; a layout tier at 3 to 10 times the rate lands between roughly $11,500 and $38,400. Even the top figure is small beside the LLM step, so choose the engine on accuracy for this corpus. See cost and throughput.
Your preflight shows 30% of the sample below the OCR confidence threshold, and the customer wants a blanket 95% accuracy SLA. What do you do?

Show answer
Do not sign the blanket figure. Show the report, carve out or separately price the low-quality subset, and ask for better scans of it. The preflight is evidence that the subset cannot meet the same bar. See the OCR preflight.

Takeaway

Decide per page whether OCR is needed, because mixed PDFs are common and native pages parse exactly.
Every OCR result must carry text, a bounding box and a confidence, and boxes must be put in one coordinate system.
The default stack is native parsing plus a cloud OCR service for scans; use open-source engines when cost or data residency forces it.
The failure modes are corpus-specific, so sample the customer's files before you promise an SLA.
Judge OCR by accuracy on the fields you extract, not by character error rate alone.
OCR cost is small beside the LLM step; cache results by file hash so no page is paid for twice.

Words and boxes are in hand; the next chapter covers models that read both together. → Layout-Aware Models