Field Extraction Patterns

The architectural decision

You're going from "this region of this page contains text about renewal terms" to "renewal_term: 'auto-renew, 12 months, 60 days notice'." Four architectural choices, each with tradeoffs:

Most production contract-intelligence systems run hybrid. Rules for the easy 60% of fields (effective_date when consistently formatted, parties from a signature-block pattern), models for the hard 40%.

Rule-based extraction

Regex, pattern templates, anchor-relative position lookups. Brittle on non-templated documents; fast and debuggable on consistent ones.

When it works

• Customer's documents follow a consistent template (e.g., supplier agreements all generated from one MSA template).
• Fields appear in predictable positions (date below the title, signature block at end of last page).
• Field values follow regex-able patterns (USD amounts, ISO dates).

Example: extracting effective date from a templated MSA

import re
from datetime import datetime

EFFECTIVE_DATE_PATTERNS = [
    r"effective\s+(?:as\s+of\s+|date[:\s]+)([A-Z][a-z]+\s+\d{1,2},?\s+\d{4})",
    r"effective\s+(?:as\s+of\s+|date[:\s]+)(\d{1,2}/\d{1,2}/\d{4})",
    r"entered\s+into\s+(?:as\s+of\s+|on\s+)([A-Z][a-z]+\s+\d{1,2},?\s+\d{4})",
    r"dated\s+([A-Z][a-z]+\s+\d{1,2},?\s+\d{4})",
]

def extract_effective_date(text: str) -> str | None:
    for pattern in EFFECTIVE_DATE_PATTERNS:
        m = re.search(pattern, text, flags=re.IGNORECASE)
        if m:
            return normalize_date(m.group(1))
    return None

def normalize_date(s: str) -> str | None:
    for fmt in ("%B %d, %Y", "%B %d %Y", "%m/%d/%Y"):
        try:
            return datetime.strptime(s.strip(), fmt).date().isoformat()
        except ValueError:
            continue
    return None

Where rules break

• "This agreement, effective as of the later of the dates set forth on the signature page below" — the date isn't here.
• Documents that don't use the templated phrase ("Initial Term: 12 months from execution"). The regex misses.
• Multi-language documents.
• Hand-drafted bespoke agreements.

Rules are excellent as the first layer of a hybrid pipeline — they handle the cleanly-templated documents fast and free, and you save the LLM budget for the documents rules miss.

Task-specific encoders

Fine-tuned BERT-class (or LayoutLM-class) models for token classification. Per-field or per-task models.

The shape

• Label 1000–10000 documents with span-level annotations (B-EFFECTIVE_DATE, I-EFFECTIVE_DATE, O for outside).
• Fine-tune a pretrained encoder (BERT for text-only, LayoutLMv3 for layout-aware).
• At inference, classify each token; merge contiguous spans of the same label.

When it's the right call

• You have or can produce labeled data (1000+ examples per field).
• The corpus is large enough that LLM cost per document matters.
• Fields are structured-extractable (named entities, fixed-format values) rather than requiring reasoning ("what's the effective term length considering all amendments").
• You need predictable latency and cost.

Where it falls short

• Bespoke / one-off fields that don't justify a label set.
• Fields that require reasoning across multiple sections of the document.
• Free-form summarization (the renewal term might be "auto-renew unless terminated with 60 days notice prior to anniversary, except in Year 1 where notice is 90 days" — a token classifier can't summarize that).

LLM-based extraction

Send the document (or extracted region) to an LLM with a structured-output prompt. Most flexible; most expensive per call; non-trivial to fully control.

The shape

• Send the document text (and possibly image) to Claude / GPT-4 / Gemini.
• Include a JSON schema or Pydantic model defining the expected output structure.
• Parse the model's response into the structured schema.
• Validate against the schema; surface failures.

When it wins

• Diverse, non-templated documents.
• Fields that require reasoning or summarization.
• Many distinct fields where labeling 1000 examples per field is prohibitive.
• Multi-language documents (LLMs handle this near-natively).
• Edge cases that rules and small models miss.

Where to be careful

• Hallucination. LLMs will produce fluent-but-wrong values. Especially for fields the model "wants" to find but isn't actually present.
• Inconsistency across runs. Same input can produce slightly different outputs. Use temperature=0; even then, prompt sensitivity is real.
• Cost at scale. $0.01–$0.05 per document × 50,000 documents = $500–$2500 per backfill.
• Confidence calibration. LLMs don't naturally produce well-calibrated confidence; you need to engineer it (see calibration chapter).

Structured-output techniques

Getting LLMs to produce reliable, parseable structured output is its own engineering discipline. Three approaches:

1. JSON-mode / response_format=json

Most modern LLM APIs support a "respond in JSON" mode. Pair with a schema in the prompt. The model usually produces parseable JSON.

from anthropic import Anthropic
import json

client = Anthropic()

EXTRACTION_PROMPT = """You are extracting structured fields from a Master Service Agreement.

Document text:
<document>
{text}
</document>

Extract the following fields and return as JSON. If a field cannot be found, return null for that field.

Schema:
{{
  "parties": list of party names (the contracting entities),
  "effective_date": ISO date string (YYYY-MM-DD),
  "term_length_months": integer or null,
  "renewal_terms": {{
    "auto_renew": boolean,
    "term_length_months": integer,
    "notice_period_days": integer
  }} or null,
  "governing_law": string or null
}}

Return only valid JSON. No explanation."""

def extract_msa(text: str) -> dict:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        temperature=0,
        messages=[{"role": "user", "content": EXTRACTION_PROMPT.format(text=text)}],
    )
    return json.loads(response.content[0].text)

2. Function/tool calling with structured arguments

Define the extraction as a "tool" the model calls with structured arguments. The provider enforces schema validity.

3. Constrained decoding (open-source)

Libraries like Outlines, Guidance, or LMQL constrain the LLM's token sampling to match a grammar or schema. Useful for open-source models or stricter output control. More setup than provider-native modes.

Pydantic + provider tool-calling

The modern default: define a Pydantic model, use the LLM provider's tool-calling, get back a validated object.

from pydantic import BaseModel, Field
from typing import Optional

class RenewalTerms(BaseModel):
    auto_renew: bool
    term_length_months: int
    notice_period_days: int

class MSAExtraction(BaseModel):
    parties: list[str]
    effective_date: str = Field(description="ISO date, YYYY-MM-DD")
    term_length_months: Optional[int] = None
    renewal_terms: Optional[RenewalTerms] = None
    governing_law: Optional[str] = None

# Pass MSAExtraction's JSON schema to the LLM as the function definition;
# parse the result with MSAExtraction.model_validate(response_dict).

The hybrid default

What most production contract-intelligence systems actually run:

1. Cloud OCR + layout regions → text + region boundaries.
2. Rule-based pre-pass attempts to extract high-confidence "easy" fields (effective_date, parties from signature block) using regex against the OCR text. Wins ~50–70% of fields cheap.
3. LLM-based extraction for fields the rule pass missed, or for fields that require summarization (renewal_terms, indemnification clauses).
4. Confidence calibration combines per-field confidence from each layer.
5. HITL routing for any field below the threshold.

The hybrid wins on cost (the easy fields don't hit the LLM), latency (rules are fast), and quality (LLM has more attention budget per remaining field).

Prompt engineering for extraction

Production extraction prompts are not "extract everything you can" — they're engineered for accuracy and reliability. Patterns:

Explicit schema in the prompt

Even with JSON mode, include the schema textually. The model performs better when the schema is in its context, not just as a separate parameter.

Instructions about uncertainty

"If a field is not present or cannot be confidently extracted, return null. Do not guess. Do not fabricate values that aren't supported by the text." This single instruction reduces hallucination materially.

Per-field guidance

For ambiguous fields, give the model definitional guidance. "renewal_terms refers to the contractual auto-renewal clause if present; do not extract termination clauses here."

Few-shot examples

Two or three worked examples in the prompt dramatically improve accuracy on idiosyncratic field formats. Token cost is usually worth it.

Chain-of-thought for reasoning fields

For fields that require reasoning ("what's the effective term length considering this amendment"), have the model reason first, then produce the structured output. Increases token usage; improves accuracy.

Citation requirements

Require the model to cite the page or quote the supporting text. Trivially expensive; massively improves debuggability and trustworthiness.

class FieldWithCitation(BaseModel):
    value: str
    quote: str = Field(description="The exact text from the document supporting this value")
    page: int

class MSAExtraction(BaseModel):
    effective_date: Optional[FieldWithCitation] = None
    # ... other fields with same pattern

Cost & latency at scale

For a typical contract-intelligence platform extracting ~10 fields per document over a 50,000-document corpus:

• Pure LLM extraction: $500–$5000 backfill cost depending on model and document length. 1–5 seconds latency per document.
• Hybrid (rules + LLM): $50–$1000 for the same corpus. Median latency near-zero; tail latency 1–5s.
• Token-classification encoder: $5–$50 of inference cost. ~10ms latency per document. Plus the upfront label-and-train cost.

For long-lived deployments, token-classification encoders win on cost. For shorter or one-off deployments, LLM-based or hybrid wins because the upfront training cost doesn't amortize.