Layout-Aware Models

Why spatial features matter

Pure text models (BERT, RoBERTa, T5) can read a contract — but they only see the linear sequence of tokens after the page has been flattened to text. A "30 days" appearing in the payment-terms section reads the same as a "30 days" appearing in the termination-notice section. The model has to infer which is which from surrounding text.

Layout-aware models consume the text and the (x, y) position of each token, plus often font size and visual features. This lets them learn relationships like "the 30 days that appears in the bottom-right of page 2 is part of the signature page, not a financial term" or "the dollar amount in a cell next to 'Total' is the contract value."

For document classes with consistent layouts (invoices, forms, receipts), layout-aware models are dramatically better than text-only. For unstructured prose (clauses in a contract body), the advantage shrinks but still helps for region classification.

The model family

The major published layout-aware models, in rough order of release:

• LayoutLM (2020, Microsoft) — first widely-used. Text + 2D position embeddings.
• LayoutLMv2 (2021) — adds visual features (CNN over the page image).
• LayoutLMv3 (2022) — replaces CNN with patch-based vision transformer; cleaner pretraining objective.
• DocFormer (2021, Amazon) — different fusion architecture; competitive with LayoutLMv2 on form understanding.
• LiLT (2022) — Language-independent Layout Transformer. Separates text encoder (any language) from layout encoder; useful for multi-language deployments.
• DocLLM (2024, JPMorgan AI) — adapts LLM architecture to layout. Marketed for legal/financial documents.
• Donut, Nougat, Pix2Struct — end-to-end pixel-to-structured-output models; skip the OCR step entirely. More research than production-default today.

You don't need to know all of these for FDE work. The two worth understanding deeply are LayoutLMv3 (current default for the LayoutLM family) and LiLT (when multi-language matters). The rest are useful as alternatives if vendor relationships or compute constraints favor them.

LayoutLM family

Microsoft's progression of layout-aware encoders. LayoutLMv3 is the current state-of-the-art in this family.

Architecture, abbreviated

• Text tokens (from OCR) embedded as in BERT.
• 2D position embeddings added per token — encoding (x_min, y_min, x_max, y_max) of the token's bounding box on the page.
• Visual features (LayoutLMv3) via patch-based ViT over the page image, fused with text via cross-attention.
• Pretraining objectives include masked text + masked image + word-patch alignment.

What you do with it

Two common patterns:

• Token classification: BIO tagging for fields (B-PARTY, I-PARTY, O for outside) — fine-tune the model head, get per-token labels.
• Region classification: classify each layout region as a type (paragraph, table, signature). Often a pre-step before LLM-based extraction.

Code shape

from transformers import LayoutLMv3ForTokenClassification, LayoutLMv3Processor
from PIL import Image

processor = LayoutLMv3Processor.from_pretrained("microsoft/layoutlmv3-base")
model = LayoutLMv3ForTokenClassification.from_pretrained("path/to/finetuned")

image = Image.open("page.png").convert("RGB")
# words and boxes come from your OCR output
words = ["This", "Master", "Service", "Agreement", ...]
boxes = [[121, 240, 480, 280], ...]  # (x0, y0, x1, y1) per word

encoding = processor(image, words, boxes=boxes, return_tensors="pt")
outputs = model(**encoding)
predictions = outputs.logits.argmax(-1).squeeze().tolist()

The handle in practice: LayoutLMv3 is the default reach when you want a layout-aware encoder you'll fine-tune yourself on a labeled corpus. Microsoft's pretrained checkpoint is on Hugging Face; fine-tuning on 500–5000 labeled documents is the standard recipe.

DocFormer

Amazon's contribution. Different fusion architecture — text, layout, and image features are merged earlier in the transformer stack rather than late-fused. Empirically competitive with LayoutLMv2 on form-understanding benchmarks (FUNSD, CORD, SROIE) and sometimes wins on documents with dense visual structure.

Less momentum than LayoutLMv3 in the open-source community; uses include Amazon's own Textract Analyze Document under the hood and some commercial document AI platforms. Worth knowing exists; not where most production work starts.

LiLT

Language-independent Layout Transformer. The clever architectural move: decouple the text encoder from the layout encoder. The text encoder can be swapped (XLM-RoBERTa for multilingual, BERT for English-only) while the layout encoder stays the same.

When this matters:

• Customer corpus spans multiple languages (English MSAs + German amendments + French SOWs).
• You want to fine-tune once on multilingual data and serve all languages.
• Compute is constrained — LiLT is smaller than LayoutLMv3.

The cost: empirically slightly behind LayoutLMv3 on monolingual English benchmarks. For an English-only contract platform, LayoutLMv3 is usually the right call.

DocLLM & newer alternatives

The newer trend (2024–2026) is adapting LLM architectures to document layout — either by training an LLM with layout inputs (DocLLM) or by using vision-capable foundation models directly (Claude with vision, GPT-4o, Gemini).

DocLLM

JPMorgan's release. Decoder-only LLM that takes text + spatial position as input. Promising for tasks that need free-form generation rather than classification (e.g., "summarize the indemnification clause").

Vision-LLMs as extractors

Claude / GPT-4o / Gemini can ingest a page image directly and produce structured output. Cleanest UX; you skip OCR + layout-aware-model + extractor as separate steps and let the LLM do all three. Tradeoffs:

• Cost: per-page vision-LLM cost is significantly higher than dedicated OCR + small extractor.
• Latency: vision-LLMs are slower per page than dedicated stacks.
• Accuracy: comparable on simple layouts; sometimes weaker on dense, multi-column documents.
• Throughput: dedicated OCR scales horizontally cheaper.

For prototypes and low-volume use cases, the vision-LLM approach is excellent. For production at corpus scale, dedicated OCR + LayoutLM-family + LLM-as-extractor is usually more economical.

When to use which

What layout-aware models are collectively bad at

• Hand-drawn annotations and corrections. Models trained on typeset documents struggle when humans write on the page.
• Multi-page reasoning. Most layout-aware models process one page at a time. A contract where an obligation is defined on page 2 and modified on page 8 requires multi-page logic on top of the model.
• Tables with merged cells, footnotes, complex nesting. Layout models infer row/column relationships heuristically; complex tables break them.
• Heavily customized documents. A bespoke legal document that doesn't look like the training distribution gets lower-quality predictions. Fine-tuning helps if you have labels.
• Documents where reading order is ambiguous. Sidebar content, footnotes, multi-column with embedded boxes. Often the wrong text ends up next to the wrong text in the linearized sequence.

The collective fix isn't a better layout-aware model — it's the validation and HITL layer (chapters 06–07) catching the cases the model gets wrong and routing them to human review.