Layout-Aware Models

Why spatial features matter

Pure text models (BERT, RoBERTa, T5) can read a contract — but they only see the linear sequence of tokens after the page has been flattened to text. A "30 days" appearing in the payment-terms section reads the same as a "30 days" appearing in the termination-notice section. The model has to infer which is which from surrounding text.

Layout-aware models consume the text and the (x, y) position of each token, plus often font size and visual features. This lets them learn relationships like "the 30 days that appears in the bottom-right of page 2 is part of the signature page, not a financial term" or "the dollar amount in a cell next to 'Total' is the contract value."

For document classes with consistent layouts (invoices, forms, receipts), layout-aware models are dramatically better than text-only. For unstructured prose (clauses in a contract body), the advantage shrinks but still helps for region classification.

The model family

The major published layout-aware models, in rough order of release:

• LayoutLM (2020, Microsoft) — first widely-used. Text + 2D position embeddings.
• LayoutLMv2 (2021) — adds visual features (CNN over the page image).
• LayoutLMv3 (2022) — replaces CNN with patch-based vision transformer; cleaner pretraining objective.
• DocFormer (2021, Amazon) — different fusion architecture; competitive with LayoutLMv2 on form understanding.
• LiLT (2022) — Language-independent Layout Transformer. Separates text encoder (any language) from layout encoder; useful for multi-language deployments.
• DocLLM (2024, JPMorgan AI) — adapts LLM architecture to layout. Marketed for legal/financial documents.
• Donut, Nougat, Pix2Struct — end-to-end pixel-to-structured-output models; skip the OCR step entirely. More research than production-default today.

You don't need to know all of these for FDE work. The two worth understanding deeply are LayoutLMv3 (current default for the LayoutLM family) and LiLT (when multi-language matters). The rest are useful as alternatives if vendor relationships or compute constraints favor them.

LayoutLM family

Microsoft's progression of layout-aware encoders. LayoutLMv3 is the current state-of-the-art in this family.

Architecture, abbreviated

• Text tokens (from OCR) embedded as in BERT.
• 2D position embeddings added per token — encoding (x_min, y_min, x_max, y_max) of the token's bounding box on the page.
• Visual features (LayoutLMv3) as linear patch embeddings of the page image, placed in the same input sequence as the text tokens and processed by one multimodal transformer.
• Pretraining objectives include masked text + masked image + word-patch alignment.

What you do with it

Two common patterns:

• Token classification: BIO tagging for fields (B-PARTY, I-PARTY, O for outside) — fine-tune the model head, get per-token labels.
• Region classification: classify each layout region as a type (paragraph, table, signature). Often a pre-step before LLM-based extraction.

Code shape

from transformers import LayoutLMv3ForTokenClassification, LayoutLMv3Processor
from PIL import Image

processor = LayoutLMv3Processor.from_pretrained("microsoft/layoutlmv3-base")
model = LayoutLMv3ForTokenClassification.from_pretrained("path/to/finetuned")

image = Image.open("page.png").convert("RGB")
# words and boxes come from your OCR output
words = ["This", "Master", "Service", "Agreement", ...]
boxes = [[121, 240, 480, 280], ...]  # (x0, y0, x1, y1) per word

encoding = processor(image, words, boxes=boxes, return_tensors="pt")
outputs = model(**encoding)
predictions = outputs.logits.argmax(-1).squeeze().tolist()

The handle in practice: LayoutLMv3 is the default reach when you want a layout-aware encoder you'll fine-tune yourself on a labeled corpus. Microsoft's pretrained checkpoint is on Hugging Face; fine-tuning on 500–5000 labeled documents is the standard recipe.

DocFormer

Amazon's contribution. Different fusion architecture — text, layout, and image features are merged earlier in the transformer stack rather than late-fused. Empirically competitive with LayoutLMv2 on form-understanding benchmarks (FUNSD, CORD, SROIE) and sometimes wins on documents with dense visual structure.

Less momentum than LayoutLMv3 in the open-source community. Worth knowing exists; not where most production work starts.

LiLT

Language-independent Layout Transformer. The clever architectural move: decouple the text encoder from the layout encoder. The text encoder can be swapped (XLM-RoBERTa for multilingual, BERT for English-only) while the layout encoder stays the same.

When this matters:

• Customer corpus spans multiple languages (English MSAs + German amendments + French SOWs).
• You want to fine-tune once on multilingual data and serve all languages.
• Compute is constrained — LiLT is smaller than LayoutLMv3.

The cost: empirically slightly behind LayoutLMv3 on monolingual English benchmarks. For an English-only contract platform, LayoutLMv3 is usually the right call.

DocLLM & newer alternatives

The newer trend (2024–2026) is adapting LLM architectures to document layout — either by training an LLM with layout inputs (DocLLM) or by using vision-capable foundation models directly (Claude with vision, GPT-4o, Gemini).

DocLLM

JPMorgan's release. Decoder-only LLM that takes text + spatial position as input. Promising for tasks that need free-form generation rather than classification (e.g., "summarize the indemnification clause").

Vision-LLMs as extractors

Claude / GPT-4o / Gemini can ingest a page image directly and produce structured output. Cleanest UX; you skip OCR + layout-aware-model + extractor as separate steps and let the LLM do all three. Tradeoffs:

• Cost: per-page vision-LLM cost is significantly higher than dedicated OCR + small extractor.
• Latency: vision-LLMs are slower per page than dedicated stacks.
• Accuracy: comparable on simple layouts; sometimes weaker on dense, multi-column documents.
• Throughput: dedicated OCR scales horizontally cheaper.

For prototypes and low-volume use cases, the vision-LLM approach is excellent. For production at corpus scale, dedicated OCR + LayoutLM-family + LLM-as-extractor is usually more economical.

When to use which

What layout-aware models are collectively bad at

• Hand-drawn annotations and corrections. Models trained on typeset documents struggle when humans write on the page.
• Multi-page reasoning. Most layout-aware models process one page at a time. A contract where an obligation is defined on page 2 and modified on page 8 requires multi-page logic on top of the model.
• Tables with merged cells, footnotes, complex nesting. Layout models infer row/column relationships heuristically; complex tables break them.
• Heavily customized documents. A bespoke legal document that doesn't look like the training distribution gets lower-quality predictions. Fine-tuning helps if you have labels.
• Documents where reading order is ambiguous. Sidebar content, footnotes, multi-column with embedded boxes. Often the wrong text ends up next to the wrong text in the linearized sequence.

The collective fix isn't a better layout-aware model — it's the validation and HITL layer (chapters 06–07) catching the cases the model gets wrong and routing them to human review.

✓ Check yourself

Answer from memory before opening each one.

1. An invoice shows 03/15/2024 beside "Ship date" and again beside "Payment due". After OCR flattens the page, why can a layout-aware model tell them apart when a text-only encoder struggles?

   Show answer

   The layout model also receives each token's bounding box, so position relative to the label becomes a feature. A text-only encoder sees two identical tokens in a line and must guess from neighbouring words. See why spatial features matter.

2. You fine-tune a LayoutLMv3 token classifier. What must the pipeline supply at inference, and which upstream layer caps its accuracy?

   Show answer

   It needs the words, one box per word and the page image, all produced by OCR. The classifier cannot recover a word that OCR misread or a box that OCR misplaced, so OCR quality caps it. See the LayoutLM family.

3. A platform serves English MSAs today and will add German amendments next quarter. Do you stay with LayoutLMv3 or move to LiLT?

   Show answer

   Lean to LiLT if the German volume is real, since its text encoder can be swapped for a multilingual one while the layout encoder stays. If English stays dominant and German is a trickle, LayoutLMv3 is a little stronger on English, and an LLM extractor can absorb the German. The deciding factor is German volume. See LiLT.

4. An insurer sends 200 bespoke contracts a month with no labelled data. Which approach do you propose first, and why not fine-tune a layout model?

   Show answer

   Propose cloud OCR with layout regions and an LLM extractor. Fine-tuning needs hundreds to thousands of labelled documents, and bespoke layouts fall outside any training distribution anyway. Revisit once labelled corrections accumulate. See when to use which.

5. A termination right is defined on page 2 and modified on page 8. The layout model scores each page on its own. What goes wrong, and where do you fix it?

   Show answer

   No page-level model sees both clauses, so the extracted value reflects only one of them. Fix it above the model: gather the related passages across pages for the LLM extractor, and route cross-page fields to review. A better layout model does not help. See what these models are bad at.

6. A product manager wants to send page images straight to a vision LLM and drop OCR. When is that right, and when does it fail?

   Show answer

   It is right for prototypes and low volume, where it removes three components and works well on simple layouts. At corpus scale the cost and latency per page are higher than dedicated OCR with a small extractor, and dense multi-column pages can degrade. See DocLLM and newer alternatives.

Takeaway

• Layout-aware models add each token's position to its text, which separates identical values that mean different things.
• LayoutLMv3 is the usual choice for a model you fine-tune yourself, and LiLT is the choice when languages vary.
• These models read OCR output, so OCR quality caps their accuracy.
• For contract prose with varied layouts, cloud OCR with layout regions plus an LLM extractor often makes fine-tuning unnecessary.
• Vision LLMs suit prototypes and low volume; at corpus scale dedicated OCR is cheaper and faster.
• Handwriting, multi-page reasoning, complex tables and odd reading order defeat all of them, so the validation and review layers must catch those cases.

Once a model has located the text, you must choose how to turn it into field values. → Field Extraction Patterns

Further reading

• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking — Huang et al., 2022. The architecture and pretraining objectives behind the default layout-aware encoder.
• LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding — Wang et al., 2022. The decoupled text and layout design for multilingual use.
• DocFormer: End-to-End Transformer for Document Understanding — Appalaraju et al., 2021. The alternative fusion design and its benchmark results.
• DocLLM: A layout-aware generative language model for multimodal document understanding — Wang et al., 2023. How a decoder-only LLM can use spatial information without an image encoder.
• LayoutLMv3 in Transformers — Hugging Face documentation, 2026. The processor and model classes used in the chapter's code, including the expected box format.