Section D · Operating it

Failure Modes & Debugging

The catalog of how extraction goes wrong in production, organized by layer. For each failure mode: symptom, root cause, diagnostic, fix. The debugging protocol when something feels off.

The debugging protocol

When an extraction looks wrong, walk down through the layers in this order:

Is the OCR text correct? Look at the OCR output for the field's region. If garbled, the problem is OCR.
Is the right region identified? If OCR is clean but the extractor pulled from the wrong section, the problem is layout.
Is the model picking the right span? If layout is right but the model picked a wrong substring, the problem is extraction.
Is normalization clobbering a correct value? If extraction is right but the canonical value is wrong, the problem is normalization.
Did HITL miss this? If a low-confidence prediction was incorrectly auto-approved, the problem is calibration or threshold tuning.

Walking the layers takes 10–15 minutes per failure. It's tempting to skip to "must be a model issue." Usually it's an earlier layer.

OCR-layer failures

Multi-column reading order garbled

Symptom: extracted text is nonsensical even though the PDF looks fine in Acrobat.
Root cause: OCR engine read columns row-by-row across columns instead of top-to-bottom within each.
Diagnostic: compare OCR text to source PDF; look for "shuffled" sentences.
Fix: switch OCR engine, change page-segmentation mode (Tesseract --psm 1), or use a layout-aware OCR service (Textract, Document AI Layout).

Table flattening

Symptom: tabular data extracts as flat text; row/column structure is lost.
Root cause: OCR engine doesn't detect tables, or table-detection is disabled.
Diagnostic: look at the OCR output for any table on a page. Are cells structured?
Fix: enable table extraction in the OCR service (Textract Analyze Document with Tables, Document AI Form). For complex tables, post-process with a layout-aware model.

Rotated pages

Symptom: a specific page returns garbage; OCR confidence on that page is near zero.
Root cause: page is rotated 90°/180°/270° and rotation detection failed.
Diagnostic: open the PDF; visually check for rotation.
Fix: add rotation-detection pre-processing (most OCR services have this; ensure it's enabled). For mass corpus issues, use deskew preprocessing libraries.

Watermark contamination

Symptom: extractions include "DRAFT" or "CONFIDENTIAL" tokens mixed with content.
Root cause: OCR is reading semi-transparent watermarks as content.
Diagnostic: check OCR text for known watermark strings.
Fix: post-process OCR text to strip known watermarks. For severe cases, image-preprocess with HSV filtering before OCR.

Layout-layer failures

Signature block read as paragraph

Symptom: "John Smith" or "/s/ John Smith" appears as an extracted party name.
Root cause: layout analysis didn't classify the signature block as such; the extractor treated the text as content.
Diagnostic: check region classifications for the affected document.
Fix: improve layout-aware model coverage (fine-tune on signature-block examples); add post-extraction validation that signature-page text isn't treated as content.

Header/footer pollution

Symptom: page-number tokens or company-letterhead text appear in extracted fields.
Root cause: layout analysis didn't identify recurring header/footer regions; they were treated as content.
Diagnostic: look at the field's provenance — does the bbox fall in the header/footer region of the page?
Fix: configure layout analysis to detect and exclude recurring headers/footers; add a heuristic filter on bbox position.

Wrong region selected

Symptom: extraction is from the wrong section — e.g., termination-notice value extracted as renewal-notice.
Root cause: layout identified the regions correctly but the extractor picked the wrong one (often because it lacked semantic context).
Diagnostic: check provenance bbox; is it in the right section of the document?
Fix: add region-classification step before extraction; tighten the extraction prompt to specify which region to look at.

Extraction-layer failures

Hallucination

Symptom: extracted value isn't in the source document at all.
Root cause: LLM filled in a plausible-sounding value rather than abstaining.
Diagnostic: search the OCR text for the extracted value. If not found, hallucination.
Fix: require citations in the extraction prompt; validate citations by string-match; fail extraction if citation isn't in the text.

Wrong span selected

Symptom: extracted value is in the document, but it's the wrong instance — e.g., the address of one party when you wanted the other.
Root cause: model couldn't disambiguate; both candidate values are plausible.
Diagnostic: check provenance; trace what the model attended to.
Fix: tighten prompt with disambiguation instructions; add few-shot examples of the ambiguous case; route similar cases to HITL.

Format / type mismatch

Symptom: extracted "value" is a string when an integer was expected, or fails schema validation.
Root cause: LLM generated a value that doesn't match the schema constraint.
Diagnostic: schema-validation logs.
Fix: use provider tool-calling with strict schema enforcement (most providers reject invalid schemas); add a parse-or-retry wrapper.

Over-extraction on absent fields

Symptom: extracted values for fields that don't appear in the document.
Root cause: model is biased toward producing non-null outputs when uncertain.
Diagnostic: compare extraction outputs to ground truth on a sample.
Fix: include explicit "return null if not present" instructions in the prompt; few-shot examples with null values; calibrate the threshold so low-confidence non-null predictions go to HITL.

Normalization failures

Date parsing failure

Symptom: extracted_date is null even though the model returned a value.
Root cause: date format wasn't in the normalizer's recognized list (e.g., "the fifteenth day of March, two thousand twenty-four").
Diagnostic: check raw_value vs normalized value.
Fix: expand the date-parser; for unusual formats, route to HITL rather than silently dropping.

Currency normalization off

Symptom: amounts in a non-USD currency are stored without the currency code.
Root cause: normalizer assumed USD; lost the currency signal.
Diagnostic: spot-check international customers' contracts.
Fix: always extract currency code alongside amount; normalize to a (amount, currency) tuple.

Silent entity-resolution merges

Symptom: two distinct supplier entities are silently mapped to the same canonical ID.
Root cause: fuzzy-match threshold too aggressive.
Diagnostic: audit the supplier-resolution log; look for false merges.
Fix: tighten the threshold; route ambiguous matches to HITL; never merge silently without a confidence record.

HITL-related failures

Confidence miscalibration causing wrong routing

Symptom: many auto-approved predictions are wrong; HITL queue is too small to catch them.
Root cause: confidence is over-stated; calibration is off.
Diagnostic: reliability diagram; ECE.
Fix: recompute calibrator on recent ground truth; raise threshold until calibration catches up.

Reviewer drift

Symptom: customer-confirmed extractions disagree with later customer feedback.
Root cause: reviewer calibration has shifted; field interpretation has drifted.
Diagnostic: inter-rater agreement on calibration set; Cohen's kappa over time.
Fix: re-train reviewers on calibration set; tighten field-definition spec.

Queue overflow masking quality

Symptom: HITL queue is so deep that reviewers are rushing; quality slips.
Root cause: capacity isn't matched to volume; or threshold is too generous.
Diagnostic: queue depth, throughput, per-task review time trends.
Fix: add capacity, throttle ingest, or tighten the threshold so fewer items route to HITL (paid in accuracy).

Cross-layer failures

Some failures look like one layer's problem but originate elsewhere. Two common patterns worth recognizing:

"The model is wrong" → it was OCR

The extraction looks wrong; first instinct is that the model failed. Actually the OCR text the model was working from was already garbled. The model produced a plausible answer from corrupted input.

Diagnostic: always check OCR output first. If OCR is wrong, the model is correct conditional on bad input.

"The customer's data drifted" → it was a vendor change

Output suddenly looks different. First instinct: the customer's documents changed. Actually the OCR vendor (Textract, Document AI) pushed a model update that changed the text output.

Diagnostic: check vendor release notes; look for model-version changes on the OCR side, not just your platform side.

The "everything's fine" failure

All monitoring is green. SLA dashboard shows accuracy above threshold. Customer reports specific extractions are wrong.

Usually: the rolling audit sample didn't include the affected document class or field. Aggregated metrics hide concentrated failures.

Diagnostic: drill into per-class, per-field accuracy. Look for cells where the SLA dashboard hides the problem.

The senior diagnostic instinct

Walk the layers in order. Don't skip to the layer you think it must be. Most "this is obviously the model" tickets turn out to be OCR, layout, or calibration. The 10 minutes of walking the layers saves the 2 hours of "investigating" an issue in the wrong place.

✓ Check yourself

Answer from memory before opening each one.

A customer says the supplier address is wrong on one contract. In what order do you check, and why not start with the model?

Show answer
Check the OCR text for the field's region, then whether layout chose the right region, then whether the extractor chose the right span, then the normaliser, then whether review should have caught it. Most model-looking faults are earlier, and the walk takes 10 to 15 minutes. See the debugging protocol.
Output changed on Monday. The customer's documents look the same and you deployed nothing. What do you suspect first and where do you look?

Show answer
Suspect a change at the OCR vendor, since the text the model sees may have changed. Check the vendor's release notes and compare OCR output for the same file before and after. Customer drift is the second guess, not the first. See cross-layer failures.
An indemnification cap is present in 30% of sampled contracts, yet the extractor fills it on 80%. Which failure is it, and what are three fixes?

Show answer
This is over-extraction on absent fields: the model prefers a non-null answer when unsure. Instruct it to return null when absent, add few-shot examples with null answers, and require a citation that is checked against the text. Route low-confidence non-null values to review. See extraction-layer failures.
A wrong party name's provenance box sits at the bottom of the last page. Which layer failed, and what are two fixes?

Show answer
Layout failed: the signature block was read as content. Improve region coverage for signature blocks, and add a check that rejects party values whose box falls inside a signature region. See layout-layer failures.
Every monitor is green, but a customer reports a run of wrong extractions on one document class. What explains that, and what is your next step?

Show answer
Aggregated metrics can hide a concentrated failure, or the audit sample may include few documents of that class or field. Drill into accuracy per class and per field, and enlarge the sample there. See cross-layer failures.
The review queue is deep and reviewers are rushing, so quality is slipping. You can add capacity, throttle ingest or raise the review threshold. How do you choose?

Show answer
Add capacity if the volume is permanent and the budget allows. Throttle ingest if the surge is temporary and the latency promise matters. Raise the threshold only as a last resort, since it moves accuracy into customer output. The deciding factors are duration and the SLA. See review-related failures.

Takeaway

Debug by walking the layers in order, because most faults that look like the model are earlier in the stack.
For every failure mode record the symptom, root cause, diagnostic and fix, so the next ticket is faster.
Hallucination and over-extraction on absent fields are controlled by null instructions, checked citations and review of low-confidence values.
Normalisation fails quietly, so compare raw and normalised values and send unusual formats to review instead of dropping them.
A sudden output change without your deploy often comes from a vendor model update.
Green aggregate dashboards can hide a failing class or field, so drill down before declaring everything fine.

That closes the guide; the build guide puts these layers into a working pipeline. → Build a Contract-Intelligence Pipeline