Failure Modes & Debugging

The debugging protocol

When an extraction looks wrong, walk down through the layers in this order:

1. Is the OCR text correct? Look at the OCR output for the field's region. If garbled, the problem is OCR.
2. Is the right region identified? If OCR is clean but the extractor pulled from the wrong section, the problem is layout.
3. Is the model picking the right span? If layout is right but the model picked a wrong substring, the problem is extraction.
4. Is normalization clobbering a correct value? If extraction is right but the canonical value is wrong, the problem is normalization.
5. Did HITL miss this? If a low-confidence prediction was incorrectly auto-approved, the problem is calibration or threshold tuning.

Walking the layers takes 10–15 minutes per failure. It's tempting to skip to "must be a model issue." Usually it's an earlier layer.

OCR-layer failures

Multi-column reading order garbled

• Symptom: extracted text is nonsensical even though the PDF looks fine in Acrobat.
• Root cause: OCR engine read columns row-by-row across columns instead of top-to-bottom within each.
• Diagnostic: compare OCR text to source PDF; look for "shuffled" sentences.
• Fix: switch OCR engine, change page-segmentation mode (Tesseract --psm 1), or use a layout-aware OCR service (Textract, Document AI Layout).

Table flattening

• Symptom: tabular data extracts as flat text; row/column structure is lost.
• Root cause: OCR engine doesn't detect tables, or table-detection is disabled.
• Diagnostic: look at the OCR output for any table on a page. Are cells structured?
• Fix: enable table extraction in the OCR service (Textract Analyze Document with Tables, Document AI Form). For complex tables, post-process with a layout-aware model.

Rotated pages

• Symptom: a specific page returns garbage; OCR confidence on that page is near zero.
• Root cause: page is rotated 90°/180°/270° and rotation detection failed.
• Diagnostic: open the PDF; visually check for rotation.
• Fix: add rotation-detection pre-processing (most OCR services have this; ensure it's enabled). For mass corpus issues, use deskew preprocessing libraries.

Watermark contamination

• Symptom: extractions include "DRAFT" or "CONFIDENTIAL" tokens mixed with content.
• Root cause: OCR is reading semi-transparent watermarks as content.
• Diagnostic: check OCR text for known watermark strings.
• Fix: post-process OCR text to strip known watermarks. For severe cases, image-preprocess with HSV filtering before OCR.

Layout-layer failures

Signature block read as paragraph

• Symptom: "John Smith" or "/s/ John Smith" appears as an extracted party name.
• Root cause: layout analysis didn't classify the signature block as such; the extractor treated the text as content.
• Diagnostic: check region classifications for the affected document.
• Fix: improve layout-aware model coverage (fine-tune on signature-block examples); add post-extraction validation that signature-page text isn't treated as content.

Header/footer pollution

• Symptom: page-number tokens or company-letterhead text appear in extracted fields.
• Root cause: layout analysis didn't identify recurring header/footer regions; they were treated as content.
• Diagnostic: look at the field's provenance — does the bbox fall in the header/footer region of the page?
• Fix: configure layout analysis to detect and exclude recurring headers/footers; add a heuristic filter on bbox position.

Wrong region selected

• Symptom: extraction is from the wrong section — e.g., termination-notice value extracted as renewal-notice.
• Root cause: layout identified the regions correctly but the extractor picked the wrong one (often because it lacked semantic context).
• Diagnostic: check provenance bbox; is it in the right section of the document?
• Fix: add region-classification step before extraction; tighten the extraction prompt to specify which region to look at.

Extraction-layer failures

Hallucination

• Symptom: extracted value isn't in the source document at all.
• Root cause: LLM filled in a plausible-sounding value rather than abstaining.
• Diagnostic: search the OCR text for the extracted value. If not found, hallucination.
• Fix: require citations in the extraction prompt; validate citations by string-match; fail extraction if citation isn't in the text.

Wrong span selected

• Symptom: extracted value is in the document, but it's the wrong instance — e.g., the address of one party when you wanted the other.
• Root cause: model couldn't disambiguate; both candidate values are plausible.
• Diagnostic: check provenance; trace what the model attended to.
• Fix: tighten prompt with disambiguation instructions; add few-shot examples of the ambiguous case; route similar cases to HITL.

Format / type mismatch

• Symptom: extracted "value" is a string when an integer was expected, or fails schema validation.
• Root cause: LLM generated a value that doesn't match the schema constraint.
• Diagnostic: schema-validation logs.
• Fix: use provider tool-calling with strict schema enforcement (most providers reject invalid schemas); add a parse-or-retry wrapper.

Over-extraction on absent fields

• Symptom: extracted values for fields that don't appear in the document.
• Root cause: model is biased toward producing non-null outputs when uncertain.
• Diagnostic: compare extraction outputs to ground truth on a sample.
• Fix: include explicit "return null if not present" instructions in the prompt; few-shot examples with null values; calibrate the threshold so low-confidence non-null predictions go to HITL.

Normalization failures

Date parsing failure

• Symptom: extracted_date is null even though the model returned a value.
• Root cause: date format wasn't in the normalizer's recognized list (e.g., "the fifteenth day of March, two thousand twenty-four").
• Diagnostic: check raw_value vs normalized value.
• Fix: expand the date-parser; for unusual formats, route to HITL rather than silently dropping.

Currency normalization off

• Symptom: amounts in a non-USD currency are stored without the currency code.
• Root cause: normalizer assumed USD; lost the currency signal.
• Diagnostic: spot-check international customers' contracts.
• Fix: always extract currency code alongside amount; normalize to a (amount, currency) tuple.

Silent entity-resolution merges

• Symptom: two distinct supplier entities are silently mapped to the same canonical ID.
• Root cause: fuzzy-match threshold too aggressive.
• Diagnostic: audit the supplier-resolution log; look for false merges.
• Fix: tighten the threshold; route ambiguous matches to HITL; never merge silently without a confidence record.

HITL-related failures

Confidence miscalibration causing wrong routing

• Symptom: many auto-approved predictions are wrong; HITL queue is too small to catch them.
• Root cause: confidence is over-stated; calibration is off.
• Diagnostic: reliability diagram; ECE.
• Fix: recompute calibrator on recent ground truth; raise threshold until calibration catches up.

Reviewer drift

• Symptom: customer-confirmed extractions disagree with later customer feedback.
• Root cause: reviewer calibration has shifted; field interpretation has drifted.
• Diagnostic: inter-rater agreement on calibration set; Cohen's kappa over time.
• Fix: re-train reviewers on calibration set; tighten field-definition spec.

Queue overflow masking quality

• Symptom: HITL queue is so deep that reviewers are rushing; quality slips.
• Root cause: capacity isn't matched to volume; or threshold is too generous.
• Diagnostic: queue depth, throughput, per-task review time trends.
• Fix: add capacity, throttle ingest, or tighten the threshold so fewer items route to HITL (paid in accuracy).

Cross-layer failures

Some failures look like one layer's problem but originate elsewhere. Two common patterns worth recognizing:

"The model is wrong" → it was OCR

The extraction looks wrong; first instinct is that the model failed. Actually the OCR text the model was working from was already garbled. The model produced a plausible answer from corrupted input.

Diagnostic: always check OCR output first. If OCR is wrong, the model is correct conditional on bad input.

"The customer's data drifted" → it was a vendor change

Output suddenly looks different. First instinct: the customer's documents changed. Actually the OCR vendor (Textract, Document AI) pushed a model update that changed the text output.

Diagnostic: check vendor release notes; look for model-version changes on the OCR side, not just your platform side.

The "everything's fine" failure

All monitoring is green. SLA dashboard shows accuracy above threshold. Customer reports specific extractions are wrong.

Usually: the rolling audit sample didn't include the affected document class or field. Aggregated metrics hide concentrated failures.

Diagnostic: drill into per-class, per-field accuracy. Look for cells where the SLA dashboard hides the problem.