Drift Detection & Monitoring

Three layers of drift

"Drift" is a generic term for "things changed in production in a way that affects extraction quality." Three distinct kinds, each with its own detection mechanism:

• Input drift — the documents flowing in have shifted in some way. Customer's data is different from what the system was tuned on.
• Output drift — the extractions are systematically different from before, even if inputs haven't changed visibly. Often signals a model or pipeline change.
• Outcome drift — the accuracy itself has slipped, measured against ground truth.

Output and outcome drift often have the same root cause but surface differently. Input drift is upstream of both.

Input drift

What the customer is sending changed. Examples:

• A new procurement system is producing documents in a layout the model wasn't trained on.
• The customer started accepting French-language supplier contracts in addition to English.
• An acquisition brought in 10,000 legacy documents from a different template family.
• Scan quality degraded (new copier; remote work; rushed uploads).

What to monitor

• OCR confidence distribution over time. Sudden drops signal scan-quality regression.
• Layout-region distribution: are we seeing more / fewer tables, signature blocks, multi-column pages?
• Document length distribution: shorter or longer than typical?
• Language distribution: any non-English text appearing where none did before?
• Source distribution: changes in which customer-side system is producing documents.

Detection method

Compute distribution statistics weekly; compare to the prior 90-day baseline. Surface anything that shifts by more than 2 standard deviations.

import numpy as np

def detect_distribution_shift(
    current: np.ndarray,
    baseline: np.ndarray,
    threshold: float = 2.0,
) -> dict:
    """Z-score-style shift detection for a univariate distribution."""
    baseline_mean = baseline.mean()
    baseline_std = baseline.std()
    current_mean = current.mean()
    z = (current_mean - baseline_mean) / max(baseline_std, 1e-6)
    return {
        "current_mean": current_mean,
        "baseline_mean": baseline_mean,
        "z_score": z,
        "shifted": abs(z) > threshold,
    }

Output drift

The extractions are systematically different even when input statistics look stable. Often signals:

• A model version change.
• A prompt change.
• An upstream OCR vendor change (e.g., Textract pushed a quality update).
• A customer-config change (someone adjusted thresholds).

What to monitor

• Confidence distribution: deciles of confidence per document class over time. A sudden shift toward lower confidence often precedes accuracy drops.
• Field-population rates: what fraction of documents have a non-null value for each field? Sudden changes here are sometimes the first signal of layout drift.
• HITL routing rate: what fraction of extractions hit HITL? Spikes mean the model is less confident; drops mean the threshold may be too lax.
• Pipeline timing: average inference latency per document. Slowdowns can indicate model swaps or queue issues.

Outcome drift

The ground-truth signal. Accuracy on labeled samples is below where it was. This is the metric customers contractually care about; it's also the slowest to detect because it requires labeled outcomes.

Sources

• Rolling audit sample: the online-eval sample from chapter 08.
• HITL corrections: when reviewers correct a previously-auto-approved value (unusual but it happens), or when the rate of "model was right" in HITL drops.
• Customer-reported corrections: customers occasionally flag specific extractions as wrong. Their feedback is high-signal.

The lag problem

Outcome drift is slow to detect because labeling takes time. By the time accuracy on a 30-day rolling window shows a drop, the underlying cause may have been live for 4–6 weeks. This is why output and input drift monitoring matter — they're leading indicators of outcome drift.

Drift metrics

Population Stability Index (PSI)

Compares the distribution of a feature between two time periods. PSI < 0.1: no shift. 0.1–0.25: moderate. > 0.25: significant. Easy to compute, easy to interpret.

import numpy as np

def psi(baseline: np.ndarray, current: np.ndarray, n_bins: int = 10) -> float:
    edges = np.quantile(baseline, np.linspace(0, 1, n_bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    b_counts, _ = np.histogram(baseline, bins=edges)
    c_counts, _ = np.histogram(current, bins=edges)
    # Laplace-smooth so no zero division
    b_pct = (b_counts + 1) / (b_counts.sum() + n_bins)
    c_pct = (c_counts + 1) / (c_counts.sum() + n_bins)
    return float(np.sum((c_pct - b_pct) * np.log(c_pct / b_pct)))

Kolmogorov–Smirnov test

Tests whether two samples come from the same distribution. Returns a p-value; threshold on it for alerts. More statistically principled than PSI; less interpretable for non-statisticians.

Per-bucket relative change

Sometimes the simplest is best. For each confidence decile, what fraction of documents fall in it this week vs last week? Surface deciles with > 30% relative change.

Alerting thresholds

The art is alerting often enough to catch real drift but not so often that alerts get ignored. Rough thresholds:

Each P1 alert pages someone. P2 goes to a Slack channel where it's reviewed daily. P3 lands in a weekly digest.

What to surface to customers

Not every alert needs customer visibility. The rule of thumb:

• Customer-facing dashboard: the SLA accuracy and rolling 30-day trend. Always visible.
• Proactive notification: when accuracy is trending toward the threshold (within 2pp, declining), tell the customer before they ask.
• Active incident: SLA is below threshold — formal communication via email + dashboard banner, plus a remediation plan with timeline.
• Internal-only: most input and output drift signals — debug-grade information, not customer-facing.