Drift Detection & Monitoring

Three layers of drift

"Drift" is a generic term for "things changed in production in a way that affects extraction quality." Three distinct kinds, each with its own detection mechanism:

• Input drift — the documents flowing in have shifted in some way. Customer's data is different from what the system was tuned on.
• Output drift — the extractions are systematically different from before, even if inputs haven't changed visibly. Often signals a model or pipeline change.
• Outcome drift — the accuracy itself has slipped, measured against ground truth.

Output and outcome drift often have the same root cause but surface differently. Input drift is upstream of both.

Input drift

What the customer is sending changed. Examples:

• A new procurement system is producing documents in a layout the model wasn't trained on.
• The customer started accepting French-language supplier contracts in addition to English.
• An acquisition brought in 10,000 legacy documents from a different template family.
• Scan quality degraded (new copier; remote work; rushed uploads).

What to monitor

• OCR confidence distribution over time. Sudden drops signal scan-quality regression.
• Layout-region distribution: are we seeing more / fewer tables, signature blocks, multi-column pages?
• Document length distribution: shorter or longer than typical?
• Language distribution: any non-English text appearing where none did before?
• Source distribution: changes in which customer-side system is producing documents.

Detection method

Compute distribution statistics weekly; compare to the prior 90-day baseline. Surface anything that shifts by more than 2 standard deviations.

import numpy as np

def detect_distribution_shift(
    current: np.ndarray,
    baseline: np.ndarray,
    threshold: float = 2.0,
) -> dict:
    """Z-score-style shift detection for a univariate distribution."""
    baseline_mean = baseline.mean()
    baseline_std = baseline.std()
    current_mean = current.mean()
    z = (current_mean - baseline_mean) / max(baseline_std, 1e-6)
    return {
        "current_mean": current_mean,
        "baseline_mean": baseline_mean,
        "z_score": z,
        "shifted": abs(z) > threshold,
    }

Output drift

The extractions are systematically different even when input statistics look stable. Often signals:

• A model version change.
• A prompt change.
• An upstream OCR vendor change (e.g., Textract pushed a quality update).
• A customer-config change (someone adjusted thresholds).

What to monitor

• Confidence distribution: deciles of confidence per document class over time. A sudden shift toward lower confidence often precedes accuracy drops.
• Field-population rates: what fraction of documents have a non-null value for each field? Sudden changes here are sometimes the first signal of layout drift.
• HITL routing rate: what fraction of extractions hit HITL? Spikes mean the model is less confident; drops mean the threshold may be too lax.
• Pipeline timing: average inference latency per document. Slowdowns can indicate model swaps or queue issues.

Outcome drift

The ground-truth signal. Accuracy on labeled samples is below where it was. This is the metric customers contractually care about; it's also the slowest to detect because it requires labeled outcomes.

Sources

• Rolling audit sample: the online-eval sample from chapter 08.
• HITL corrections: when reviewers correct a previously-auto-approved value (unusual but it happens), or when the rate of "model was right" in HITL drops.
• Customer-reported corrections: customers occasionally flag specific extractions as wrong. Their feedback is high-signal.

The lag problem

Outcome drift is slow to detect because labeling takes time. By the time accuracy on a 30-day rolling window shows a drop, the underlying cause may have been live for 4–6 weeks. This is why output and input drift monitoring matter — they're leading indicators of outcome drift.

Drift metrics

Population Stability Index (PSI)

Compares the distribution of a feature between two time periods. PSI < 0.1: no shift. 0.1–0.25: moderate. > 0.25: significant. Easy to compute, easy to interpret.

import numpy as np

def psi(baseline: np.ndarray, current: np.ndarray, n_bins: int = 10) -> float:
    edges = np.quantile(baseline, np.linspace(0, 1, n_bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    b_counts, _ = np.histogram(baseline, bins=edges)
    c_counts, _ = np.histogram(current, bins=edges)
    # Laplace-smooth so no zero division
    b_pct = (b_counts + 1) / (b_counts.sum() + n_bins)
    c_pct = (c_counts + 1) / (c_counts.sum() + n_bins)
    return float(np.sum((c_pct - b_pct) * np.log(c_pct / b_pct)))

Kolmogorov–Smirnov test

Tests whether two samples come from the same distribution. Returns a p-value; threshold on it for alerts. More statistically principled than PSI; less interpretable for non-statisticians.

Per-bucket relative change

Sometimes the simplest is best. For each confidence decile, what fraction of documents fall in it this week vs last week? Surface deciles with > 30% relative change.

Alerting thresholds

The art is alerting often enough to catch real drift but not so often that alerts get ignored. Rough thresholds:

Each P1 alert pages someone. P2 goes to a Slack channel where it's reviewed daily. P3 lands in a weekly digest.

What to surface to customers

Not every alert needs customer visibility. The rule of thumb:

• Customer-facing dashboard: the SLA accuracy and rolling 30-day trend. Always visible.
• Proactive notification: when accuracy is trending toward the threshold (within 2pp, declining), tell the customer before they ask.
• Active incident: SLA is below threshold — formal communication via email + dashboard banner, plus a remediation plan with timeline.
• Internal-only: most input and output drift signals — debug-grade information, not customer-facing.

✓ Check yourself

Answer from memory before opening each one.

1. Accuracy on the rolling audit sample fell last week. Why was the cause probably live for weeks, and which signals would have warned you sooner?

   Show answer

   Labelling takes time and the window is 30 days, so a drop appears four to six weeks after the cause. Input measures such as OCR confidence and layout mix, and output measures such as field-population and review rates, change immediately, so they lead. See outcome drift.

2. A weekly check compares the current mean OCR confidence to the baseline mean and divides by the baseline standard deviation of individual pages. Mean confidence falls 4 points, the baseline spread is 6 points, and nothing alerts. What is wrong?

   Show answer

   The statistic is 0.67, because it divides by the spread of single pages rather than the uncertainty of a mean. Over hundreds of pages a 4-point fall is large. Use a distribution measure such as PSI, or divide by the standard error. See input drift.

3. The effective-date field was populated on 97% of documents and is now at 81%. The accuracy dashboard is still green. What does this suggest and what do you do?

   Show answer

   It suggests input drift, such as a new template or scan source, that the extractor now misses. Accuracy lags, so compare layout and OCR distributions week over week, pull a sample of the unpopulated documents, and fix before the audit sample moves. See output drift.

4. PSI on the confidence of one field is 0.31 for supplier agreements. What is the alert level, and what do you check first?

   Show answer

   It crosses the 0.25 line for significant shift, a lower-severity alert that goes to the weekly digest. Check what changed in that window: model or prompt version, customer config, and the OCR vendor. See drift metrics.

5. SLA accuracy is 96.4%, the contract says 95%, and it has declined for three days. Do you tell the customer?

   Show answer

   Tell them if it is within 2 points and trending down; that is the proactive notice rule. Send it before they ask, with the cause and the plan. Keep routine PSI findings internal. See what to surface to customers.

6. A weekly digest could use PSI, a KS test p-value or per-decile relative change. Which do you choose for people who are not statisticians, and what is the risk with the p-value?

   Show answer

   PSI or per-decile change, because they have simple thresholds a reader can act on. A p-value flags tiny shifts once samples are large, so it alerts on changes that do not matter. See drift metrics.

Takeaway

• Drift comes in three kinds: input, output and outcome, and outcome is the slowest to see.
• Watch OCR confidence, layout mix, language and length for input drift, and confidence, field-population and review rates for output drift.
• A z-score on single-item spread misses real shifts, so use PSI or a similar distribution measure.
• Map signals to severity: a breached SLA pages someone, a declining trend goes to a daily channel, and drift statistics go to a weekly digest.
• A vendor model update can look like customer drift, so check release notes when output changes without a deploy of yours.
• Tell the customer about a downward trend before they see it, and keep debug-level drift internal.

Monitoring tells you something is wrong; the next chapter is the catalog for finding which layer broke. → Failure Modes & Debugging

Further reading

• Alerting on SLOs — Google SRE Workbook, 2018. How to alert on a service-level objective by burn rate without paging on noise.
• scipy.stats.ks_2samp — SciPy documentation, 2026. The two-sample Kolmogorov-Smirnov test used for distribution shift.
• Evidently documentation — Evidently AI, 2026. Data drift and model monitoring checks, including PSI and statistical tests.
• Amazon Textract document history — AWS documentation, 2026. Service changes that can alter OCR output without any change on your side.
• What's new in Document Intelligence — Microsoft documentation, 2026. Model and API version changes to watch when output shifts.