Eval Methodology

Why eval is the differentiator

Two organizations can have identical extraction stacks but very different operational outcomes — because one of them measures accuracy rigorously and the other doesn't. Without measurement:

• You can't tell when quality slips until the customer complains.
• You can't ship a model change confidently — you have no scoreboard.
• You can't price the SLA — you don't know what you can stand behind.
• You can't tune the HITL threshold — you don't know the auto-approved accuracy.

Eval isn't optional. The discipline of measurement is what makes SLA-backed accuracy a real product rather than a marketing claim.

Ground-truth sources

Eval requires knowing the right answer for each test example. Four sources, in order of preference:

1. Customer-supplied gold set

The customer's team confirms the right values for a sample of their documents. Best quality (it's their domain), best customer credibility (they can't argue with their own labels later). Most expensive to obtain — customer time is hard to get.

2. Internal analyst audit

Platform-employed analysts re-read sampled documents and produce ground truth. Most common in practice. Cheaper than customer time; quality is high if analysts are well-trained.

3. Customer-confirmed extractions

Extractions that the customer's reviewers confirmed in the HITL workflow. Free in volume; biased toward documents that hit HITL (i.e., low-confidence ones). Useful for calibration; less useful for top-line accuracy measurement.

4. Synthetic ground truth

Generate fake contracts with known ground truth. Useful for development testing only — never for SLA measurement against real customers.

Inter-rater agreement on ground truth

Even ground truth has noise. Two analysts labeling the same document independently should agree ≥ 90% on simple fields. Below that, the field definition is too ambiguous to measure cleanly — fix the spec before fixing the model.

Sample sizing

To estimate a population accuracy with confidence, you need enough labeled samples. The math:

For estimating a proportion p with margin of error E at confidence level (1−α), sample size n ≈ z² × p × (1−p) / E², where z is the z-score for the confidence level.

For 95% accuracy with ±2 percentage points at 95% confidence: n ≈ 456. Round up to 500 for safety. For ±1pp: n ≈ 1800. For ±5pp: n ≈ 73.

This is per-document-class. If your SLA covers 5 classes, you need ~500 labeled samples per class to measure each independently.

from math import ceil
from scipy.stats import norm

def required_sample_size(target_accuracy: float, margin: float, confidence: float = 0.95) -> int:
    """For a binary outcome (correct / not), how many samples for the margin?"""
    p = target_accuracy
    z = norm.ppf(1 - (1 - confidence) / 2)
    n = (z ** 2) * p * (1 - p) / (margin ** 2)
    return ceil(n)

print(required_sample_size(0.95, 0.02))   # → 456
print(required_sample_size(0.95, 0.01))   # → 1825
print(required_sample_size(0.95, 0.05))   # → 73

Rolling windows

SLA measurement is typically a rolling 30-day window. The sample is recomputed daily; oldest examples drop off. Keeps the metric responsive to recent drift without single-day noise.

Metrics

Field-level accuracy

Of all extracted field values, what fraction match ground truth (within tolerance)? The headline number on the SLA dashboard.

• String fields: exact match, or fuzzy (Levenshtein, normalized casing/whitespace) within a tolerance — defined in the contract.
• Date fields: exact, or within ±N days.
• Numeric fields: exact, or within ±X% of the true value.
• Categorical fields: exact match against canonical taxonomy.

Precision and recall

For field presence (did the extractor find a value when there should be one, abstain when there shouldn't):

• Precision = of fields the extractor returned, what fraction matched ground truth.
• Recall = of fields where ground truth said a value exists, what fraction did the extractor find.

Document-level accuracy

Stricter — a document is correct only if all of its required fields are correct. Often what customers care about ("if I trust this row, is everything in it right?").

Lift over baseline

What's the accuracy of a trivial baseline (e.g., regex-only, or "always return null")? Your model's accuracy minus the baseline shows the model's actual contribution.

Calibration metrics

ECE and reliability diagrams (chapter 06). Track alongside accuracy; if calibration drifts, the HITL threshold needs recompute even before accuracy slips.

Per-class measurement

Aggregating accuracy across document classes is misleading. A 95% overall accuracy might hide:

• MSAs at 98%.
• SOWs at 96%.
• Supplier agreements at 91% (below SLA, hidden by aggregation).

Always report per-class. If your SLA contract has per-class thresholds, the per-class number is what's reportable.

Per-field within class

Drill one level deeper: per-field within each class. The dashboard's heatmap is doc-class × field × accuracy, with the SLA threshold drawn as a line. Failures are immediately visible.

Offline eval before deploys

Before any model or prompt change ships, run it against a frozen eval set.

The frozen eval set

• 500–1000 documents per class with labeled ground truth.
• Sampled to be representative of production distribution.
• Frozen — doesn't change month-to-month. Lets you compare results across model versions over time.
• Versioned. When the eval set is updated, it's a new version (eval-v2.1) so you can interpret historical reports correctly.

The eval run

1. Run the new model/prompt against the eval set.
2. Score per-field accuracy, precision, recall, calibration.
3. Generate a comparison report vs the current production model.
4. Approve / reject the change based on the report.

Build this into CI. Every PR that touches extraction code runs the eval and fails CI if accuracy regresses on any class above the noise floor.

Online eval in production

Offline eval doesn't catch drift in production. Online eval continuously samples production output for audit.

The online sample

• Randomly sample N documents per day across each class.
• Route them to internal analysts for ground-truth labeling (or use customer-confirmed extractions if available).
• Score and aggregate into the rolling 30-day metric.
• Surface anomalies (sudden drops, class-specific issues) via alerts.

The sample-vs-audit overhead

Online eval is itself a labeling cost. Budget 1–5% of HITL capacity for it. The cost is paid in accuracy assurance — without it, the platform doesn't know what its own quality looks like.

Customer-facing reporting

The SLA dashboard the customer sees should include:

Headline numbers

Rolling 30-day accuracy per class, plotted against the SLA threshold as a line. Color-code by status (green / yellow / red).

Sample composition

How many documents are in the rolling sample? When was it last refreshed? Source of ground truth?

Methodology disclosure

How is accuracy measured? Tolerance rules? Inter-rater agreement? This is the transparency layer; sophisticated customers want it.

Historical trend

Accuracy over time (last 6 months). Shows trajectory; surfaces drift.

Failure-mode breakdown

For accuracy below SLA, which fields are dragging it down? Lets the customer see what's being remediated.

Calibration disclosure

Especially for sophisticated customers: "Of the fields auto-approved at confidence ≥ 0.85, observed accuracy is 0.94." Demonstrates that the calibration math is honest.