Section C · Quality & eval

Eval Methodology

An SLA-backed platform measures accuracy continuously. Ground-truth sourcing, sample sizing, the metrics that matter, the cadence of offline vs online eval, and how to report it to the customer.

Why eval is the differentiator

Two organizations can have identical extraction stacks but very different operational outcomes — because one of them measures accuracy rigorously and the other doesn't. Without measurement:

You can't tell when quality slips until the customer complains.
You can't ship a model change confidently — you have no scoreboard.
You can't price the SLA — you don't know what you can stand behind.
You can't tune the HITL threshold — you don't know the auto-approved accuracy.

Eval isn't optional. The discipline of measurement is what makes SLA-backed accuracy a real product rather than a marketing claim.

Ground-truth sources

Eval requires knowing the right answer for each test example. Four sources, in order of preference:

1. Customer-supplied gold set

The customer's team confirms the right values for a sample of their documents. Best quality (it's their domain), best customer credibility (they can't argue with their own labels later). Most expensive to obtain — customer time is hard to get.

2. Internal analyst audit

Platform-employed analysts re-read sampled documents and produce ground truth. Most common in practice. Cheaper than customer time; quality is high if analysts are well-trained.

3. Customer-confirmed extractions

Extractions that the customer's reviewers confirmed in the HITL workflow. Free in volume; biased toward documents that hit HITL (i.e., low-confidence ones). Useful for calibration; less useful for top-line accuracy measurement.

4. Synthetic ground truth

Generate fake contracts with known ground truth. Useful for development testing only — never for SLA measurement against real customers.

Inter-rater agreement on ground truth

Even ground truth has noise. Two analysts labeling the same document independently should agree ≥ 90% on simple fields. Below that, the field definition is too ambiguous to measure cleanly — fix the spec before fixing the model.

Sample sizing

To estimate a population accuracy with confidence, you need enough labeled samples. The math:

For estimating a proportion p with margin of error E at confidence level (1−α), sample size n ≈ z² × p × (1−p) / E², where z is the z-score for the confidence level.

For 95% accuracy with ±2 percentage points at 95% confidence: n ≈ 457. Round up to 500 for safety. For ±1pp: n ≈ 1800. For ±5pp: n ≈ 73.

This is per-document-class. If your SLA covers 5 classes, you need ~500 labeled samples per class to measure each independently.

sample_size.py

from math import ceil
from scipy.stats import norm

def required_sample_size(target_accuracy: float, margin: float, confidence: float = 0.95) -> int:
    """For a binary outcome (correct / not), how many samples for the margin?"""
    p = target_accuracy
    z = norm.ppf(1 - (1 - confidence) / 2)
    n = (z ** 2) * p * (1 - p) / (margin ** 2)
    return ceil(n)

print(required_sample_size(0.95, 0.02))   # → 457
print(required_sample_size(0.95, 0.01))   # → 1825
print(required_sample_size(0.95, 0.05))   # → 73

Rolling windows

SLA measurement is typically a rolling 30-day window. The sample is recomputed daily; oldest examples drop off. Keeps the metric responsive to recent drift without single-day noise.

Metrics

Field-level accuracy

Of all extracted field values, what fraction match ground truth (within tolerance)? The headline number on the SLA dashboard.

String fields: exact match, or fuzzy (Levenshtein, normalized casing/whitespace) within a tolerance — defined in the contract.
Date fields: exact, or within ±N days.
Numeric fields: exact, or within ±X% of the true value.
Categorical fields: exact match against canonical taxonomy.

Precision and recall

For field presence (did the extractor find a value when there should be one, abstain when there shouldn't):

Precision = of fields the extractor returned, what fraction matched ground truth.
Recall = of fields where ground truth said a value exists, what fraction did the extractor find.

Document-level accuracy

Stricter — a document is correct only if all of its required fields are correct. Often what customers care about ("if I trust this row, is everything in it right?").

Lift over baseline

What's the accuracy of a trivial baseline (e.g., regex-only, or "always return null")? Your model's accuracy minus the baseline shows the model's actual contribution.

Calibration metrics

ECE and reliability diagrams (chapter 06). Track alongside accuracy; if calibration drifts, the HITL threshold needs recompute even before accuracy slips.

Per-class measurement

Aggregating accuracy across document classes is misleading. A 95% overall accuracy might hide:

MSAs at 98%.
SOWs at 96%.
Supplier agreements at 91% (below SLA, hidden by aggregation).

Always report per-class. If your SLA contract has per-class thresholds, the per-class number is what's reportable.

Per-field within class

Drill one level deeper: per-field within each class. The dashboard's heatmap is doc-class × field × accuracy, with the SLA threshold drawn as a line. Failures are immediately visible.

Offline eval before deploys

Before any model or prompt change ships, run it against a frozen eval set.

The frozen eval set

500–1000 documents per class with labeled ground truth.
Sampled to be representative of production distribution.
Frozen — doesn't change month-to-month. Lets you compare results across model versions over time.
Versioned. When the eval set is updated, it's a new version (eval-v2.1) so you can interpret historical reports correctly.

The eval run

Run the new model/prompt against the eval set.
Score per-field accuracy, precision, recall, calibration.
Generate a comparison report vs the current production model.
Approve / reject the change based on the report.

Build this into CI. Every PR that touches extraction code runs the eval and fails CI if accuracy regresses on any class above the noise floor.

Online eval in production

Offline eval doesn't catch drift in production. Online eval continuously samples production output for audit.

The online sample

Randomly sample N documents per day across each class.
Route them to internal analysts for ground-truth labeling (or use customer-confirmed extractions if available).
Score and aggregate into the rolling 30-day metric.
Surface anomalies (sudden drops, class-specific issues) via alerts.

The sample-vs-audit overhead

Online eval is itself a labeling cost. Budget 1–5% of HITL capacity for it. The cost is paid in accuracy assurance — without it, the platform doesn't know what its own quality looks like.

Customer-facing reporting

The SLA dashboard the customer sees should include:

Headline numbers

Rolling 30-day accuracy per class, plotted against the SLA threshold as a line. Color-code by status (green / yellow / red).

Sample composition

How many documents are in the rolling sample? When was it last refreshed? Source of ground truth?

Methodology disclosure

How is accuracy measured? Tolerance rules? Inter-rater agreement? This is the transparency layer; sophisticated customers want it.

Historical trend

Accuracy over time (last 6 months). Shows trajectory; surfaces drift.

Failure-mode breakdown

For accuracy below SLA, which fields are dragging it down? Lets the customer see what's being remediated.

Calibration disclosure

Especially for sophisticated customers: "Of the fields auto-approved at confidence ≥ 0.85, observed accuracy is 0.94." Demonstrates that the calibration math is honest.

The customer-facing dashboard discipline

The instinct is to hide the dashboard until accuracy is green. The opposite is right. Show the customer the dashboard early and often. Building familiarity when things are healthy means they trust the dashboard when things slip. Hiding it until renewal builds suspicion.

✓ Check yourself

Answer from memory before opening each one.

You need to measure accuracy at 95% with a margin of plus or minus 2 points, at 95% confidence, across 5 document classes. How many labelled documents, and what does halving the margin do?

Show answer
About 457 per class, so roughly 2,285 in total. Halving the margin to 1 point needs about four times as many, near 1,825 per class, because sample size scales with the inverse square of the margin. See sample sizing.
A teammate wants to report headline accuracy from extractions that customer reviewers confirmed in the review queue. Why is that a poor measure?

Show answer
Only low-confidence values reach reviewers, so the sample is biased toward hard cases and says nothing about the auto-approved values that make up most of the output. It suits calibration, not top-line accuracy. Use a random audit sample instead. See ground-truth sources.
Each of 10 required fields is right 98% of the time, independently. What is the document-level accuracy, and when does the customer care about it?

Show answer
About 82%, since 0.98 to the tenth power is 0.817. The customer cares when they load whole rows downstream and ask whether everything in a row is right. A strong field-level figure can hide a weak document-level one. See metrics.
Overall accuracy is 95.1% against a 95% contract, and the contract sets 95% per class. Why might you still be in breach, and what do you do?

Show answer
High-volume classes dominate an overall figure and can hide a weak, low-volume class such as supplier agreements. Report per class, then per field within class, with the SLA drawn as a line. See per-class measurement.
The team wants to add 100 newly found hard documents to the frozen eval set each month. What is the problem, and how do you handle it?

Show answer
Scores from before and after the change are no longer comparable, so a regression can look like a shift in the set. Publish the change as a new versioned set and rerun the current production model on it to get a fresh baseline. See offline eval before deploys.
Your sponsor wants the SLA dashboard kept private until accuracy is green. Do you agree?

Show answer
No. Show it early, when numbers are healthy, so the customer learns how to read it and trusts it when a number slips. Hiding it until renewal reads as concealment. Include the sample size, ground-truth source and method so the number can be trusted. See customer-facing reporting.

Takeaway

Measuring accuracy is what turns an SLA from a claim into a product, and it also enables safe model changes and threshold tuning.
Prefer customer gold sets, then internal analyst audits; synthetic data is for development only.
Sample size is about z squared times p times one minus p over the margin squared, about 457 labels per class for a 2-point margin.
Report per class and per field, because aggregates hide the weak cells.
Run a frozen, versioned eval set in CI before each model or prompt change, and a random production audit sample continuously.
Show the customer the dashboard with sample size, method and calibration, early and often.

A measurement taken once goes stale, so the next chapter covers watching for change. → Drift Detection & Monitoring