Section C · Data craft

SLA-Backed Quality & Evaluation

The single most differentiated piece of this category. How to measure extraction quality, calibrate human-in-the-loop, ship to a contracted SLA, and surface quality problems to the customer before they surface them to you.

Why this chapter is the differentiator

Most data platforms ship "best-effort" data — accuracy is good, but not contractually committed. The SLA-backed model is different: the platform commits to a specific accuracy threshold by document class, monitored continuously, with remediation if it slips.

This shapes the role:

Quality measurement is a first-class deliverable, not an afterthought.
Human-in-the-loop is part of the architecture, not a stopgap.
The FDE-DE owns the customer-facing quality story end to end.
Tradeoffs between coverage and accuracy are explicit conversations with the customer, not implicit engineering choices.

Loops at platforms in this category test this chapter heavily. Expect at least one deep-dive round on quality measurement.

What an SLA actually is

An accuracy SLA is a contracted commitment with several explicit parts:

What's measured. Per-document-class accuracy on a defined set of extracted fields. Not "the platform is 95% accurate" — "for master service agreements, parties / effective date / renewal terms achieve ≥ 95% field-level accuracy."
The measurement window. Rolling 30 days, monthly snapshot, etc.
The measurement method. How is ground truth established? Random sample audited by analysts? Customer-supplied gold set? Inter-rater agreement threshold?
What "below SLA" triggers. A remediation period (e.g., 30 days), a service credit, an escalation procedure.
Coverage carve-outs. What document classes / fields / time periods are excluded.

The implicit promise

Beyond the contract, the customer expects:

You'll surface quality drops before they do.
You'll explain what's happening when quality drops.
You'll have a remediation plan ready.
You won't argue about the measurement when the data is clear.

Quality metrics

Field-level accuracy

The headline number. Of extracted values, what fraction match ground truth exactly (or within an agreed tolerance)?

String fields: exact match, or fuzzy (Levenshtein/edit distance) within threshold. Define this in the contract.
Date fields: exact or within ±N days.
Numeric fields: exact or within ±percent.
Categorical fields: exact match on canonical value.

Precision / recall (field-presence)

Did the extraction find a value when there should have been one? Did it abstain when there wasn't?

Precision = of fields the model extracted, what fraction match ground truth.
Recall = of fields where ground truth says a value exists, what fraction did the model extract.

Tradeoff: aggressive extraction increases recall but decreases precision (more false-positive fields); conservative extraction is the opposite.

Document-level accuracy

Stricter: a document is "correct" only if all of its required fields are correct. Often what customers care about — "if I trust this contract row, is everything in it right?"

Confidence calibration

For thresholded HITL, calibration matters: if the model says 95% confidence, is it actually right 95% of the time? Plot a reliability diagram. Miscalibrated confidence breaks the HITL queue economics.

Ground truth

The hardest part. Quality measurement requires knowing the right answer, which means somebody had to read the documents.

Sources

Customer-supplied gold set. Best when available. The customer's procurement / legal team confirms the right values for a sample. Expensive for them; ideal for measurement.
Internal analyst audit. Platform-employed analysts re-read sampled documents and produce ground truth. More common.
Customer-confirmed extractions. Extractions the customer's team confirms in their workflow become labeled data. Slower to accumulate but cheap.
Synthetic ground truth. Avoid in production measurement. Useful for development only.

Sample size

To estimate 95% accuracy ±2 percentage points at 95% confidence, you need ~460 documents per class per measurement window. The SLA contract usually specifies the sample size in advance.

Inter-rater agreement

If two analysts produce different ground truths on the same document, your measurement noise is at least as large as their disagreement. Aim for ≥ 90% agreement on a calibration set; if you can't achieve it, the field definition is too ambiguous to measure cleanly.

HITL calibration

The HITL queue exists to catch low-confidence extractions before they hit the customer. Its economics are sensitive to model calibration.

The queue equation

If you push too much to HITL, the queue blows up — reviewers can't keep up, latency to the customer breaks SLA. If you push too little, low-confidence extractions land in customer-visible outputs, accuracy slips.

Tuning the threshold

For each document class:

Plot calibration: bucket predictions by confidence (deciles), measure accuracy per bucket.
Find the confidence threshold above which model accuracy meets SLA without HITL.
Below that threshold, route to HITL.
HITL queue throughput = volume × (1 - threshold-pass-rate). Make sure analyst capacity can absorb it.
Recompute monthly — drift happens.

The "obvious" tradeoff customers ask about

Customer: "Why can't you just HITL everything for perfect accuracy?" Answer: cost and latency. HITL on 100% of documents would be uneconomic and would create 24–48 hour latency on customer-visible data. The threshold-based design is what makes SLA-backed quality scalable.

Monitoring

Three layers:

1. Live accuracy estimate

Rolling 30-day accuracy on the audit sample, per document class. Plotted with SLA threshold drawn as a line. Update daily.

2. Confidence distribution drift

If the model's confidence distribution shifts, it might be telling you something about input drift. Track the deciles of model confidence over time per document class. Sudden shifts warrant investigation.

3. HITL queue health

Queue depth, average review time, reviewer agreement on the calibration set. If queue depth grows or agreement drops, the SLA is at risk before it shows up in accuracy.

The customer-facing dashboard

Show the customer all three — not just the headline accuracy. Sophisticated customers want to see the leading indicators, and showing them builds trust.

When the SLA is at risk

The protocol

Internal alert. Your monitoring fires before the customer notices. Diagnose: is this drift, regression, model change, customer-data change?
Containment. Push more documents to HITL to stay above the line in the short term. Tighten the confidence threshold.
Customer notification. Tell the customer proactively, before they notice. Include diagnosis, containment, and remediation timeline.
Root-cause work. File platform bug if it's a model regression. Update extraction configuration if it's customer data drift. Adjust HITL guidance if it's an inter-rater issue.
Remediation review. Once accuracy recovers, demonstrate the recovery with data. Don't declare victory until the rolling window is back above SLA.

The proactive-notification rule

The single biggest trust signal in this kind of deployment: you tell the customer about a quality dip before they tell you. Customers who learn about quality issues from their own users (the procurement analyst noticed an obviously wrong renewal date) are much harder to recover than customers you proactively warned. Surface bad news early.

Interview probes

Show probe 1: "What's in an extraction SLA?"

Five explicit parts. (1) What's measured — per-document-class accuracy on named fields. (2) Measurement window — usually rolling 30 days. (3) Measurement method — sample size, ground-truth source, inter-rater protocol. (4) What "below SLA" triggers — remediation period, service credits, escalation. (5) Coverage carve-outs — which classes/fields/periods are excluded. Beyond the contract, the implicit promise: proactive notification of quality drops, clear diagnosis, ready remediation.

Show probe 2: "Walk me through how you'd measure extraction accuracy."

(1) Define ground truth — customer-supplied gold set if available, internal analyst audit otherwise. (2) Sample per document class — ~460 docs for 95% accuracy ±2pp at 95% confidence. (3) Define field-level match criteria — exact for some, tolerance-based for dates/numbers. (4) Compute field-level accuracy, precision, recall, and document-level accuracy. (5) Track confidence calibration via reliability diagrams. (6) Recompute monthly; track drift.

Show probe 3: "Why not just HITL every document?"

Cost and latency. HITL on 100% of documents is uneconomic at scale and creates 24–48 hour latency on customer-visible data. The threshold-based design — automated for high-confidence, HITL for borderline — is what makes SLA-backed quality scalable. The threshold is tuned per document class against calibration data: above threshold, accuracy meets SLA without review; below, queue routes to analysts.

Show probe 4: "Accuracy on a customer's supplier contracts is trending toward the SLA line. What do you do?"

Five-step protocol. (1) Diagnose: drift, model regression, customer data change, HITL miscalibration? (2) Contain: push more docs to HITL temporarily; tighten confidence threshold. (3) Notify the customer proactively before they notice. (4) Root-cause: file platform bug, update extraction config, adjust HITL guidance — whatever applies. (5) Wait for the rolling window to recover above SLA before declaring done; don't claim victory on point-in-time metrics. The senior signal is the proactive customer notification — losing trust from a quality dip the customer discovered first is much harder to recover than from one you flagged.

Show probe 5: "What's confidence calibration and why does it matter?"

Calibration is whether the model's stated confidence matches reality — if the model says 0.9, is it right 90% of the time. A miscalibrated model breaks the HITL queue economics: if the model is over-confident, low-confidence cases get auto-approved and accuracy slips; if under-confident, the HITL queue fills with cases the model would have gotten right and reviewers are wasted. Tune thresholds against measured calibration (reliability diagrams), recompute periodically, and apply isotonic or Platt scaling if the model is systematically miscalibrated.

✓ Check yourself

Answer from memory before opening each one.

A contract says the platform is "95% accurate". What is missing before that sentence is an enforceable accuracy commitment?

Show answer
It must say what is measured, per document class and named fields, over which window, by which method and ground truth, what below-target triggers, and what is carved out. Without these, the two sides will argue about the measurement instead of the data. See what an SLA actually is.
You promise 95% accuracy, measured to within 2 points at 95% confidence. Roughly how many audited documents per class do you need, and what happens to that number if the true rate is nearer 85%?

Show answer
About 460, from 1.96 squared times 0.95 times 0.05 divided by 0.02 squared. At 85% accuracy the same precision needs about 1,225, because variance grows as the rate moves away from 100%. Agree the sample size in the contract. See ground truth.
Field-level accuracy is 96% across ten required fields, but the customer says about a third of the rows they open contain an error. Reconcile the two numbers.

Show answer
If errors are roughly independent, the chance that all ten fields are right is 0.96 to the tenth power, about 66%, so a third of rows have at least one error. Customers feel document-level accuracy. Report both. See quality metrics.
Buckets of extractions with confidence above 0.85 turn out to be right only 75% of the time. What is happening, and what do you change?

Show answer
The model is overconfident, so auto-approval leaks errors into customer-visible data. Raise the threshold per document class until the auto-approved bucket meets the SLA, check that reviewers can absorb the larger queue, and recompute monthly because drift happens. See HITL calibration. The queue those reviewers work from is described in human-in-the-loop.
A customer asks why you cannot send every document to a human for perfect accuracy. Answer on the economics, and say when full review would be right.

Show answer
Review of everything is uneconomic at scale and adds a day or two of delay to customer-visible data. Thresholded review is what makes the SLA affordable. Full review fits a small corpus where each error is very costly. See HITL calibration.
Rolling accuracy has slipped to 94.2% against a 95% line. Give the order of actions, including what you tell the customer and when. Which two monitors would have moved first?

Show answer
Diagnose, contain by routing more to review, and tell the customer before they notice. Then fix the root cause and show recovery over the full rolling window. Queue depth and reviewer agreement, or a shift in the confidence distribution, usually move before accuracy. See when the SLA is at risk.

Takeaway

Under an accuracy SLA, quality is contracted: per class, on named fields, over a window, by a stated method, with a remedy and carve-outs.
Field-level accuracy flatters; document-level accuracy, precision and recall show what customers feel.
Ground truth is the hardest part; about 460 audited documents give plus or minus 2 points at a true rate near 95%, and weak reviewer agreement means the field definition is too loose.
Set the human-review threshold per class from a calibration plot, check queue capacity, and recompute monthly.
Watch live accuracy, confidence drift and queue health; the last two lead.
On a breach, contain, tell the customer before they notice, fix the cause, and declare recovery only when the rolling window is back above the line.

Next is the coding round that tests the data work above. → Coding Fundamentals