Section C · Data craft

SLA-Backed Quality & Evaluation

The single most differentiated piece of this category. How to measure extraction quality, calibrate human-in-the-loop, ship to a contracted SLA, and surface quality problems to the customer before they surface them to you.

Why this chapter is the differentiator

Most data platforms ship "best-effort" data — accuracy is good, but not contractually committed. The SLA-backed model is different: the platform commits to a specific accuracy threshold by document class, monitored continuously, with remediation if it slips.

This shapes the role:

  • Quality measurement is a first-class deliverable, not an afterthought.
  • Human-in-the-loop is part of the architecture, not a stopgap.
  • The FDE-DE owns the customer-facing quality story end to end.
  • Tradeoffs between coverage and accuracy are explicit conversations with the customer, not implicit engineering choices.

Loops at platforms in this category test this chapter heavily. Expect at least one deep-dive round on quality measurement.

What an SLA actually is

An accuracy SLA is a contracted commitment with several explicit parts:

  • What's measured. Per-document-class accuracy on a defined set of extracted fields. Not "the platform is 95% accurate" — "for master service agreements, parties / effective date / renewal terms achieve ≥ 95% field-level accuracy."
  • The measurement window. Rolling 30 days, monthly snapshot, etc.
  • The measurement method. How is ground truth established? Random sample audited by analysts? Customer-supplied gold set? Inter-rater agreement threshold?
  • What "below SLA" triggers. A remediation period (e.g., 30 days), a service credit, an escalation procedure.
  • Coverage carve-outs. What document classes / fields / time periods are excluded.

The implicit promise

Beyond the contract, the customer expects:

  • You'll surface quality drops before they do.
  • You'll explain what's happening when quality drops.
  • You'll have a remediation plan ready.
  • You won't argue about the measurement when the data is clear.

Quality metrics

Field-level accuracy

The headline number. Of extracted values, what fraction match ground truth exactly (or within an agreed tolerance)?

  • String fields: exact match, or fuzzy (Levenshtein/edit distance) within threshold. Define this in the contract.
  • Date fields: exact or within ±N days.
  • Numeric fields: exact or within ±percent.
  • Categorical fields: exact match on canonical value.

Precision / recall (field-presence)

Did the extraction find a value when there should have been one? Did it abstain when there wasn't?

  • Precision = of fields the model extracted, what fraction match ground truth.
  • Recall = of fields where ground truth says a value exists, what fraction did the model extract.

Tradeoff: aggressive extraction increases recall but decreases precision (more false-positive fields); conservative extraction is the opposite.

Document-level accuracy

Stricter: a document is "correct" only if all of its required fields are correct. Often what customers care about — "if I trust this contract row, is everything in it right?"

Confidence calibration

For thresholded HITL, calibration matters: if the model says 95% confidence, is it actually right 95% of the time? Plot a reliability diagram. Miscalibrated confidence breaks the HITL queue economics.

Ground truth

The hardest part. Quality measurement requires knowing the right answer, which means somebody had to read the documents.

Sources

  • Customer-supplied gold set. Best when available. The customer's procurement / legal team confirms the right values for a sample. Expensive for them; ideal for measurement.
  • Internal analyst audit. Platform-employed analysts re-read sampled documents and produce ground truth. More common.
  • Customer-confirmed extractions. Extractions the customer's team confirms in their workflow become labeled data. Slower to accumulate but cheap.
  • Synthetic ground truth. Avoid in production measurement. Useful for development only.

Sample size

To estimate 95% accuracy ±2 percentage points at 95% confidence, you need ~600 documents per class per measurement window. The SLA contract usually specifies the sample size in advance.

Inter-rater agreement

If two analysts produce different ground truths on the same document, your measurement noise is at least as large as their disagreement. Aim for ≥ 90% agreement on a calibration set; if you can't achieve it, the field definition is too ambiguous to measure cleanly.

HITL calibration

The HITL queue exists to catch low-confidence extractions before they hit the customer. Its economics are sensitive to model calibration.

The queue equation

If you push too much to HITL, the queue blows up — reviewers can't keep up, latency to the customer breaks SLA. If you push too little, low-confidence extractions land in customer-visible outputs, accuracy slips.

Tuning the threshold

For each document class:

  1. Plot calibration: bucket predictions by confidence (deciles), measure accuracy per bucket.
  2. Find the confidence threshold above which model accuracy meets SLA without HITL.
  3. Below that threshold, route to HITL.
  4. HITL queue throughput = volume × (1 - threshold-pass-rate). Make sure analyst capacity can absorb it.
  5. Recompute monthly — drift happens.

The "obvious" tradeoff customers ask about

Customer: "Why can't you just HITL everything for perfect accuracy?" Answer: cost and latency. HITL on 100% of documents would be uneconomic and would create 24–48 hour latency on customer-visible data. The threshold-based design is what makes SLA-backed quality scalable.

Monitoring

Three layers:

1. Live accuracy estimate

Rolling 30-day accuracy on the audit sample, per document class. Plotted with SLA threshold drawn as a line. Update daily.

2. Confidence distribution drift

If the model's confidence distribution shifts, it might be telling you something about input drift. Track the deciles of model confidence over time per document class. Sudden shifts warrant investigation.

3. HITL queue health

Queue depth, average review time, reviewer agreement on the calibration set. If queue depth grows or agreement drops, the SLA is at risk before it shows up in accuracy.

The customer-facing dashboard

Show the customer all three — not just the headline accuracy. Sophisticated customers want to see the leading indicators, and showing them builds trust.

When the SLA is at risk

The protocol

  1. Internal alert. Your monitoring fires before the customer notices. Diagnose: is this drift, regression, model change, customer-data change?
  2. Containment. Push more documents to HITL to stay above the line in the short term. Tighten the confidence threshold.
  3. Customer notification. Tell the customer proactively, before they notice. Include diagnosis, containment, and remediation timeline.
  4. Root-cause work. File platform bug if it's a model regression. Update extraction configuration if it's customer data drift. Adjust HITL guidance if it's an inter-rater issue.
  5. Remediation review. Once accuracy recovers, demonstrate the recovery with data. Don't declare victory until the rolling window is back above SLA.
The proactive-notification rule

The single biggest trust signal in this kind of deployment: you tell the customer about a quality dip before they tell you. Customers who learn about quality issues from their own users (the procurement analyst noticed an obviously wrong renewal date) are much harder to recover than customers you proactively warned. Surface bad news early.

Interview probes

Show probe 1: "What's in an extraction SLA?"

Five explicit parts. (1) What's measured — per-document-class accuracy on named fields. (2) Measurement window — usually rolling 30 days. (3) Measurement method — sample size, ground-truth source, inter-rater protocol. (4) What "below SLA" triggers — remediation period, service credits, escalation. (5) Coverage carve-outs — which classes/fields/periods are excluded. Beyond the contract, the implicit promise: proactive notification of quality drops, clear diagnosis, ready remediation.

Show probe 2: "Walk me through how you'd measure extraction accuracy."

(1) Define ground truth — customer-supplied gold set if available, internal analyst audit otherwise. (2) Sample per document class — ~600 docs for 95% accuracy ±2pp at 95% confidence. (3) Define field-level match criteria — exact for some, tolerance-based for dates/numbers. (4) Compute field-level accuracy, precision, recall, and document-level accuracy. (5) Track confidence calibration via reliability diagrams. (6) Recompute monthly; track drift.

Show probe 3: "Why not just HITL every document?"

Cost and latency. HITL on 100% of documents is uneconomic at scale and creates 24–48 hour latency on customer-visible data. The threshold-based design — automated for high-confidence, HITL for borderline — is what makes SLA-backed quality scalable. The threshold is tuned per document class against calibration data: above threshold, accuracy meets SLA without review; below, queue routes to analysts.

Show probe 4: "Accuracy on a customer's supplier contracts is trending toward the SLA line. What do you do?"

Five-step protocol. (1) Diagnose: drift, model regression, customer data change, HITL miscalibration? (2) Contain: push more docs to HITL temporarily; tighten confidence threshold. (3) Notify the customer proactively before they notice. (4) Root-cause: file platform bug, update extraction config, adjust HITL guidance — whatever applies. (5) Wait for the rolling window to recover above SLA before declaring done; don't claim victory on point-in-time metrics. The senior signal is the proactive customer notification — losing trust from a quality dip the customer discovered first is much harder to recover than from one you flagged.

Show probe 5: "What's confidence calibration and why does it matter?"

Calibration is whether the model's stated confidence matches reality — if the model says 0.9, is it right 90% of the time. A miscalibrated model breaks the HITL queue economics: if the model is over-confident, low-confidence cases get auto-approved and accuracy slips; if under-confident, the HITL queue fills with cases the model would have gotten right and reviewers are wasted. Tune thresholds against measured calibration (reliability diagrams), recompute periodically, and apply isotonic or Platt scaling if the model is systematically miscalibrated.