Section C · Quality & eval

HITL Economics

Human-in-the-Loop is what lets SLA-backed accuracy scale economically. Queue design, reviewer capacity, threshold tuning, and the cost arithmetic that determines where the threshold should sit.

Why HITL is the differentiator

"Best-effort" extraction doesn't justify an SLA. Customers who pay for SLA-backed accuracy are buying the right to trust the output without spot-checking. The only way to deliver that economically at corpus scale is HITL: route low-confidence extractions to human reviewers, auto-approve high-confidence ones.

The economic tension:

Pure auto-extraction (no HITL): cheap, accuracy capped at model capability.
Full HITL (review every extraction): high accuracy, uneconomic at scale, latency unacceptable.
Threshold-based HITL: route the bottom N% by confidence to review. Accuracy near full-HITL; cost much lower.

The threshold is the critical lever. Its value is determined by calibration (chapter 06), reviewer capacity, and the cost ratio.

The HITL flow

Extraction produces structured fields with per-field confidence.
Routing: any field below the threshold goes to the HITL queue.
Queue assignment: a reviewer claims a task; sees the document, the extracted value, the suggested correction.
Review: reviewer confirms, corrects, or escalates ambiguity.
Persistence: the corrected value writes back to the extractions table with review_status = 'human_confirmed' and the reviewer's identifier.
Feedback: corrections become training signal for the next model version.

The piece most often missed: step 6. Reviewer corrections are the highest-quality labeled data the platform has access to. Plumbing them into training and calibration is what makes the system improve over time.

Queue design

FIFO vs priority

FIFO is the default. Priority queues let urgent extractions jump (e.g., a customer's quarter-end backfill needs to clear by Friday). Use sparingly — priority queues drift toward "everything is urgent" without enforcement.

Per-document-class queues

Different document classes need different reviewer expertise. MSAs require legal-document fluency; invoices require procurement vocabulary. Route to specialized queues; reviewer skill is a real constraint.

Customer separation

Multi-tenant platforms must separate customer queues for data-isolation reasons (GDPR, contractual confidentiality). Cross-customer reviewer assignment requires explicit access controls.

Capacity-aware backpressure

When queue depth exceeds N hours of reviewer capacity, throttle ingest. Customers expect predictable latency; an exploding queue silently violates that expectation.

SLA tiers in the queue

If your platform offers multiple SLA tiers (e.g., 4-hour vs 24-hour turnaround), the queue needs SLA-aware priority and visible deadlines per task.

Reviewer capacity

The other side of the threshold lever. Capacity = (number of reviewers) × (tasks per reviewer per hour) × (productive hours).

Per-task review time

Wildly variable by document class and field complexity. Typical ranges:

Simple field (date, supplier name): 30 seconds to 2 minutes.
Complex field (renewal terms, indemnification): 3 to 10 minutes.
Ambiguous case (reviewer needs to consult the document body, not just the highlighted region): 10–30 minutes.

Productive hours

A full-time reviewer might be productive 5–6 hours per day. Plan capacity against productive hours, not headcount × 8.

Skill calibration

New reviewers are slower and less accurate. Plan a ramp: first 2 weeks at 30% of senior throughput; 4 weeks at 60%; full speed at week 8. Inter-rater agreement (next section) catches calibration drift before it costs you.

Inter-rater agreement

If two reviewers given the same task produce different answers, your measurement noise is at least as large as their disagreement. This sets a floor on achievable accuracy.

Measuring it

Periodically (weekly or monthly), assign the same task to two reviewers without telling them. Track per-field agreement rate. Aim for ≥ 90% on simple fields, ≥ 80% on complex ones.

Cohen's kappa

The standard inter-rater agreement metric, adjusts for chance agreement. Above 0.8 is excellent; 0.6–0.8 is acceptable; below 0.6 means the field definition is too ambiguous to measure reliably.

Calibration sets

Maintain a "calibration set" of 50–100 representative tasks per document class with known answers. New reviewers complete it before going live; existing reviewers refresh quarterly. Drift in calibration-set scores signals reviewer drift.

When agreement is low

Cohen's kappa < 0.6 on a field means the field definition is too fuzzy. Three options:

Tighten the field definition (write a clearer spec for reviewers).
Split the field into sub-fields with clearer criteria.
Acknowledge the ambiguity and adjust the SLA — some fields are inherently noisy.

Threshold tuning

The threshold determines what fraction of extractions go to HITL. Tuning is iterative:

Start with a conservative threshold (e.g., 0.85). Route generously to HITL.
Measure: of the predictions above 0.85 (auto-approved), what's the actual accuracy on a held-out sample?
If auto-approved accuracy > SLA target by > 2pp: you can lower the threshold; more goes to auto-approval; reviewer cost drops.
If auto-approved accuracy is at or below SLA target: raise the threshold; more goes to HITL; reviewer cost rises.
Iterate monthly. The threshold drifts as the model and customer data change.

The threshold is per-document-class per-field. A single global threshold is operationally simpler but materially worse for cost-accuracy.

The cost-per-extraction math

Combine the pieces:

Auto-approved cost: LLM/model inference + storage. Typically $0.001–$0.05 per document depending on model.
HITL cost: reviewer time × loaded reviewer cost. At $25–$60/hr loaded cost and 5 minutes per task, that's $2–$5 per HITL task.
HITL rate: fraction of extractions routed to HITL. Determined by the threshold and the model's confidence distribution.

Total per-extraction cost ≈ auto_cost + (hitl_rate × hitl_cost). At 15% HITL rate and $3 per task: $0.45 in HITL cost per document, plus $0.01–$0.05 in model cost.

Implications:

A 10% reduction in HITL rate (achievable through better calibration or model improvement) saves $0.30 per document. At 100k documents/month, that's $30k/month.
The model-cost ladder (Claude Haiku vs Sonnet vs Opus, GPT-3.5 vs GPT-4) trades model accuracy for cost. The breakeven is at the HITL boundary — a more expensive model with higher accuracy reduces HITL rate; whether that's worth it is a per-customer calculation.

Feedback loop

The HITL queue is also the platform's training-data engine. Reviewer corrections are high-quality labels; they should flow back into:

Calibration retraining: every correction is a (raw_confidence, was_correct) pair. Aggregate weekly; refit calibrators.
Model retraining: corrections become labeled examples for the next model version.
Prompt iteration (for LLM extraction): patterns in corrections inform prompt edits. "The model keeps mistaking termination-notice for renewal-notice → add a disambiguation example to the prompt."
Customer-specific configuration: corrections may surface customer-specific patterns ("their MSAs use a non-standard renewal clause structure") that warrant per-customer prompt customization.

The feedback loop is what makes the platform improve over time. Without it, HITL is just expensive labeling that never gets cheaper.

✓ Check yourself

Answer from memory before opening each one.

Review volume is 2,000 field tasks a day at about 3 minutes each, and reviewers are productive 5 hours a day. How many reviewers do you need, and what does planning on 8 hours give?

Show answer
The work is 6,000 minutes, or 100 hours, so 20 reviewers at 5 productive hours each. Planning on 8 hours gives 12.5, which leaves you about 7 reviewers short and the queue growing every day. See reviewer capacity.
Auto-approved accuracy measured on a held-out sample is 97.5% against an SLA of 95%. What do you do with the threshold, and how often?

Show answer
The gap is above 2 points, so lower the threshold to send less to review and recheck the auto-approved accuracy on a fresh sample. Do it per document class and field, and repeat monthly because the model and data shift. See threshold tuning.
Better calibration cuts the review rate from 20% to 14% at $3 per task over 80,000 documents a month. What is the saving, and what limits how much you should pay for it?

Show answer
Six points at $3 is $0.18 per document, about $14,400 a month. The spend is capped by that saving and by whether accuracy holds at the new rate, which you must re-measure. See the cost-per-extraction math.
Two reviewers agree on 85% of renewal-term tasks, yet kappa is 0.45. How can both be true, and what are your options?

Show answer
Kappa discounts agreement expected by chance, and when most tasks share one answer chance agreement is high. A value of 0.45 means the field definition is too loose to measure reliably. Tighten the written spec, split the field into sub-fields, or accept the noise and adjust the SLA. See inter-rater agreement.
A year in, the review rate has not fallen. What is probably missing?

Show answer
The corrections are not flowing back. Feed them into calibration refits, model retraining, prompt changes and per-customer configuration, or review stays expensive labelling that never gets cheaper. See the feedback loop.
A customer's quarter-end backfill arrives at five times normal volume. Do you jump it ahead in the queue or throttle ingest?

Show answer
Prioritise it if it has a real deadline, with an expiry so priority does not spread to everything. Throttle ingest as well once queue depth passes your hours-of-capacity limit, because an exploding queue silently breaks the latency promise to other customers. See queue design.

Takeaway

Review of low-confidence fields is what makes an accuracy SLA affordable at scale: people check the doubtful values and the system approves the rest.
The threshold comes from calibration, reviewer capacity and the cost ratio, and it is tuned monthly per document class and field.
Plan reviewer capacity on productive hours, about five to six a day, and allow for a ramp for new reviewers.
Inter-rater agreement sets the ceiling on measurable accuracy, and a low kappa means the field definition needs work.
Cost per extraction is the model cost plus the review rate times the cost per review task, and the review rate dominates.
Reviewer corrections must flow into calibration, models, prompts and customer configuration, or the cost never falls.

A threshold is only as good as the accuracy you can measure, which is the subject of the next chapter. → Eval Methodology