HITL Economics

Why HITL is the differentiator

"Best-effort" extraction doesn't justify an SLA. Customers who pay for SLA-backed accuracy are buying the right to trust the output without spot-checking. The only way to deliver that economically at corpus scale is HITL: route low-confidence extractions to human reviewers, auto-approve high-confidence ones.

The economic tension:

• Pure auto-extraction (no HITL): cheap, accuracy capped at model capability.
• Full HITL (review every extraction): high accuracy, uneconomic at scale, latency unacceptable.
• Threshold-based HITL: route the bottom N% by confidence to review. Accuracy near full-HITL; cost much lower.

The threshold is the critical lever. Its value is determined by calibration (chapter 06), reviewer capacity, and the cost ratio.

The HITL flow

1. Extraction produces structured fields with per-field confidence.
2. Routing: any field below the threshold goes to the HITL queue.
3. Queue assignment: a reviewer claims a task; sees the document, the extracted value, the suggested correction.
4. Review: reviewer confirms, corrects, or escalates ambiguity.
5. Persistence: the corrected value writes back to the extractions table with review_status = 'human_confirmed' and the reviewer's identifier.
6. Feedback: corrections become training signal for the next model version.

The piece most often missed: step 6. Reviewer corrections are the highest-quality labeled data the platform has access to. Plumbing them into training and calibration is what makes the system improve over time.

Queue design

FIFO vs priority

FIFO is the default. Priority queues let urgent extractions jump (e.g., a customer's quarter-end backfill needs to clear by Friday). Use sparingly — priority queues drift toward "everything is urgent" without enforcement.

Per-document-class queues

Different document classes need different reviewer expertise. MSAs require legal-document fluency; invoices require procurement vocabulary. Route to specialized queues; reviewer skill is a real constraint.

Customer separation

Multi-tenant platforms must separate customer queues for data-isolation reasons (GDPR, contractual confidentiality). Cross-customer reviewer assignment requires explicit access controls.

Capacity-aware backpressure

When queue depth exceeds N hours of reviewer capacity, throttle ingest. Customers expect predictable latency; an exploding queue silently violates that expectation.

SLA tiers in the queue

If your platform offers multiple SLA tiers (e.g., 4-hour vs 24-hour turnaround), the queue needs SLA-aware priority and visible deadlines per task.

Reviewer capacity

The other side of the threshold lever. Capacity = (number of reviewers) × (tasks per reviewer per hour) × (productive hours).

Per-task review time

Wildly variable by document class and field complexity. Typical ranges:

• Simple field (date, supplier name): 30 seconds to 2 minutes.
• Complex field (renewal terms, indemnification): 3 to 10 minutes.
• Ambiguous case (reviewer needs to consult the document body, not just the highlighted region): 10–30 minutes.

Productive hours

A full-time reviewer might be productive 5–6 hours per day. Plan capacity against productive hours, not headcount × 8.

Skill calibration

New reviewers are slower and less accurate. Plan a ramp: first 2 weeks at 30% of senior throughput; 4 weeks at 60%; full speed at week 8. Inter-rater agreement (next section) catches calibration drift before it costs you.

Inter-rater agreement

If two reviewers given the same task produce different answers, your measurement noise is at least as large as their disagreement. This sets a floor on achievable accuracy.

Measuring it

Periodically (weekly or monthly), assign the same task to two reviewers without telling them. Track per-field agreement rate. Aim for ≥ 90% on simple fields, ≥ 80% on complex ones.

Cohen's kappa

The standard inter-rater agreement metric, adjusts for chance agreement. Above 0.8 is excellent; 0.6–0.8 is acceptable; below 0.6 means the field definition is too ambiguous to measure reliably.

Calibration sets

Maintain a "calibration set" of 50–100 representative tasks per document class with known answers. New reviewers complete it before going live; existing reviewers refresh quarterly. Drift in calibration-set scores signals reviewer drift.

When agreement is low

Cohen's kappa < 0.6 on a field means the field definition is too fuzzy. Three options:

• Tighten the field definition (write a clearer spec for reviewers).
• Split the field into sub-fields with clearer criteria.
• Acknowledge the ambiguity and adjust the SLA — some fields are inherently noisy.

Threshold tuning

The threshold determines what fraction of extractions go to HITL. Tuning is iterative:

1. Start with a conservative threshold (e.g., 0.85). Route generously to HITL.
2. Measure: of the predictions above 0.85 (auto-approved), what's the actual accuracy on a held-out sample?
3. If auto-approved accuracy > SLA target by > 2pp: you can lower the threshold; more goes to auto-approval; reviewer cost drops.
4. If auto-approved accuracy is at or below SLA target: raise the threshold; more goes to HITL; reviewer cost rises.
5. Iterate monthly. The threshold drifts as the model and customer data change.

The threshold is per-document-class per-field. A single global threshold is operationally simpler but materially worse for cost-accuracy.

The cost-per-extraction math

Combine the pieces:

• Auto-approved cost: LLM/model inference + storage. Typically $0.001–$0.05 per document depending on model.
• HITL cost: reviewer time × loaded reviewer cost. At $25–$60/hr loaded cost and 5 minutes per task, that's $2–$5 per HITL task.
• HITL rate: fraction of extractions routed to HITL. Determined by the threshold and the model's confidence distribution.

Total per-extraction cost ≈ auto_cost + (hitl_rate × hitl_cost). At 15% HITL rate and $3 per task: $0.45 in HITL cost per document, plus $0.01–$0.05 in model cost.

Implications:

• A 10% reduction in HITL rate (achievable through better calibration or model improvement) saves $0.30 per document. At 100k documents/month, that's $30k/month.
• The model-cost ladder (Claude Haiku vs Sonnet vs Opus, GPT-3.5 vs GPT-4) trades model accuracy for cost. The breakeven is at the HITL boundary — a more expensive model with higher accuracy reduces HITL rate; whether that's worth it is a per-customer calculation.

Feedback loop

The HITL queue is also the platform's training-data engine. Reviewer corrections are high-quality labels; they should flow back into:

• Calibration retraining: every correction is a (raw_confidence, was_correct) pair. Aggregate weekly; refit calibrators.
• Model retraining: corrections become labeled examples for the next model version.
• Prompt iteration (for LLM extraction): patterns in corrections inform prompt edits. "The model keeps mistaking termination-notice for renewal-notice → add a disambiguation example to the prompt."
• Customer-specific configuration: corrections may surface customer-specific patterns ("their MSAs use a non-standard renewal clause structure") that warrant per-customer prompt customization.

The feedback loop is what makes the platform improve over time. Without it, HITL is just expensive labeling that never gets cheaper.