Evaluation & Quality for HR Agents

Why evals matter — even more in HR than in compliance

HR agents fail in ways that are quiet and consequential. A bad policy answer goes to one employee; they read it; they make a decision based on it. The error doesn't bounce back as a stack trace. The error becomes a leave taken on a wrong basis, a benefit forfeited, a manager misled, a regulatory exposure created.

This makes pre-deployment evals non-negotiable and continuous post-deployment monitoring table stakes. The interview question "how do you know your agent is good enough?" has one acceptable answer: "I have a labeled eval set, I track these metrics against it, I have thresholds for shipping, and I rerun on every model or prompt change."

The shape of an eval

An eval is three things:

1. A dataset — inputs paired with expected outputs (or expected behaviors / rubrics).
2. A scorer — code or LLM-as-judge that grades outputs.
3. A run harness — runs the agent against the dataset, aggregates scores, surfaces regressions.

For HR agents, the dataset is the hardest piece. You can't just scrape a benchmark — you have to curate from your own ticket history, your own policy library, your own past survey runs. Building the eval set is half the job.

Policy retrieval accuracy

For the policy Q&A pattern, three layered metrics:

def evaluate_policy_qa(eval_set, agent):
    results = []
    for ex in eval_set:
        out = agent.answer(question=ex["question"], country=ex["country"])
        results.append({
            "id": ex["id"],
            "retrieval_recall@5": ex["correct_chunk_id"] in [c["id"] for c in out.retrieved],
            "answer_correct": llm_judge_correctness(out.answer, ex["reference"]),
            "all_claims_cited": all_claims_have_citation(out.answer, out.citations),
            "right_jurisdiction": all(c["country"] == ex["country"] for c in out.citations),
        })
    return results

Sentiment & theme quality (survey synthesis)

For survey synthesis, you can't ground-truth themes the way you can ground-truth policy answers. Approach:

• Two-rater agreement — two humans independently label a sample of 100 responses with theme + sentiment. Cohen's kappa > 0.7 is a fair bar. If humans can't agree, neither can the agent.
• Agent-vs-human alignment — agent labels the same sample. Score against the human consensus.
• Stability across runs — run the agent 3 times on the same sample with temperature=0. Stable clusters?
• Coverage — does the agent capture rare-but-important themes (e.g. 3 mentions of harassment in 500 responses)? Recall on the long tail matters more than precision.

PII leakage tests

A non-negotiable eval surface. You're constructing adversarial inputs to verify the agent doesn't leak employee data across boundaries.

Fairness & bias checks

Several HR patterns have direct fairness implications: promotion-cycle analytics, recruiter pipeline summary, survey synthesis. You should know how to design checks even if you've never run them in production:

• Demographic parity in agent outputs — does the agent's "tier yes / no / discuss" distribution differ across protected classes? Compute confidence intervals; flag deltas.
• Equal accuracy across groups — false positive rate and false negative rate per group should be comparable.
• Counterfactual probing — swap names / gendered pronouns in inputs, see if outputs change. They shouldn't.
• Calibration by group — when agent says "high confidence," is it equally accurate across groups?

Edge-case evals — the HR-specific list

Build named eval examples around the cases that bite production. A starter list:

• Multi-jurisdiction employee — works in UK, paid through US entity, moves to Singapore mid-year. Policy retrieval must respect the right jurisdiction at the right time.
• Contractor → employee conversion — tenure questions get confusing. Continuous service date vs. hire date.
• Rehire — duplicate-worker risk. Tenure-based benefits trigger differently.
• Leave-of-absence return — comp may have changed during leave; benefits eligibility windows reopen.
• Future-dated change in flight — agent asked about Jane on 2026-07-01 when she has a future-dated promo to 2026-08-01.
• Payroll lock-window queries — agent must refuse certain proposed actions.
• Termination during onboarding — yes, it happens. Rescinded offers, no-shows.
• Worker on EOR — Workday record exists but employment is through a third party. Many actions don't apply.
• Personnel data with non-ASCII characters — names from 70+ countries. Don't assume Latin-1.
• Effective-date in the past — retro changes flag for higher scrutiny.

Each of these is a row in your eval set with the expected agent behavior.

LLM-as-judge — when and how

For tasks without a clean reference answer (narrative summaries, drafted emails), LLM-as-judge is acceptable with discipline:

• Use a different / larger model than the system under test (Opus judging a Sonnet agent)
• Provide a rubric, not just "rate 1-5"
• Score on orthogonal dimensions — accuracy, completeness, tone, citation, refusal-when-appropriate
• Calibrate against human scores on a held-out sample. If LLM-judge agreement with humans is < 0.7, the judge isn't usable yet.

RUBRIC = """
Score the agent's policy answer on a 0-3 scale per dimension:
- Accuracy: are the factual claims correct against the reference policy?
- Completeness: does it address the employee's specific situation (country, role)?
- Citations: does every claim cite a specific chunk?
- Tone: clear, professional, employee-respecting?
- Boundary: appropriately refuses or routes when out-of-scope?

Output JSON: {accuracy:int, completeness:int, citations:int, tone:int, boundary:int, notes:str}
"""

def llm_judge(answer, reference_policy):
    return client.messages.create(
        model="claude-opus-4-5",  # larger than system under test
        max_tokens=512,
        system=RUBRIC,
        messages=[{"role": "user", "content": f"REFERENCE:\n{reference_policy}\n\nAGENT ANSWER:\n{answer}"}],
    )

Regression discipline

Without enforced regression discipline, models drift. The minimum:

• Every prompt or model change runs the full eval set in CI
• A per-metric threshold gates promotion to production
• PII tests are zero-tolerance — any regression blocks merge
• Fairness metrics have absolute thresholds and delta-vs-baseline thresholds (no sliding backward)
• Failed evals are not silently re-baselined — they go to ownership for explicit review

Observability in production

Pre-deployment evals catch known cases. Production observability catches the unknown. Instrument:

• Run-level metrics — duration, token counts, tool call counts, cost, model used
• Outcome metrics — approval rate (HITL), edit rate (humans edited before approving), reject rate
• Tool failure rates — by tool, by error class
• Refusal rate — by reason (out-of-scope, payroll lock, missing data)
• Drift signals — week-over-week change in approval/reject ratios, in retrieval recall on shadow eval samples
• Sampled human review — even auto-committed runs get periodic spot-check

The metric that matters most in this domain is edit rate. If humans are editing 60% of drafted responses, the agent isn't ready for higher autonomy. If it's 5%, you have room to loosen.

Interview framings worth memorizing

• "I'd build the eval set first, in partnership with HRKX, from real historical tickets. The agent comes second."
• "PII tests are zero-tolerance. A regression there is a halt, not a debate."
• "For survey synthesis I use LLM-as-judge with a rubric, calibrated against humans on a held-out sample. If the judge can't agree with humans, the metric is noise."
• "Edit rate is my primary production health signal. It's the closest proxy to 'is the human in the loop actually trusting this?'"
• "Fairness in promotion-cycle outputs is a partnered question with Legal and DEI. I bring the instrumentation; the thresholds are not my unilateral call."