Evaluation & Quality

Why finance evals are different

Chatbot evals tend to measure helpfulness, harmlessness, tone, factuality on a small benchmark. Finance evals measure something much narrower and harder:

• Numerical correctness to the penny. "Close" is not OK. $4,231.07 ≠ $4,231.06.
• Citation correctness. Every claim must trace back to a real transaction or balance.
• No fabrication. A made-up vendor name or invented transaction is a SOX issue.
• Stability across cycles. The agent should make the same call on the same data two months later.
• Auditability of the eval itself. If you can't show external auditors how you measure agent quality, the agent is not production-grade.

Golden datasets

The foundation of any eval program: a curated, versioned set of inputs with known-correct outputs. For finance:

How to build one

1. Sample real historical cases. Take last 6 months of completed reconciliations, post-Controller-approval. These are your ground truth.
2. De-identify where required. Strip PII / vendor confidentiality where policy demands; preserve transaction structure.
3. Capture intermediate state too: not just final outputs, but tool-call sequences from prior runs.
4. Curate edge cases. The 5% of cases that broke historically are 50% of the value of the dataset.
5. Version it. Golden v1, v2, v3 — each tied to a model version and a prompt version.

Categories your golden set must include

• Happy path: clean recons with all items matching.
• Timing differences: outstanding checks, in-transit transfers.
• Foreign currency complications: FX revaluation, settlement vs. trade dates.
• Edge cases: fee accruals, intercompany breaks, on-chain confirmations vs. accounting events.
• Known-wrong cases: inputs designed to fail reconciliation. The right answer is "halt and escalate."
• Adversarial cases: malformed inputs, duplicated transactions, missing fields. The agent must not invent values.

Metrics that matter

Pick metrics that map to business outcomes. A working list for a recon agent:

The metric that separates senior from junior is false-confidence rate. Anyone can measure accuracy. Measuring "how often is the agent both wrong and confident" is what makes the system trustworthy.

LLM-as-judge — and when not to

For some outputs (variance commentary, audit narratives) the right answer is text, not a number, and exact-match doesn't apply. LLM-as-judge is the technique: a second LLM call grades the first's output against a rubric.

JUDGE_RUBRIC = """Grade the following variance commentary on three axes:
1. Citation accuracy: every numeric claim is supported by the input transactions. 0-5.
2. Driver identification: the commentary correctly identifies the top 1-3 drivers. 0-5.
3. No fabrication: no invented vendor names, dates, or amounts. PASS/FAIL.

Return JSON: {citation_accuracy, driver_identification, no_fabrication, rationale}.
"""

Where LLM-judge is useful

• Narrative quality (variance, flux, audit responses).
• Structural correctness ("did the agent follow the required output format?").
• Pre-flighting outputs in production (judge runs as a critic before the output ships).

Where LLM-judge is dangerous

• Numerical correctness. Don't use an LLM to grade a number. Compute it.
• Citation existence. Don't ask an LLM if a transaction id is valid. Look it up.
• SOX-relevant decisions. "Did the agent's proposed JE follow the controls policy?" — code the rules; don't trust an LLM to evaluate compliance.

The rule: deterministic checks for numbers and rules; LLM-judge for text quality.

Regression suites

Every prompt change, model upgrade, or tool change runs the full golden set and compares to the previous baseline. Failures block the change.

def run_regression(prompt_version, model_version, golden_set):
    results = []
    for case in golden_set:
        out = run_agent(case.inputs, prompt_version, model_version)
        results.append({
            "case_id": case.id,
            "variance_accurate": out.variance == case.expected_variance,
            "match_precision": precision(out.matches, case.true_matches),
            "match_recall": recall(out.matches, case.true_matches),
            "citation_accuracy": citation_check(out, case.universe),
            "tokens": out.tokens, "latency_ms": out.latency_ms,
        })
    return aggregate(results)

def gate(new_results, baseline):
    # Block if any metric drops more than tolerance
    return all(
        new_results[m] >= baseline[m] - tolerance[m]
        for m in CRITICAL_METRICS
    )

The CI pipeline that runs this is part of your SOX change-management story: "Every change to the production agent is gated by a regression run that exercises 500 golden cases, with results archived 7 years."

Drift detection

Inputs change over time. New transaction types appear. Counterparty mix shifts. Crypto introduces new chains. Your agent's performance can quietly degrade even if the prompt is unchanged.

• Distribution drift: track input statistics (volume, account mix, transaction sizes) and alert when current distribution diverges from the training/eval distribution.
• Outcome drift: track the rate of human overrides on agent proposals. If overrides are climbing, the agent is getting it wrong more often.
• Tool-call drift: the average sequence of tool calls per case. If the agent suddenly calls 12 tools where it used to call 5, something has shifted.

Surface these in a finance dashboard. The Process Owner should see them weekly.

Production evals — every run is a sample

Offline regression is necessary but not sufficient. In production:

• Every production run logs structured tool calls, outputs, confidence, and ultimate human decision (approved / edited / rejected).
• A weekly job samples N production runs and re-grades them with the same metrics as the golden set.
• Human overrides become candidate additions to the golden set (with PII review).

This closes the loop: the agent's failures in production feed back into the eval set that catches future regressions.

Evals as SOX change gates

The deepest reason to invest in evals: they are the substrate of your change-management story. The control narrative reads:

External auditors will walk through this control quarterly. If your eval suite is just "we tried it and it seemed fine," you fail. If you have versioned golden sets, archived regression runs, and a gate that blocks deploys, you pass.

A first eval suite in a week

If you're starting from zero, here's the cheapest viable eval program for your first agent:

1. Day 1-2: collect 50 historical reconciliations the Controller approved. Get inputs + expected outputs.
2. Day 3: write a runner that executes the agent on each case and computes variance accuracy, match precision, citation accuracy.
3. Day 4: baseline the current prompt/model. Document the numbers as v1.
4. Day 5: wire the runner into CI. Any PR that changes the prompt must run it; comments post the diff.
5. Day 6: add 10 adversarial cases (malformed inputs, fake transactions, edge currencies). Verify agent halts on each.
6. Day 7: share the dashboard with the Process Owner.

Not glamorous, but it is the artifact that turns your agent from "demo" into "production." It is also exactly the artifact the JD's section 4 (governance) requires.

The eval answer in interviews

When asked "how do you eval a finance agent," a strong four-beat answer:

1. Golden set: curated historical cases, including adversarial. Versioned.
2. Metrics: numeric correctness deterministically; narrative quality via LLM-judge; and a false-confidence metric.
3. Regression as change gate: every prompt/model change runs the suite; thresholds block deploys; runs archived for SOX.
4. Production loop: human overrides feed the next golden set; drift dashboards surface degradation early.

If you say "false confidence" out loud, an experienced interviewer will lean in. Most candidates skip it.