Error Handling & Failure Modes

Failure taxonomy

Group failures by what causes them. The mitigations are very different.

Hallucinated numbers & entries

The most dangerous failure in finance, and the easiest to hide.

What it looks like

• Agent claims "GL balance is $4,231,007.42" but never actually called the GL tool.
• Agent cites a transaction id that doesn't exist.
• Agent infers a counterparty name from context and runs with it.
• Agent computes a sum in its head (i.e., from the language model's arithmetic) rather than via deterministic code.

Defenses

1. Never accept a numeric claim that didn't come from a tool call. Validate every number in the output against the tool-call log. If a number appears in the output but wasn't returned by any tool, fail the run.
2. Citation validation. Every cited id must exist; verify against the universe before passing the output downstream.
3. Tool-do-math. Don't let the LLM compute totals or variances. Expose a sum / subtract / variance tool, or do the math in Python after the agent finishes.
4. Anti-fabrication prompt clause. "If a value is not present in the inputs, return null and the flag missing. Do not infer."

def validate_numbers_came_from_tools(agent_output: dict, tool_log: list[dict]) -> list[str]:
    """Return a list of numbers in output that did NOT come from a tool result."""
    tool_numbers = {n for call in tool_log for n in extract_numbers(call["result"])}
    output_numbers = extract_numbers(agent_output)
    return [n for n in output_numbers if n not in tool_numbers]

Double-booking & idempotency

The retry that breaks production: an agent's run failed halfway, the orchestration retried, and the same JE got posted twice.

Why this happens

• Worker crashed after tool call but before checkpoint.
• Webhook redelivery from the upstream system.
• Human approval clicked twice.
• Two concurrent runs racing on the same close task.

Defenses (defense in depth)

1. Idempotency key on every write. Deterministic from the logical operation: sha256(entity|period|account|action|hash(inputs)).
2. Downstream uniqueness. NetSuite has external-id uniqueness; use it. BlackLine similarly.
3. Workflow-level locks. A close task can be claimed by exactly one runner; second runner sees "in progress" and waits.
4. Two-phase commits where possible. Stage → confirm. Never combine.
5. Explicit replay tooling. When something needs to be redone, an operator runs a replay tool that knows about idempotency, rather than re-running the agent.

Missed entries

The silent failure: a transaction that should have been reconciled was never seen by the agent. No error fired; just a missing match.

Defenses

• Manifest checks. Pull a count + total from both sides. After matching, both sides should account for all items (matched + unmatched + reconciling). If counts don't balance, halt.
• Period completeness. Before running, confirm both sides have published their period-close data. If GL period is closed but bank statement is mid-month, halt.
• Watermarks. Track last-seen transaction id / timestamp per source. Detect gaps.

Broken integrations

NetSuite is rate-limited and reloads can stall. Fireblocks webhook deliveries can spike. Bank file feeds can be delayed.

• Health checks per integration. A periodic ping with expected response. Dashboard surfaces stale connections.
• Graceful degradation. If Lukka is unreachable, the close-orchestrator should mark crypto-recon tasks as "blocked: Lukka unavailable" and proceed with what it can do.
• Schema drift detection. If a tool gets a field it didn't expect, log and alert. Don't ignore.
• Version pinning. Pin SuiteTalk version, BlackLine API version. Upgrades are change-managed.

Partial workflow failure

A close has 80 tasks. Task 47 fails. What do you do?

• Don't roll back what's already done — most tasks aren't rollback-able cheaply (a posted JE, a sent email).
• Mark the failure on the task; surface to the Process Owner; halt anything downstream of the failed task; let upstream-complete work stand.
• Provide a resume tool that picks up from the failed step after the human fixes the underlying issue.
• Never silently skip. A skipped task in close is the silent killer.

Retry, repair, rollback — picking the right one

The pattern: retry mechanical; repair semantic; rollback in finance means reversing entries, not deleting them. Finance has no DELETE.

Fail-safe defaults

Concrete instances:

• Confidence below threshold → halt, escalate. Never "best guess."
• Multiple plausible matches → halt, do not pick.
• Period mismatch between sources → halt, do not reconcile.
• Tool returned an unexpected field → halt, do not interpret.
• Two retries failed → halt, do not try a third.

"Halt to a human" is always cheaper than "guess and recover from a SOX issue."

Circuit breakers

Past a certain failure rate, you stop letting the agent attempt anything.

• Per-agent error budget: e.g., > 5% of runs fail in an hour → open circuit, pause agent, page on-call.
• Per-integration breaker: NetSuite 429 rate > threshold → pause writes until backoff clears.
• Per-Process-Owner breaker: agent's outputs being > X% rejected by humans → flag for review and pause.

Circuit-breaker state is itself logged and dashboarded. A breaker tripping is a signal, not just an outage.

Incident response — when something does go wrong

You will have an incident. The shape of a good response:

1. Halt the agent class immediately. Not just the failing run — the type of run. (Pause the agent, not the data.)
2. Identify scope: query the audit log for every action this agent class took since last known good. Bound the exposure.
3. Notify Process Owner and (depending on materiality) SOX Lead and Controller.
4. Triage: is this a code bug, a prompt regression, a model regression, an integration drift, an input data issue?
5. Reverse if needed (reversing JEs, not deletes; with approvals).
6. Add the case to the golden set and write a regression test.
7. Post-mortem: structured, blameless, signed by the Process Owner.
8. Controlled re-enable: only after eval passes and a Controller approves.

This is exactly the discipline that SOX expects of any control. Apply it to your agents from day one and you'll never get caught off-guard by a regulator's question.

Interview-ready failure stories

Interviewers love these. Have one of each shape ready, drawn from real (or recent prototype) experience:

• A time you halted a build because it wasn't ready.
• A time an agent hallucinated and how you caught it.
• A time you found a silent failure mode (missed entry, drift, integration schema change).
• A time a retry caused a double-action and how you redesigned for idempotency.

If you don't have these stories from production, you can have them from a prototype. The story shape is identical: situation → what broke → what you saw in logs → what you changed.