Error Handling & Failure Modes

Failure taxonomy

Group failures by what causes them. The mitigations are very different.

Hallucinated numbers & entries

The most dangerous failure in finance, and the easiest to hide.

What it looks like

• Agent claims "GL balance is $4,231,007.42" but never actually called the GL tool.
• Agent cites a transaction id that doesn't exist.
• Agent infers a counterparty name from context and runs with it.
• Agent computes a sum in its head (i.e., from the language model's arithmetic) rather than via deterministic code.

Defenses

1. Never accept a numeric claim that didn't come from a tool call. Validate every number in the output against the tool-call log. If a number appears in the output but wasn't returned by any tool, fail the run.
2. Citation validation. Every cited id must exist; verify against the universe before passing the output downstream.
3. Tool-do-math. Don't let the LLM compute totals or variances. Expose a sum / subtract / variance tool, or do the math in Python after the agent finishes.
4. Anti-fabrication prompt clause. "If a value is not present in the inputs, return null and the flag missing. Do not infer."

def validate_numbers_came_from_tools(agent_output: dict, tool_log: list[dict]) -> list[str]:
    """Return a list of numbers in output that did NOT come from a tool result."""
    tool_numbers = {n for call in tool_log for n in extract_numbers(call["result"])}
    output_numbers = extract_numbers(agent_output)
    return [n for n in output_numbers if n not in tool_numbers]

Double-booking & idempotency

The retry that breaks production: an agent's run failed halfway, the orchestration retried, and the same JE got posted twice.

Why this happens

• Worker crashed after tool call but before checkpoint.
• Webhook redelivery from the upstream system.
• Human approval clicked twice.
• Two concurrent runs racing on the same close task.

Defenses (defense in depth)

1. Idempotency key on every write. Deterministic from the logical operation: sha256(entity|period|account|action|hash(inputs)).
2. Downstream uniqueness. NetSuite has external-id uniqueness; use it. BlackLine similarly.
3. Workflow-level locks. A close task can be claimed by exactly one runner; second runner sees "in progress" and waits.
4. Two-phase commits where possible. Stage → confirm. Never combine.
5. Explicit replay tooling. When something needs to be redone, an operator runs a replay tool that knows about idempotency, rather than re-running the agent.

Missed entries

The silent failure: a transaction that should have been reconciled was never seen by the agent. No error fired; just a missing match.

Defenses

• Manifest checks. Pull a count + total from both sides. After matching, both sides should account for all items (matched + unmatched + reconciling). If counts don't balance, halt.
• Period completeness. Before running, confirm both sides have published their period-close data. If GL period is closed but bank statement is mid-month, halt.
• Watermarks. Track last-seen transaction id / timestamp per source. Detect gaps.

Broken integrations

NetSuite is rate-limited and reloads can stall. Fireblocks webhook deliveries can spike. Bank file feeds can be delayed.

• Health checks per integration. A periodic ping with expected response. Dashboard surfaces stale connections.
• Graceful degradation. If Lukka is unreachable, the close-orchestrator should mark crypto-recon tasks as "blocked: Lukka unavailable" and proceed with what it can do.
• Schema drift detection. If a tool gets a field it didn't expect, log and alert. Don't ignore.
• Version pinning. Pin SuiteTalk version, BlackLine API version. Upgrades are change-managed.

Partial workflow failure

A close has 80 tasks. Task 47 fails. What do you do?

• Don't roll back what's already done — most tasks aren't rollback-able cheaply (a posted JE, a sent email).
• Mark the failure on the task; surface to the Process Owner; halt anything downstream of the failed task; let upstream-complete work stand.
• Provide a resume tool that picks up from the failed step after the human fixes the underlying issue.
• Never silently skip. A skipped task in close is the silent killer.

Retry, repair, rollback — picking the right one

The pattern: retry mechanical; repair semantic; rollback in finance means reversing entries, not deleting them. Finance has no DELETE.

Fail-safe defaults

Concrete instances:

• Confidence below threshold → halt, escalate. Never "best guess."
• Multiple plausible matches → halt, do not pick.
• Period mismatch between sources → halt, do not reconcile.
• Tool returned an unexpected field → halt, do not interpret.
• Two retries failed → halt, do not try a third.

"Halt to a human" is always cheaper than "guess and recover from a SOX issue."

Circuit breakers

Past a certain failure rate, you stop letting the agent attempt anything.

• Per-agent error budget: e.g., > 5% of runs fail in an hour → open circuit, pause agent, page on-call.
• Per-integration breaker: NetSuite 429 rate > threshold → pause writes until backoff clears.
• Per-Process-Owner breaker: agent's outputs being > X% rejected by humans → flag for review and pause.

Circuit-breaker state is itself logged and dashboarded. A breaker tripping is a signal, not just an outage.

Incident response — when something does go wrong

You will have an incident. The shape of a good response:

1. Halt the agent class immediately. Not just the failing run — the type of run. (Pause the agent, not the data.)
2. Identify scope: query the audit log for every action this agent class took since last known good. Bound the exposure.
3. Notify Process Owner and (depending on materiality) SOX Lead and Controller.
4. Triage: is this a code bug, a prompt regression, a model regression, an integration drift, an input data issue?
5. Reverse if needed (reversing JEs, not deletes; with approvals).
6. Add the case to the golden set and write a regression test.
7. Post-mortem: structured, blameless, signed by the Process Owner.
8. Controlled re-enable: only after eval passes and a Controller approves.

This is exactly the discipline that SOX expects of any control. Apply it to your agents from day one and you'll never get caught off-guard by a regulator's question.

Interview-ready failure stories

Interviewers love these. Have one of each shape ready, drawn from real (or recent prototype) experience:

• A time you halted a build because it wasn't ready.
• A time an agent hallucinated and how you caught it.
• A time you found a silent failure mode (missed entry, drift, integration schema change).
• A time a retry caused a double-action and how you redesigned for idempotency.

If you don't have these stories from production, you can have them from a prototype. The story shape is identical: situation → what broke → what you saw in logs → what you changed.

✓ Check yourself

Answer from memory before opening each one.

1. An agent's reconciliation summary states a GL balance of $4,231,007.42, but no tool call returned that figure. How does the harness catch it, and why is that stronger than a line in the prompt?

   Show answer

   Extract every number from the output and compare it to the numbers in the tool-call log, then fail the run on any that no tool returned. A prompt clause asks the model to behave, while this check tests what it did. It also catches arithmetic done in the model's head. See hallucinated numbers.

2. Every write carries a deterministic idempotency key, yet two concurrent runs on the same close task both post. What was missing?

   Show answer

   A key only helps if something enforces it. Downstream uniqueness, such as an external-id constraint, must reject the second write, and a workflow lock must let exactly one runner claim the task. Layering key, uniqueness and lock means one failure does not become a double post. See double-booking.

3. A reconciliation finishes with no errors and no breaks, but an auditor later finds a deposit that was never matched. What check would have caught it?

   Show answer

   A manifest check: pull the count and total from both sides, and require matched plus unmatched plus reconciling items to account for every one. Period completeness and per-source watermarks catch the other ways an item is never seen. The failure was silent, so the check must fail loudly. See missed entries.

4. Task 47 of 80 in the close fails. What do you do, and what do you refuse to do?

   Show answer

   Mark the failure, tell the Process Owner, halt only the tasks downstream of 47, and let completed upstream work stand. Provide a resume tool that continues once the cause is fixed. Do not roll back posted entries wholesale and do not silently skip the task, since a skipped task in close is the quiet failure. See partial workflow failure.

5. A posted agent entry is wrong. A colleague wants to delete it and re-run the agent. What do you do instead?

   Show answer

   Post an approved reversing entry and then the correct one, so the ledger shows both and the trail stays intact. Finance has no delete, and removing the record would also remove the evidence of what happened. Then add the case to the golden set. See retry, repair, rollback.

6. You set a circuit breaker to pause an agent when over 5% of its runs fail in an hour. How tight should it be for a Tier 3 posting agent compared with a Tier 1 morning brief?

   Show answer

   Tighter for the posting agent. The rule is to fail in the direction of doing less, and an unneeded pause on a gated write costs a delay, while a bad run costs a SOX issue. A paused read-only brief is cheap to resume, so it can tolerate a looser setting. Log every trip, since a trip is a signal. See circuit breakers.

Takeaway

• Group failures by cause, transient, contract, semantic, idempotency, state and governance, because each class has a different defence.
• Never accept a number the tools did not return, and have code do the arithmetic, not the model.
• Stop double posts in layers: a deterministic key, downstream uniqueness, a lock on the task and a replay tool that knows about all three.
• Silent failures need positive checks: count-and-total manifests, period completeness and watermarks.
• Retry mechanical errors, repair semantic ones with a capped corrective retry, and roll back by reversing entries, never by deleting.
• When uncertain, halt and hand to a human. Every threshold should fail toward doing less.
• In an incident, halt the whole agent class, bound the exposure from the audit log, reverse with approval, add the case to the golden set and re-enable only after evals pass.

The safeguards here only count if they are provable, so the next chapter covers governance, audit trails and evidence. → Governance & Audit

Further reading

• Errors — Anthropic docs, accessed 2026. Error types and which are safe to retry.
• Rate limits — Anthropic docs, accessed 2026. How limits are measured and how to back off.
• Timeouts, retries and backoff with jitter — Amazon Builders' Library. Why retries need jitter and idempotent operations.
• Idempotent requests — Stripe API docs. A reference design for idempotency keys.
• Circuit Breaker — Martin Fowler. The pattern, its states and what to monitor.
• Postmortem Culture: Learning from Failure — Google SRE book, 2016. Blameless post-mortems.