Core Fundamentals

LLM orchestration basics

A modern LLM is a stateless function: input tokens → output tokens. Everything else — memory, tools, loops, multi-agent coordination — is orchestration you build around that function. The vocabulary:

• Inference call: one round-trip to the model.
• Turn: one user message + one assistant response.
• Conversation / thread: a sequence of turns sharing context.
• System prompt: the instructions that frame every call.
• Tools / functions: declared callables the model can request.
• Agent loop: an automated loop where the model decides actions and the harness executes them until a stop condition.

In your interview, when asked "how does an agent work," answer at this level first, then add concrete pieces. Don't dive into MCP transports before establishing the loop.

Agentic system primitives

An agent is a loop with five primitives:

1. Goal — what the agent is trying to accomplish ("reconcile the May Coinbase Custody balance to the GL").
2. Context — system prompt, knowledge, conversation history, tool definitions.
3. Tools — the verbs the agent can take (fetch balance, query GL, post comment).
4. Loop — model proposes an action → harness executes → result fed back → repeat.
5. Stop condition — task complete, max steps reached, error budget exceeded, human gate hit.

The most common finance failure mode is a missing or weak stop condition. An agent that's "trying hard" to reconcile a balance that's structurally unreconcilable will burn tokens and hallucinate matches. Always design the bail-out gate first.

Prompts in production

Three layers in production prompts:

What good system prompts contain in finance

• Identity & scope: "You are a reconciliation assistant. You do not post journal entries. You produce proposed entries that a human approver reviews."
• Bail conditions: "If confidence < 0.85, or if the variance exceeds $X, halt and flag for human."
• Output schema: exact JSON shape the downstream system expects.
• Citations requirement: every numerical claim must reference a source document/system.
• Anti-fabrication rule: "If a value is not present in the inputs, do not invent it. Return null and a flag."

Tool use & function calling

Tools are how the model takes actions. You declare a tool by name, description, and JSON schema; the model decides to call it; your code executes; you return the result in the conversation. A minimal example for finance:

tools = [
    {
        "name": "fetch_gl_balance",
        "description": "Fetch the General Ledger balance for an account on a date.",
        "input_schema": {
            "type": "object",
            "properties": {
                "account_code": {"type": "string", "description": "e.g. 1010-Cash-USD"},
                "as_of_date": {"type": "string", "description": "ISO date YYYY-MM-DD"},
                "entity_id": {"type": "string"}
            },
            "required": ["account_code", "as_of_date", "entity_id"]
        }
    },
    {
        "name": "fetch_bank_balance",
        "description": "Fetch balance from a banking API for an account on a date.",
        "input_schema": {
            "type": "object",
            "properties": {
                "bank_account_id": {"type": "string"},
                "as_of_date": {"type": "string"}
            },
            "required": ["bank_account_id", "as_of_date"]
        }
    },
    {
        "name": "propose_reconciling_item",
        "description": "Propose a reconciling item for human approval. Does not post.",
        "input_schema": {
            "type": "object",
            "properties": {
                "amount": {"type": "string", "description": "Decimal as string, never float"},
                "description": {"type": "string"},
                "evidence": {"type": "array", "items": {"type": "string"}}
            },
            "required": ["amount", "description", "evidence"]
        }
    }
]

Notice: there is no "post to NetSuite" tool exposed to the agent at this tier. Posting is a separate human-gated step. The tool catalog encodes the risk tier.

Tool design rules for finance

• Read tools never mutate. If a tool's name doesn't start with fetch_ / get_ / list_, it should require human approval.
• Write tools write to staging — propose-and-approve, not commit.
• Money is decimal in every schema. Never floats.
• Every tool call is logged with inputs, outputs, latency, model version, prompt hash, caller identity.

Structured output

For any finance step that hands off to a downstream system, demand a structured output and validate it before acting. Two patterns:

1. JSON schema in the prompt + post-call validation with Pydantic.
2. Tool-call as output: ask the model to "call" a tool whose only purpose is to emit the structured answer.

from decimal import Decimal
from pydantic import BaseModel, field_validator
from typing import Literal

class ReconciliationResult(BaseModel):
    status: Literal["matched", "partial", "unmatched"]
    gl_balance: Decimal
    bank_balance: Decimal
    variance: Decimal
    confidence: float  # 0..1
    proposed_items: list[dict] = []
    needs_human: bool

    @field_validator("confidence")
    @classmethod
    def in_range(cls, v):
        if not 0.0 <= v <= 1.0:
            raise ValueError("confidence must be in [0,1]")
        return v

If validation fails, you do not pass the output to the next step. You retry with a corrective prompt or escalate.

Prompt caching

Anthropic's prompt caching gives ~90% discount and ~85% latency reduction on cached tokens. In finance the wins are large because most calls reuse the same preamble:

• Chart of accounts (~5K-50K tokens depending on entity).
• Controls catalog & policy.
• Journal posting rules.
• Recent N periods of trial balance (for comparatives).

Pattern: put the long stable preamble in a cached block; put the per-call inputs after the cache breakpoint. A monthly close that calls Claude 5,000 times can cost 1/8 of an uncached design.

Retries & idempotency

Two classes of failure to design for:

1. Transient — API timeout, 429 rate limit, transient 5xx. Retry with exponential backoff and jitter.
2. Semantic — model produced output that fails validation. Retry with a corrective system message ("your last output failed validation because X — fix and resubmit"). Cap at 2-3 attempts then escalate.

Idempotency keys are non-negotiable on any tool that mutates state. The pattern:

import hashlib

def idempotency_key(entity_id: str, period: str, account: str, action: str) -> str:
    # Deterministic, stable across retries of the same logical operation.
    raw = f"{entity_id}|{period}|{account}|{action}"
    return hashlib.sha256(raw.encode()).hexdigest()

The downstream system (NetSuite, BlackLine) must reject duplicate idempotency keys. Without this, an agent retry can double-post a journal entry — and at close, doubled JEs are the kind of bug that ends careers.

Latency vs. cost tradeoffs

You'll be asked some version of "when do you use Opus vs. Sonnet vs. Haiku?" An interview-ready framing:

The shape: pick the cheapest model that meets the eval bar. Don't reach for Opus by default. Don't reach for Haiku to save pennies on a tier-3 control.

Context engineering

A long context window is not the same as useful context. Two principles:

• Recency & relevance bias. Models attend better to information at the start and end of context. Put the most important policy at the top of the system prompt; put the task at the bottom of the user turn.
• Retrieve, don't dump. 1M-token contexts let you stuff everything in, but retrieval (RAG over the chart of accounts, prior recons, policies) usually produces better answers at a fraction of the cost.

For close, a good default is: cache the long preamble + retrieve the 5-10 most relevant prior recons + include only the current period's transactions in the per-call input.

The finance overlay on every primitive

If you take one thing from this chapter into your interview, take this. Every fundamental above has a finance-specific overlay:

That last row is subtle and high-signal in interviews. Swapping Sonnet for Sonnet-newer in a Tier 3 workflow is a SOX change. You don't just deploy it on Tuesday because the new model came out. That sentence alone tells a senior interviewer you've thought about regulated AI for real.

✓ Check yourself

Answer from memory before opening each one.

1. A reconciliation agent is asked to tie out a custody balance that is structurally off because of a known timing gap. It runs for 40 steps and proposes three matches that do not exist. What was missing, and what do you build before anything else?

   Show answer

   The stop condition was missing or too weak. An agent trying hard to reach a target it cannot reach will burn tokens and invent matches. Define the bail-out first: a step cap, an error budget, a variance or confidence threshold, and a human gate for high-risk classes. See agentic primitives.

2. Each reconciliation call carries about 30,000 stable tokens of chart of accounts and policy plus 2,000 tokens of task input. Roughly what share of the uncached input cost do you pay once caching works, and what must be true of the prompt for it to work?

   Show answer

   Cache reads cost about a tenth of normal input price, so you pay roughly (3,000 + 2,000) / 32,000, about a sixth. It only works if the stable block comes first and is identical on every call, with the per-call input after the cache breakpoint, and if calls arrive while the cache is still live. Anything that changes early in the prompt, such as a timestamp, defeats it. See prompt caching.

3. The model returns a reconciliation result whose confidence field is 1.4 and whose variance is a string with a currency symbol. What happens next, and what do you not do?

   Show answer

   The output fails validation, so it never reaches the next step. You retry with a corrective message that says what failed, cap the attempts at two or three, then escalate to a human. You do not coerce the values, pass them on, or retry forever. See structured output and retries.

4. A staging call times out and the harness retries it. Why must the idempotency key be derived from the logical operation rather than generated fresh for each attempt?

   Show answer

   A fresh key per attempt looks like a new request to the downstream system, so if the first attempt actually landed, the retry posts a second entry. A key built from entity, period, account and action stays the same across retries, so the downstream system can reject the duplicate. Never retry a write silently without one. See retries and idempotency.

5. You must categorise 500,000 transactions overnight and also write the quarter-end audit narrative. Which model tier do you pick for each, and what rule decides?

   Show answer

   Use the smallest tier for the high-volume classification and the most capable tier for the few high-stakes narratives. The rule is to pick the cheapest model that clears the eval bar for that task, so you need an eval first. Defaulting to the largest model wastes money on bulk work, and defaulting to the smallest on a Tier 3 control saves pennies at real risk. See latency versus cost.

6. A newer, better Sonnet is released and a colleague wants to switch the Tier 3 posting workflow on Tuesday. What do you say?

   Show answer

   The model is part of the control as documented, so switching it is a change-managed event. Run the new model against the eval set, compare results, update the control narrative, and get the Process Owner's approval before deploying. Capability gains do not waive change control. See the finance overlay.

Takeaway

• An agent is a goal, context, tools, a loop and a stop condition. Design the stop condition first, because a missing one is the commonest failure.
• Production prompts have three layers, system, knowledge and task. The first two are stable and cached, and the task layer changes on every call.
• The tool catalog encodes the risk tier: read tools never mutate, write tools go to staging, and money is always a decimal string, never a float.
• Validate structured output before acting. Retry semantic failures with a corrective message, cap the attempts, then escalate.
• Retry transient failures with backoff and jitter, and give every mutating call a deterministic idempotency key so a retry cannot double-post.
• Pick the cheapest model that clears the eval bar, and treat any model swap on a gated workflow as a change-managed event.
• A large context window is not a reason to dump everything in. Cache the stable preamble and retrieve the few relevant prior examples.

With the building blocks in place, the next chapter turns them into whole workflows: patterns, MCP and the close on a whiteboard. → Agentic Workflow Architecture

Further reading

• Tool use with Claude — Anthropic docs, accessed 2026. How tools are defined, called and returned, and the loop around them.
• Prompt caching — Anthropic docs, accessed 2026. Breakpoints, pricing for reads and writes, and what invalidates a cache.
• Structured outputs — Anthropic docs, accessed 2026. Getting schema-valid JSON from the model.
• Effective context engineering for AI agents — Anthropic, 2025. Why more context is not better context, and how to curate it.
• decimal: Decimal fixed point and floating point arithmetic — Python documentation. Why money belongs in Decimal and how rounding contexts work.
• Timeouts, retries and backoff with jitter — Amazon Builders' Library. The reasoning behind jittered backoff and why retries need idempotent operations.
• Idempotent requests — Stripe API docs. A widely copied design for idempotency keys on writes.