Core Fundamentals

LLM orchestration basics

A modern LLM is a stateless function: input tokens → output tokens. Everything else — memory, tools, loops, multi-agent coordination — is orchestration you build around that function. The vocabulary:

• Inference call: one round-trip to the model.
• Turn: one user message + one assistant response.
• Conversation / thread: a sequence of turns sharing context.
• System prompt: the instructions that frame every call.
• Tools / functions: declared callables the model can request.
• Agent loop: an automated loop where the model decides actions and the harness executes them until a stop condition.

In your interview, when asked "how does an agent work," answer at this level first, then add concrete pieces. Don't dive into MCP transports before establishing the loop.

Agentic system primitives

An agent is a loop with five primitives:

1. Goal — what the agent is trying to accomplish ("reconcile the May Coinbase Custody balance to the GL").
2. Context — system prompt, knowledge, conversation history, tool definitions.
3. Tools — the verbs the agent can take (fetch balance, query GL, post comment).
4. Loop — model proposes an action → harness executes → result fed back → repeat.
5. Stop condition — task complete, max steps reached, error budget exceeded, human gate hit.

The most common finance failure mode is a missing or weak stop condition. An agent that's "trying hard" to reconcile a balance that's structurally unreconcilable will burn tokens and hallucinate matches. Always design the bail-out gate first.

Prompts in production

Three layers in production prompts:

What good system prompts contain in finance

• Identity & scope: "You are a reconciliation assistant. You do not post journal entries. You produce proposed entries that a human approver reviews."
• Bail conditions: "If confidence < 0.85, or if the variance exceeds $X, halt and flag for human."
• Output schema: exact JSON shape the downstream system expects.
• Citations requirement: every numerical claim must reference a source document/system.
• Anti-fabrication rule: "If a value is not present in the inputs, do not invent it. Return null and a flag."

Tool use & function calling

Tools are how the model takes actions. You declare a tool by name, description, and JSON schema; the model decides to call it; your code executes; you return the result in the conversation. A minimal example for finance:

tools = [
    {
        "name": "fetch_gl_balance",
        "description": "Fetch the General Ledger balance for an account on a date.",
        "input_schema": {
            "type": "object",
            "properties": {
                "account_code": {"type": "string", "description": "e.g. 1010-Cash-USD"},
                "as_of_date": {"type": "string", "description": "ISO date YYYY-MM-DD"},
                "entity_id": {"type": "string"}
            },
            "required": ["account_code", "as_of_date", "entity_id"]
        }
    },
    {
        "name": "fetch_bank_balance",
        "description": "Fetch balance from a banking API for an account on a date.",
        "input_schema": {
            "type": "object",
            "properties": {
                "bank_account_id": {"type": "string"},
                "as_of_date": {"type": "string"}
            },
            "required": ["bank_account_id", "as_of_date"]
        }
    },
    {
        "name": "propose_reconciling_item",
        "description": "Propose a reconciling item for human approval. Does not post.",
        "input_schema": {
            "type": "object",
            "properties": {
                "amount": {"type": "string", "description": "Decimal as string, never float"},
                "description": {"type": "string"},
                "evidence": {"type": "array", "items": {"type": "string"}}
            },
            "required": ["amount", "description", "evidence"]
        }
    }
]

Notice: there is no "post to NetSuite" tool exposed to the agent at this tier. Posting is a separate human-gated step. The tool catalog encodes the risk tier.

Tool design rules for finance

• Read tools never mutate. If a tool's name doesn't start with fetch_ / get_ / list_, it should require human approval.
• Write tools write to staging — propose-and-approve, not commit.
• Money is decimal in every schema. Never floats.
• Every tool call is logged with inputs, outputs, latency, model version, prompt hash, caller identity.

Structured output

For any finance step that hands off to a downstream system, demand a structured output and validate it before acting. Two patterns:

1. JSON schema in the prompt + post-call validation with Pydantic.
2. Tool-call as output: ask the model to "call" a tool whose only purpose is to emit the structured answer.

from decimal import Decimal
from pydantic import BaseModel, field_validator
from typing import Literal

class ReconciliationResult(BaseModel):
    status: Literal["matched", "partial", "unmatched"]
    gl_balance: Decimal
    bank_balance: Decimal
    variance: Decimal
    confidence: float  # 0..1
    proposed_items: list[dict] = []
    needs_human: bool

    @field_validator("confidence")
    @classmethod
    def in_range(cls, v):
        if not 0.0 <= v <= 1.0:
            raise ValueError("confidence must be in [0,1]")
        return v

If validation fails, you do not pass the output to the next step. You retry with a corrective prompt or escalate.

Prompt caching

Anthropic's prompt caching gives ~90% discount and ~85% latency reduction on cached tokens. In finance the wins are large because most calls reuse the same preamble:

• Chart of accounts (~5K-50K tokens depending on entity).
• Controls catalog & policy.
• Journal posting rules.
• Recent N periods of trial balance (for comparatives).

Pattern: put the long stable preamble in a cached block; put the per-call inputs after the cache breakpoint. A monthly close that calls Claude 5,000 times can cost 1/8 of an uncached design.

Retries & idempotency

Two classes of failure to design for:

1. Transient — API timeout, 429 rate limit, transient 5xx. Retry with exponential backoff and jitter.
2. Semantic — model produced output that fails validation. Retry with a corrective system message ("your last output failed validation because X — fix and resubmit"). Cap at 2-3 attempts then escalate.

Idempotency keys are non-negotiable on any tool that mutates state. The pattern:

import hashlib

def idempotency_key(entity_id: str, period: str, account: str, action: str) -> str:
    # Deterministic, stable across retries of the same logical operation.
    raw = f"{entity_id}|{period}|{account}|{action}"
    return hashlib.sha256(raw.encode()).hexdigest()

The downstream system (NetSuite, BlackLine) must reject duplicate idempotency keys. Without this, an agent retry can double-post a journal entry — and at close, doubled JEs are the kind of bug that ends careers.

Latency vs. cost tradeoffs

You'll be asked some version of "when do you use Opus vs. Sonnet vs. Haiku?" An interview-ready framing:

The shape: pick the cheapest model that meets the eval bar. Don't reach for Opus by default. Don't reach for Haiku to save pennies on a tier-3 control.

Context engineering

A long context window is not the same as useful context. Two principles:

• Recency & relevance bias. Models attend better to information at the start and end of context. Put the most important policy at the top of the system prompt; put the task at the bottom of the user turn.
• Retrieve, don't dump. 1M-token contexts let you stuff everything in, but retrieval (RAG over the chart of accounts, prior recons, policies) usually produces better answers at a fraction of the cost.

For close, a good default is: cache the long preamble + retrieve the 5-10 most relevant prior recons + include only the current period's transactions in the per-call input.

The finance overlay on every primitive

If you take one thing from this chapter into your interview, take this. Every fundamental above has a finance-specific overlay:

That last row is subtle and high-signal in interviews. Swapping Sonnet for Sonnet-newer in a Tier 3 workflow is a SOX change. You don't just deploy it on Tuesday because the new model came out. That sentence alone tells a senior interviewer you've thought about regulated AI for real.