Deployment & Ops

Environments & promotion

At minimum, three environments. SOX-mature shops have four.

"Move fast, break things" doesn't work here. The promotion gates are the SOX controls. Bypassing them is a control deficiency.

SOX change control — what an auditor wants to see

For any change to a Tier 2/3 production agent, the audit trail includes:

• Change request: ticket / PR with description, risk assessment, rollback plan.
• Design impact: does the design doc need amendment? Yes if scope, tier, or controls change.
• Test evidence: regression suite passed at thresholds; archived artifact.
• Process Owner approval: signed in the ticket.
• Engineering approval: PR approved by code owner.
• SOX Lead approval: required for tier change, model change, or control redesign.
• Deployment record: who deployed, when, to which env, which version.
• Post-deploy verification: smoke test passed; first prod run reviewed.

None of this is unique to AI. It's the same change-control discipline auditors already expect of every SOX-relevant production system. The mistake is treating AI as exempt because it's "just a prompt."

CI/CD with controls

A pipeline that meets SOX expectations:

1. Pre-commit: lint, type check, secret scan.
2. PR build: unit tests, schema tests, fast eval suite (subset of golden set).
3. Merge to main: full eval suite; threshold gate; deploys to staging on green.
4. UAT promotion: requires named approver in CI; runs a smoke test in UAT.
5. Prod promotion: two-key approval (engineering + Process Owner); deploy job tags the version; first prod run goes to a shadow log before active use.

What's special about the AI version:

• Prompt changes are code changes (versioned, reviewed, regression-gated).
• Model version pin is a code field; bumping it is a PR.
• Tool catalog and MCP server versions are pinned; upgrades are coordinated releases.
• The eval baseline is itself in the repo, versioned, signed off.

Secrets & credentials

• Secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) — never env vars in repo or n8n.
• Short-lived tokens where the API supports it.
• Per-environment credentials — staging never touches prod NetSuite.
• Least privilege — the bank-recon agent's credentials read GL and bank balances; they don't write to NetSuite.
• Audit trail on credential use — every API call's caller identity logged.
• Rotation on schedule; emergency rotation playbook tested.

Observability — what to instrument

Every agent run is observable on three layers:

Trace IDs matter: every agent run gets a run_id propagated to every tool call, MCP request, and downstream API write. When something goes wrong, you can stitch the full timeline from one id.

Alerts that page on-call

• Agent error rate > threshold.
• Override rate climbing for a tier 2/3 agent.
• Integration health check fails (any of NetSuite, BlackLine, Kyriba, Fireblocks, Lukka, bank feed).
• Audit-log writes failing (the most important one — if you can't log, you can't operate).
• Cost burn rate exceeds budget.

Alerts that just email

• Eval drift detected on weekly production sample.
• New unknown field in tool response.
• Slow query in audit log.

Rollback strategy

Rollback in finance is rarely "revert the deploy." It's:

1. Stop the agent. Pause processing, drain the queue.
2. Assess exposure: which entries already posted? Query audit log by version.
3. Reverse what needs reversing: reversing JEs with Controller approval. Document the reversal as a SOX control event.
4. Revert the deploy: deploy the previous version pin (code + prompts + model + tool catalog all pinned together).
5. Restart agent only after Process Owner approves.
6. Post-mortem: structured, blameless, signed.

The deploy artifact must include all the dimensions that can shift: code commit, prompt version, model id + pin, tool catalog hash, MCP server versions. Rollback means restoring all of them.

Cost & budget controls

LLM costs can run away. Controls:

• Per-agent budget: hard cap on $/day. Exceeds → circuit breaker.
• Prompt caching: the single largest cost lever in finance — chart of accounts, controls catalog, prior recons reused per call. 70-90% reduction realistic.
• Model selection: Haiku for high-volume classification; Sonnet for default; Opus only where the eval bar requires it.
• Token-per-call ceiling: alert if a call exceeds 80th-percentile recent usage.
• Batching where possible (Anthropic batch API for non-realtime).
• Dashboard showing cost per workflow, per tier, per Process Owner — so spend is attributable.

Vendor management

The JD calls out vendor relationships in the Nice-to-Have list. The vendors that matter:

For each: SOC 2 / ISO 27001 reports reviewed annually; DPA in place; named operational contact; outage history reviewed; renewal calendar.

Runbooks & on-call

The discipline that takes "ships agents" to "operates a platform." For each production agent class:

• Runbook: what each alert means, how to diagnose, how to escalate.
• Pause / kill switch: documented, tested, accessible to the Process Owner without engineering involvement.
• On-call rotation: who's responsible for AI-platform incidents.
• Escalation tree: AI Architect → Engineering Lead → CTO; Process Owner → Controller → CFO.
• Quarterly DR drill: pretend a top integration is down for 4 hours; show you can complete close manually with the audit-log story intact.

The ops dashboard

One pane the AI Architect and the Controller both look at. The metrics that matter:

• Per agent: runs/day, success/halt/escalate split, override rate, eval pass rate (weekly sample), p95 latency, cost.
• Per workflow: end-to-end SLA, FTE-equivalent saved, items processed.
• Integration health: each external API's freshness, error rate, rate-limit headroom.
• Audit log health: write latency, missing-row alerts (every run must have a row).
• SOX status: which agents have current design doc signoff; which are out of date.

This dashboard is the artifact that supports the JD's fifth responsibility area: "Build metrics, dashboards, and performance frameworks to measure efficiency gains, cost reduction, risk outcomes, and ROI."

✓ Check yourself

Answer from memory before opening each one.

1. A Tier 3 prompt fix is ready on Friday, and an engineer wants to edit it in production because the gates take a day. What would an auditor conclude, and what do you offer instead?

   Show answer

   The promotion gates are the SOX controls, so bypassing them is a control deficiency, and an unrecorded production edit leaves no evidence. Offer an expedited run of the same gates, with the regression result, the Process Owner's approval and a deployment record, instead of a shortcut around them. See change control.

2. Version 14 of an agent posted 30 wrong entries before it was caught. In what order do you act, and what must the deploy artifact contain for the rollback to work?

   Show answer

   Stop the agent, assess exposure from the audit log by version, reverse the affected entries with Controller approval, then revert the deploy and restart only with the Process Owner's approval. The artifact must pin code, prompt, model and version, tool catalog hash and MCP server versions, since any of them can shift behaviour. See rollback strategy.

3. A developer reuses the posting agent's NetSuite credentials for a new read-only agent, to save a day. Why is that wrong?

   Show answer

   The new agent would inherit write access it does not need, so a bug or a prompt injection could reach production. Give each agent its own least-privilege credentials, scoped per environment, from a secrets manager, with caller identity logged on every call. See secrets and credentials.

4. Of all the alerts that page on-call, which matters most, and why?

   Show answer

   Audit-log write failures. If actions cannot be logged, you cannot show a control operated, so the agent should not keep acting. Other alerts describe a bad run, while this one removes the evidence for every run. See observability.

5. The model bill is three times the forecast in the middle of a close. How do you diagnose it, and which levers do you pull?

   Show answer

   Use the cost-per-workflow dashboard to find the source, then check the cache hit rate first, because caching is the largest lever and a changed prefix silently breaks it. Then review model selection, per-call token ceilings and batching for non-realtime work, and rely on the per-agent budget cap as the backstop. See cost and budget controls.

6. The vendor announces that the model your Tier 3 agent pins will be retired in several months. What do you do?

   Show answer

   Treat it as a planned change: run the replacement through the full regression suite on the golden set, review the cost and latency shift, update the design and control narrative, get approvals, then promote through the environments with time to spare. The deadline is a reason to start early, not to skip the gates. See vendor management.

Takeaway

• The path from dev through staging and UAT to prod is the SOX control, and bypassing a gate is a deficiency.
• A change record holds the request, design impact, test evidence, owner and engineering approvals, SOX lead approval for tier, model or control changes, a deployment record and post-deploy checks.
• Prompts, model pins, tool catalogs, MCP versions and the eval baseline are versioned and reviewed like code, with regression gates on each change.
• Credentials live in a secrets manager, are scoped per environment and per agent with least privilege, and every use is attributed.
• Trace every run with one identifier across application, agent and business signals, and page first on audit-log write failures.
• Rollback means stopping the agent, assessing exposure, reversing with approval, restoring all pinned dimensions together and restarting on the owner's say.
• Control cost with caching, model selection, per-agent budgets and attributable dashboards, and operate with runbooks, an owner-accessible kill switch, vendor reviews and a disaster drill.

Operating safely also means knowing the business the agents serve, so the next chapter covers finance and crypto domain context. → Domain Context

Further reading

• Model deprecations — Anthropic docs, accessed 2026. Retirement timelines and how to plan migrations.
• Prompt caching — Anthropic docs, accessed 2026. The biggest cost lever and what invalidates it.
• Batch processing — Anthropic docs, accessed 2026. Discounted asynchronous runs for work that is not realtime.
• Traces — OpenTelemetry documentation. Propagating one trace across services.
• Monitoring Distributed Systems — Google SRE book, 2016. What to page on and what to leave as a ticket.
• SP 800-218: Secure Software Development Framework — NIST, 2022. Practices for secure builds, reviews and releases.
• Hosting n8n — n8n docs, accessed 2026. Self-hosting, configuration and scaling.