Deployment & Ops

Environments & promotion

At minimum, three environments. SOX-mature shops have four.

"Move fast, break things" doesn't work here. The promotion gates are the SOX controls. Bypassing them is a control deficiency.

SOX change control — what an auditor wants to see

For any change to a Tier 2/3 production agent, the audit trail includes:

• Change request: ticket / PR with description, risk assessment, rollback plan.
• Design impact: does the design doc need amendment? Yes if scope, tier, or controls change.
• Test evidence: regression suite passed at thresholds; archived artifact.
• Process Owner approval: signed in the ticket.
• Engineering approval: PR approved by code owner.
• SOX Lead approval: required for tier change, model change, or control redesign.
• Deployment record: who deployed, when, to which env, which version.
• Post-deploy verification: smoke test passed; first prod run reviewed.

None of this is unique to AI. It's the same change-control discipline auditors already expect of every SOX-relevant production system. The mistake is treating AI as exempt because it's "just a prompt."

CI/CD with controls

A pipeline that meets SOX expectations:

1. Pre-commit: lint, type check, secret scan.
2. PR build: unit tests, schema tests, fast eval suite (subset of golden set).
3. Merge to main: full eval suite; threshold gate; deploys to staging on green.
4. UAT promotion: requires named approver in CI; runs a smoke test in UAT.
5. Prod promotion: two-key approval (engineering + Process Owner); deploy job tags the version; first prod run goes to a shadow log before active use.

What's special about the AI version:

• Prompt changes are code changes (versioned, reviewed, regression-gated).
• Model version pin is a code field; bumping it is a PR.
• Tool catalog and MCP server versions are pinned; upgrades are coordinated releases.
• The eval baseline is itself in the repo, versioned, signed off.

Secrets & credentials

• Secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) — never env vars in repo or n8n.
• Short-lived tokens where the API supports it.
• Per-environment credentials — staging never touches prod NetSuite.
• Least privilege — the bank-recon agent's credentials read GL and bank balances; they don't write to NetSuite.
• Audit trail on credential use — every API call's caller identity logged.
• Rotation on schedule; emergency rotation playbook tested.

Observability — what to instrument

Every agent run is observable on three layers:

Trace IDs matter: every agent run gets a run_id propagated to every tool call, MCP request, and downstream API write. When something goes wrong, you can stitch the full timeline from one id.

Alerts that page on-call

• Agent error rate > threshold.
• Override rate climbing for a tier 2/3 agent.
• Integration health check fails (any of NetSuite, BlackLine, Kyriba, Fireblocks, Lukka, bank feed).
• Audit-log writes failing (the most important one — if you can't log, you can't operate).
• Cost burn rate exceeds budget.

Alerts that just email

• Eval drift detected on weekly production sample.
• New unknown field in tool response.
• Slow query in audit log.

Rollback strategy

Rollback in finance is rarely "revert the deploy." It's:

1. Stop the agent. Pause processing, drain the queue.
2. Assess exposure: which entries already posted? Query audit log by version.
3. Reverse what needs reversing: reversing JEs with Controller approval. Document the reversal as a SOX control event.
4. Revert the deploy: deploy the previous version pin (code + prompts + model + tool catalog all pinned together).
5. Restart agent only after Process Owner approves.
6. Post-mortem: structured, blameless, signed.

The deploy artifact must include all the dimensions that can shift: code commit, prompt version, model id + pin, tool catalog hash, MCP server versions. Rollback means restoring all of them.

Cost & budget controls

LLM costs can run away. Controls:

• Per-agent budget: hard cap on $/day. Exceeds → circuit breaker.
• Prompt caching: the single largest cost lever in finance — chart of accounts, controls catalog, prior recons reused per call. 70-90% reduction realistic.
• Model selection: Haiku for high-volume classification; Sonnet for default; Opus only where the eval bar requires it.
• Token-per-call ceiling: alert if a call exceeds 80th-percentile recent usage.
• Batching where possible (Anthropic batch API for non-realtime).
• Dashboard showing cost per workflow, per tier, per Process Owner — so spend is attributable.

Vendor management

The JD calls out vendor relationships in the Nice-to-Have list. The vendors that matter:

For each: SOC 2 / ISO 27001 reports reviewed annually; DPA in place; named operational contact; outage history reviewed; renewal calendar.

Runbooks & on-call

The discipline that takes "ships agents" to "operates a platform." For each production agent class:

• Runbook: what each alert means, how to diagnose, how to escalate.
• Pause / kill switch: documented, tested, accessible to the Process Owner without engineering involvement.
• On-call rotation: who's responsible for AI-platform incidents.
• Escalation tree: AI Architect → Engineering Lead → CTO; Process Owner → Controller → CFO.
• Quarterly DR drill: pretend a top integration is down for 4 hours; show you can complete close manually with the audit-log story intact.

The ops dashboard

One pane the AI Architect and the Controller both look at. The metrics that matter:

• Per agent: runs/day, success/halt/escalate split, override rate, eval pass rate (weekly sample), p95 latency, cost.
• Per workflow: end-to-end SLA, FTE-equivalent saved, items processed.
• Integration health: each external API's freshness, error rate, rate-limit headroom.
• Audit log health: write latency, missing-row alerts (every run must have a row).
• SOX status: which agents have current design doc signoff; which are out of date.

This dashboard is the artifact that supports the JD's fifth responsibility area: "Build metrics, dashboards, and performance frameworks to measure efficiency gains, cost reduction, risk outcomes, and ROI."