Deployment & Operations

Environments & staging

The three-environment baseline:

Workday sandbox & tenant refresh cadence

Workday provides multiple tenants — Implementation, Sandbox, Sandbox Preview, Production. Important properties:

• Sandbox is refreshed from production on a schedule (weekly or per-event). Your test data is wiped at refresh.
• Sandbox Preview previews upcoming Workday releases before they hit production — your integration must be tested there before each release weekend.
• Configuration changes (BPs, calc fields, security) made in sandbox don't auto-propagate to prod — they migrate via Workday's release/config migration tools or are manually re-keyed.

For your agent: pin to the Sandbox Preview tenant during the two weeks before each Workday release. Run a regression against it. Block release of agent changes that would conflict with the upcoming Workday release.

Change control synced to the payroll calendar

The single most People-specific operational discipline: no agent or integration change deploys to production within the payroll lock window for any major-population country.

Concretely:

• Maintain a calendar of all payroll cutoffs across all entities
• Highlight a "no-deploy" window of ~3 business days before each cutoff
• If you have multiple frequencies (monthly UK, bi-weekly US), the no-deploy windows overlap; in practice you have ~2 windows per month where you can deploy
• Emergency changes require Payroll Ops + Process Owner sign-off, with a rollback plan in hand

def can_deploy(now: date, countries: list[str], cal: PayrollCalendar) -> dict:
    """Return whether deploy is safe today, with rationale."""
    blockers = []
    for c in countries:
        win = cal.lock_window(c, now)
        if win["status"] in ("locked", "caution"):
            blockers.append({"country": c, **win})
    return {"safe": not blockers, "blockers": blockers}

CI/CD with HITL gates

CI for an HR agent pipeline has more gates than a typical web service:

1. Lint, type check — black, ruff, mypy
2. Unit tests — green, with coverage threshold on critical modules (idempotency, payroll-window, redaction)
3. Schema tests — every Pydantic model round-trips; EIB header snapshot matches
4. Eval suite — runs on a labeled set; metrics meet thresholds
5. PII tests — zero tolerance; any regression blocks merge
6. Fairness baselines — for agents with fairness implications, the dashboard delta is within threshold
7. Staging deploy + smoke test — run a known input through the deployed agent against sandbox Workday
8. HITL approval for promotion — to ship the change, the Process Owner approves in the release tool
9. Payroll-calendar check — automated, blocks if in lock window without override
10. Production deploy — with canary if relevant; full rollout after observation window

Rollback strategy

Rolling back an HR agent is harder than rolling back a web service because partial effects may have committed to Workday and downstream systems. Plan for it:

• Feature flags everywhere — every new agent capability is behind a flag, toggle-off rolls back instantly
• Idempotency keys let you safely re-run after a rollback
• The propose/commit split means a rollback at the proposing stage costs nothing — just abandon the queued proposals
• Audit log preserves intent — even rolled-back runs are logged, so you can reconstruct what would have happened
• For committed actions, you build the reversal — there's no automatic undo; the reversal is a new BP (e.g., revert job change)

Observability & run logs

Three layers of observability:

Infrastructure

• Standard APM on the n8n hosts, Python services, and MCP servers
• Workday API call rates, error rates, latency (Workday is the most common bottleneck)
• Cost dashboards per agent (Claude tokens, EIB run frequency)

Agent-level

• Run count per agent per day
• Tool call distribution (which tools fired, in what order)
• Token usage per run (input, cached, output)
• Refusal rate by reason
• Halt rate by halt-criterion

Outcome-level

• Approval rate (% of HITL proposals approved as-drafted)
• Edit rate (% approved after human edits)
• Reject rate
• Time-to-decision (proposal → approval/reject)
• Sampled human review pass rate (on auto-committed low-tier flows)

The North-Star metric for an HR agent's health: edit rate. Tracks "how often is the human modifying what the agent drafted?" High edit rate = agent isn't trusted. Low edit rate = ready to consider loosening HITL.

PII handling in logs

The rules:

• Never log Restricted-class fields verbatim. Salary, SSN, government ID, performance rating, medical info.
• Log hashes or tokens for cross-reference. worker_id_hash in log; the resolution table is access-controlled.
• Structured log fields, not free-text — easier to redact, easier to audit, easier to filter on data class.
• Error messages strip PII — "validation failed for proposal P-123" not "validation failed: SSN 123-45-6789 was wrong format."
• Per-record retention — operational logs aged out after retention; Restricted-class agent run logs retained per policy but access-restricted.
• Log access is logged — who pulled what audit entry, when.

import logging, hashlib, json

class PIISafeFormatter(logging.Formatter):
    RESTRICTED_KEYS = {"ssn", "salary", "salary_amount", "performance_rating", "govt_id"}

    def format(self, record):
        if isinstance(record.msg, dict):
            redacted = {k: ("<redacted>" if k in self.RESTRICTED_KEYS else v)
                        for k, v in record.msg.items()}
            record.msg = json.dumps(redacted)
        return super().format(record)

def hash_id(worker_id: str) -> str:
    return hashlib.sha256(worker_id.encode()).hexdigest()[:16]

Secrets & auth

• Workday integration system user (ISU) with minimum privileges for the integration — read for read-MCP, write only for write-MCP
• Workday tokens are short-lived — refresh logic is first-class, not bolt-on
• Separate ISUs per agent when feasible — limits blast radius on credential compromise
• Secrets manager, not env vars in n8n exports
• Rotation — quarterly minimum, automated where possible
• Audit on credential use — Workday's audit log captures every ISU action; correlate with your agent's run_id

On-call & runbooks

Each agent has a runbook owned by its Process Owner. The runbook contains:

• What the agent does, in 3 sentences
• Trigger and schedule
• Halt criteria
• How to read the audit log
• Common failure modes and their resolutions
• How to disable the agent (the kill switch)
• Escalation chain — who to call when
• Replay protocol (given a run_id, how to reproduce)
• Postmortem template

For Tier 3+ agents, on-call rotation includes the architect and the Process Owner. The architect debugs technical; the Process Owner decides business resolution.

Interview framings worth memorizing

• "Staging mirrors a Workday Sandbox tenant; we test against Sandbox Preview during the two weeks before each Workday release."
• "No deploy window during the 3 business days before any major-population payroll cutoff. Calendar-aware change control is a hard rule."
• "For Tier 3+, the Process Owner is a CI gate. They approve to ship. That's how adoption gets driven."
• "Rollback for payroll-affecting commits means halt + manual reversal with Payroll Ops. I don't promise instant undo I can't deliver."
• "My North-Star agent health metric is edit rate. It's the closest proxy to operator trust."
• "Logs never carry Restricted-class fields verbatim. Hashes for cross-reference, structured fields for redaction, access logged on the audit store."