Deployment & Operations

Environments & staging

The three-environment baseline:

Workday sandbox & tenant refresh cadence

Workday provides multiple tenants — Implementation, Sandbox, Sandbox Preview, Production. Important properties:

• Sandbox is refreshed from production on a schedule (weekly or per-event). Your test data is wiped at refresh.
• Sandbox Preview previews upcoming Workday releases before they hit production — your integration must be tested there before each release weekend.
• Configuration changes (BPs, calc fields, security) made in sandbox don't auto-propagate to prod — they migrate via Workday's release/config migration tools or are manually re-keyed.

For your agent: pin to the Sandbox Preview tenant during the two weeks before each Workday release. Run a regression against it. Block release of agent changes that would conflict with the upcoming Workday release.

Change control synced to the payroll calendar

The single most People-specific operational discipline: no agent or integration change deploys to production within the payroll lock window for any major-population country.

Concretely:

• Maintain a calendar of all payroll cutoffs across all entities
• Highlight a "no-deploy" window of ~3 business days before each cutoff
• If you have multiple frequencies (monthly UK, bi-weekly US), the no-deploy windows overlap; in practice you have ~2 windows per month where you can deploy
• Emergency changes require Payroll Ops + Process Owner sign-off, with a rollback plan in hand

def can_deploy(now: date, countries: list[str], cal: PayrollCalendar) -> dict:
    """Return whether deploy is safe today, with rationale."""
    blockers = []
    for c in countries:
        win = cal.lock_window(c, now)
        if win["status"] in ("locked", "caution"):
            blockers.append({"country": c, **win})
    return {"safe": not blockers, "blockers": blockers}

CI/CD with HITL gates

CI for an HR agent pipeline has more gates than a typical web service:

1. Lint, type check — black, ruff, mypy
2. Unit tests — green, with coverage threshold on critical modules (idempotency, payroll-window, redaction)
3. Schema tests — every Pydantic model round-trips; EIB header snapshot matches
4. Eval suite — runs on a labeled set; metrics meet thresholds
5. PII tests — zero tolerance; any regression blocks merge
6. Fairness baselines — for agents with fairness implications, the dashboard delta is within threshold
7. Staging deploy + smoke test — run a known input through the deployed agent against sandbox Workday
8. HITL approval for promotion — to ship the change, the Process Owner approves in the release tool
9. Payroll-calendar check — automated, blocks if in lock window without override
10. Production deploy — with canary if relevant; full rollout after observation window

Rollback strategy

Rolling back an HR agent is harder than rolling back a web service because partial effects may have committed to Workday and downstream systems. Plan for it:

• Feature flags everywhere — every new agent capability is behind a flag, toggle-off rolls back instantly
• Idempotency keys let you safely re-run after a rollback
• The propose/commit split means a rollback at the proposing stage costs nothing — just abandon the queued proposals
• Audit log preserves intent — even rolled-back runs are logged, so you can reconstruct what would have happened
• For committed actions, you build the reversal — there's no automatic undo; the reversal is a new BP (e.g., revert job change)

Observability & run logs

Three layers of observability:

Infrastructure

• Standard APM on the n8n hosts, Python services, and MCP servers
• Workday API call rates, error rates, latency (Workday is the most common bottleneck)
• Cost dashboards per agent (Claude tokens, EIB run frequency)

Agent-level

• Run count per agent per day
• Tool call distribution (which tools fired, in what order)
• Token usage per run (input, cached, output)
• Refusal rate by reason
• Halt rate by halt-criterion

Outcome-level

• Approval rate (% of HITL proposals approved as-drafted)
• Edit rate (% approved after human edits)
• Reject rate
• Time-to-decision (proposal → approval/reject)
• Sampled human review pass rate (on auto-committed low-tier flows)

The North-Star metric for an HR agent's health: edit rate. Tracks "how often is the human modifying what the agent drafted?" High edit rate = agent isn't trusted. Low edit rate = ready to consider loosening HITL.

PII handling in logs

The rules:

• Never log Restricted-class fields verbatim. Salary, SSN, government ID, performance rating, medical info.
• Log hashes or tokens for cross-reference. worker_id_hash in log; the resolution table is access-controlled.
• Structured log fields, not free-text — easier to redact, easier to audit, easier to filter on data class.
• Error messages strip PII — "validation failed for proposal P-123" not "validation failed: SSN 123-45-6789 was wrong format."
• Per-record retention — operational logs aged out after retention; Restricted-class agent run logs retained per policy but access-restricted.
• Log access is logged — who pulled what audit entry, when.

import logging, hashlib, json

class PIISafeFormatter(logging.Formatter):
    RESTRICTED_KEYS = {"ssn", "salary", "salary_amount", "performance_rating", "govt_id"}

    def format(self, record):
        if isinstance(record.msg, dict):
            redacted = {k: ("<redacted>" if k in self.RESTRICTED_KEYS else v)
                        for k, v in record.msg.items()}
            record.msg = json.dumps(redacted)
        return super().format(record)

def hash_id(worker_id: str) -> str:
    return hashlib.sha256(worker_id.encode()).hexdigest()[:16]

Secrets & auth

• Workday integration system user (ISU) with minimum privileges for the integration — read for read-MCP, write only for write-MCP
• Workday tokens are short-lived — refresh logic is first-class, not bolt-on
• Separate ISUs per agent when feasible — limits blast radius on credential compromise
• Secrets manager, not env vars in n8n exports
• Rotation — quarterly minimum, automated where possible
• Audit on credential use — Workday's audit log captures every ISU action; correlate with your agent's run_id

On-call & runbooks

Each agent has a runbook owned by its Process Owner. The runbook contains:

• What the agent does, in 3 sentences
• Trigger and schedule
• Halt criteria
• How to read the audit log
• Common failure modes and their resolutions
• How to disable the agent (the kill switch)
• Escalation chain — who to call when
• Replay protocol (given a run_id, how to reproduce)
• Postmortem template

For Tier 3+ agents, on-call rotation includes the architect and the Process Owner. The architect debugs technical; the Process Owner decides business resolution.

Interview framings worth memorizing

• "Staging mirrors a Workday Sandbox tenant; we test against Sandbox Preview during the two weeks before each Workday release."
• "No deploy window during the 3 business days before any major-population payroll cutoff. Calendar-aware change control is a hard rule."
• "For Tier 3+, the Process Owner is a CI gate. They approve to ship. That's how adoption gets driven."
• "Rollback for payroll-affecting commits means halt + manual reversal with Payroll Ops. I don't promise instant undo I can't deliver."
• "My North-Star agent health metric is edit rate. It's the closest proxy to operator trust."
• "Logs never carry Restricted-class fields verbatim. Hashes for cross-reference, structured fields for redaction, access logged on the audit store."

✓ Check yourself

Answer from memory before opening each one.

1. An EIB passes in staging and fails in production because a calculated field the template uses exists only in production. What was wrong with the environment, and what is the standing rule?

   Show answer

   Staging used a Workday tenant that did not mirror production schema and business process configuration. The rule is that staging must use a sandbox that mirrors production, refreshed on a schedule, and you must remember that refresh wipes your test data and that config changes made in a sandbox do not propagate by themselves. See Environments & staging.

2. A fix for a prompt bug is ready on the 27th, and the US and UK payroll cutoffs fall on the 28th and 30th. What does your pipeline do, and who can override?

   Show answer

   The payroll-calendar check blocks the deploy because both countries are in or near lock windows. An emergency change needs Payroll Ops and the Process Owner to sign off, with a rollback plan in hand. With overlapping frequencies you only have about two usable windows a month. See Change control synced to the payroll calendar.

3. Why is the Process Owner an approver in the release pipeline for higher-tier agents, and what goes wrong if they are not?

   Show answer

   It puts the operator who owns the workflow in charge of changes to their own tool, which both controls risk and drives adoption. Without it the architect can change behaviour the owners have not seen, and ownership never transfers in practice. See CI/CD with HITL gates.

4. An agent change went live and committed 12 wrong comp proposals before someone noticed. What can you undo instantly, and what can you only reverse by hand?

   Show answer

   You can instantly turn off the feature flag and halt further commits, abandon queued proposals, and re-run safely because of idempotency keys. Commits already in Workday have no automatic undo, so a reversal is a new business process run with Payroll Ops and named owners. Promise instant halt, not instant rollback. See Rollback strategy.

5. Edit rate for the policy Q&A agent climbs from 8% to 35% over two weeks, with no failed health checks. What does it suggest, and what do you check?

   Show answer

   That the humans trust the drafts less, even though nothing is technically down, so quality has regressed. Check recent changes to prompts, policy sources or the model, which question types get edited, and the eval and retrieval metrics on a shadow sample. Edit rate is the primary health signal because it tracks operator trust. See Observability & run logs.

6. A teammate adds the whole worker dict to an info-level log line to debug a failing proposal. What are the problems, and what do you do instead?

   Show answer

   The record may carry restricted fields such as salary or government ID that must never be logged verbatim, and free text is hard to redact. Log structured fields with hashed worker IDs and strip sensitive context from errors, keeping the lookup table access-controlled and the access to logs audited. See PII handling in logs.

Takeaway

• Staging must mirror production's Workday schema and business processes, and Sandbox Preview is used to regression-test before each Workday release.
• No agent or integration change deploys in the roughly three business days before any major payroll cutoff, with emergency exceptions signed by Payroll Ops and the Process Owner.
• The pipeline adds gates a web service does not have: eval thresholds, zero-tolerance PII tests, fairness deltas, a payroll-calendar check and the Process Owner's approval.
• Rollback means instant halt through flags and abandoned proposals, followed by a manual reversal with Payroll Ops for anything already committed.
• Edit rate is the lead health metric for an agent, supported by run, tool, halt and cost metrics.
• Logs hold hashes and structured fields, never restricted values, and secrets live in a manager with per-agent least-privilege service accounts, short-lived tokens and rotation.

You can now build, ship and run these agents; the next chapter is the vocabulary that lets you speak about them fluently. → Domain Context & Glossary

Further reading

• Release Engineering — Google SRE Book. Hermetic builds, gated promotion and rollout policy for safe releases.
• Feature Toggles — Martin Fowler. How flags work as a kill switch and their operational cost.
• OpenTelemetry documentation — OpenTelemetry project. Traces, metrics and logs instrumentation for the observability layers.
• Secrets Management Cheat Sheet — OWASP. Storage, rotation and audit practice for service credentials.
• Postmortem Culture: Learning from Failure — Google SRE Book. The blameless postmortem approach behind the incident playbook.