Section C · Identity & ops

Observability & Ops — Datadog, PagerDuty, Splunk, Grafana

Where pipeline metrics and alerts go. The customer's existing observability tooling determines where your deployment's signals land. Integrate; don't impose new tools.

Why integrate with the customer's tools

  • Their on-call team uses their tools. Alerts that don't reach their tooling are alerts no one acts on.
  • Their SLO dashboards live in their tools. Adding your platform's metrics there means the customer's leadership sees them.
  • Their runbook conventions live in their tools. Following their patterns makes handoff cleaner.

Operations is a place where conforming to the customer's existing patterns earns trust. Don't try to bring your platform's observability stack into their on-call rotation.

Datadog

What it is

Most common observability platform at modern enterprises. Metrics, logs, traces, synthetics in one. SaaS.

How to integrate

Datadog's API and agent. Pipeline metrics push via StatsD or HTTP API. Logs ingested via the Datadog Agent or HTTP. Dashboards built in Datadog's UI; alerts configured against metrics.

from datadog import initialize, statsd

initialize(statsd_host="127.0.0.1", statsd_port=8125)  # DogStatsD in the Agent; no API key
statsd.increment("contract_intel.documents.ingested",
                 tags=[f"customer:{customer_id}", f"doc_class:{doc_class}"])
statsd.histogram("contract_intel.extraction.duration_ms",
                 duration_ms, tags=[f"customer:{customer_id}"])

What to push

  • Pipeline metrics: ingestion rate, extraction latency, error rate.
  • Data-quality metrics: confidence distribution, HITL queue depth.
  • SLA metrics: accuracy by class, breaches.
  • Logs: structured JSON with correlation_id, customer_id, document_id for traceability.

Gotchas

  • Pricing scales with cardinality (number of unique tag combinations). Tag carefully.
  • Customer's Datadog org has its own conventions; check before adding new dashboards.

PagerDuty

What it is

Incident management and on-call paging. Most common alerting destination at modern enterprises.

How to integrate

Events API; trigger via HTTP POST. Datadog and other monitoring tools integrate natively.

import httpx

def trigger_pagerduty(severity: str, summary: str, dedup_key: str, customer_id: str):
    httpx.post(
        "https://events.pagerduty.com/v2/enqueue",
        json={
            "routing_key": settings.pagerduty_routing_key,
            "event_action": "trigger",
            "dedup_key": dedup_key,
            "payload": {
                "summary": summary,
                "severity": severity,
                "source": "contract-intel-platform",
                "custom_details": {"customer_id": customer_id},
            },
        },
    )

What to alert on

  • P1: pipeline failure during business hours; SLA breach.
  • P2: quality trending toward SLA line; HITL queue overflow.
  • P3: input or output drift; investigate next business day.

Embedded-ops handoff

During on-call handoff (FDE-DE Playbook §08), the customer's PagerDuty rotation picks up. Your alerts route to their rotation, following their escalation policy.

Splunk

What it is

Enterprise log management and SIEM. Common at large regulated industries (finance, healthcare, government).

How to integrate

HTTP Event Collector (HEC) for log forwarding. Datadog-like agents available. Splunk SPL (Search Processing Language) for queries.

Gotchas

  • Customer's Splunk admin gates HEC tokens; provisioning takes time.
  • Pricing model often based on ingest volume; controlling log volume matters.
  • Customer's Splunk apps and dashboards have their own conventions; integrate gradually.

Grafana / Prometheus

What it is

Open-source metrics and dashboarding stack. Common at tech-forward customers, especially those running Kubernetes.

How to integrate

  • Expose Prometheus-style metrics endpoint (/metrics) from your services.
  • Customer's Prometheus scrapes; Grafana dashboards visualize.
  • Alertmanager for alert routing.

Gotchas

  • Self-hosted; customer's reliability depends on their own operations.
  • Cardinality limits same as Datadog — tag carefully.

New Relic

Similar to Datadog. Less common at modern enterprises today; still present at customers who adopted it years ago. Integration patterns mirror Datadog's.

Embedded-ops patterns

Stage 1: FDE-primary on-call

During the first 8–10 weeks, alerts route to the FDE's own PagerDuty or Slack. Customer's on-call observes; doesn't get paged.

Stage 2: Joint on-call

Alerts route to both the FDE and the customer's data-team on-call. Customer's team triages first using runbooks; FDE backup.

Stage 3: Customer-primary on-call

Alerts route to the customer's PagerDuty rotation. FDE is on a separate escalation path only for platform-team issues. Standard production state post-handoff.

The dashboard that matters most

The single most-watched dashboard during a deployment is the customer-facing SLA dashboard. Build it in their BI tool (Looker / Power BI / Tableau) where they already look, not in Datadog where only the on-call sees it. Reserve Datadog for operational metrics — service health, pipeline throughput — that the engineering team cares about, not business outcomes.

Logging hygiene

Structured JSON logs with these fields on every log line:

  • correlation_id: traces a single request across services.
  • customer_id: which customer.
  • document_id or extraction_id: which artifact (when applicable).
  • model_version, pipeline_version: which code produced this log.
  • level: INFO / WARN / ERROR / FATAL.

This shape makes the logs searchable in any tool. The customer's incident response runs faster when their on-call can grep for a specific document_id and see the full pipeline trace.

✓ Check yourself

Answer from memory before opening each one.

  1. After your team starts tagging every ingestion metric with document_id, the Datadog bill jumps. Why, and what do you do?

    Show answer

    Pricing scales with the number of unique tag combinations, and a document ID makes every series unique. Keep tags to low-cardinality values such as customer and doc_class, and put the document ID in logs. See Datadog.

  2. A failing pipeline pages the on-call engineer every minute. Which field in the PagerDuty Events API stops the storm, and how?

    Show answer

    The dedup_key. Events that share a routing key and dedup key attach to one open incident instead of opening a new one. Derive the key from the failure, for example pipeline plus customer. See PagerDuty.

  3. It is week 6 and the customer's lead asks to be paged first. Which on-call stage does that describe, and what do you confirm before you agree?

    Show answer

    Joint on-call: the customer's team triages first using runbooks, with the FDE as backup. Confirm that runbooks exist for the alerts they will receive, and that the FDE stays on the escalation path. Customer-primary on-call comes after that. See embedded-ops patterns.

  4. Where does the customer-facing SLA dashboard go, and what stays in Datadog?

    Show answer

    Put the SLA dashboard in the customer's BI tool, such as Looker, Power BI or Tableau, where business users already look. Keep Datadog for operational metrics like service health and pipeline throughput, which the engineers care about. See embedded-ops patterns.

  5. The customer runs Prometheus and Grafana on Kubernetes and has no Datadog. How do you expose your metrics, and how do alerts reach people?

    Show answer

    Serve a Prometheus-style /metrics endpoint from your services, so their Prometheus scrapes it and Grafana draws it. Alertmanager routes the alerts. Keep label cardinality low, as with Datadog tags. See Grafana and Prometheus.

  6. At 2 a.m. the customer's on-call has only a document ID from a failed extraction. Which log fields let them rebuild what happened, and why in every tool?

    Show answer

    Structured JSON with correlation_id, customer_id, document_id or extraction_id, model_version, pipeline_version and level on every line. Searching the document ID returns the whole pipeline trace in any log tool, which suits the customer's incident response. See embedded-ops patterns.

Takeaway

  • Alerts that do not reach the customer's tooling are alerts nobody acts on, so conform to their stack and do not bring your own into their rotation.
  • Metric pricing follows tag cardinality. Tag with customer and class, and keep document IDs in logs.
  • Use a stable dedup key on every PagerDuty event so one failure produces one incident.
  • Move through three on-call stages: FDE-primary, joint, then customer-primary, each with runbooks behind it.
  • The customer-facing SLA dashboard belongs in their BI tool, and Datadog stays for operational metrics.
  • Structured logs carrying a correlation ID, customer, document and version make any incident traceable in any tool.

The on-call stages end in a handoff, and the playbook has the checklist for it. → Handoff Checklist in the FDE-DE Playbook

Further reading

  • DogStatsD — Datadog docs, accessed 2026. How the Agent receives StatsD metrics, and the client settings.
  • Custom metrics — Datadog docs, accessed 2026. How custom metrics are counted and billed, including tag cardinality.
  • Events API v2 overview — PagerDuty developer docs, accessed 2026. Trigger, acknowledge and resolve events, and how deduplication works.
  • Exposition formats — Prometheus docs, accessed 2026. The text format a /metrics endpoint serves.
  • Alertmanager — Prometheus docs, accessed 2026. Grouping, routing and silencing alerts.