Observability & Ops — Datadog, PagerDuty, Splunk, Grafana

Why integrate with the customer's tools

• Their on-call team uses their tools. Alerts that don't reach their tooling are alerts no one acts on.
• Their SLO dashboards live in their tools. Adding your platform's metrics there means the customer's leadership sees them.
• Their runbook conventions live in their tools. Following their patterns makes handoff cleaner.

Operations is a place where conforming to the customer's existing patterns earns trust. Don't try to bring your platform's observability stack into their on-call rotation.

Datadog

What it is

Most common observability platform at modern enterprises. Metrics, logs, traces, synthetics in one. SaaS.

How to integrate

Datadog's API and agent. Pipeline metrics push via StatsD or HTTP API. Logs ingested via the Datadog Agent or HTTP. Dashboards built in Datadog's UI; alerts configured against metrics.

from datadog import initialize, statsd

initialize(api_key=settings.datadog_api_key)
statsd.increment("contract_intel.documents.ingested",
                 tags=[f"customer:{customer_id}", f"doc_class:{doc_class}"])
statsd.histogram("contract_intel.extraction.duration_ms",
                 duration_ms, tags=[f"customer:{customer_id}"])

What to push

• Pipeline metrics: ingestion rate, extraction latency, error rate.
• Data-quality metrics: confidence distribution, HITL queue depth.
• SLA metrics: accuracy by class, breaches.
• Logs: structured JSON with correlation_id, customer_id, document_id for traceability.

Gotchas

• Pricing scales with cardinality (number of unique tag combinations). Tag carefully.
• Customer's Datadog org has its own conventions; check before adding new dashboards.

PagerDuty

What it is

Incident management and on-call paging. Most common alerting destination at modern enterprises.

How to integrate

Events API; trigger via HTTP POST. Datadog and other monitoring tools integrate natively.

import httpx

def trigger_pagerduty(severity: str, summary: str, dedup_key: str, customer_id: str):
    httpx.post(
        "https://events.pagerduty.com/v2/enqueue",
        json={
            "routing_key": settings.pagerduty_routing_key,
            "event_action": "trigger",
            "dedup_key": dedup_key,
            "payload": {
                "summary": summary,
                "severity": severity,
                "source": "contract-intel-platform",
                "custom_details": {"customer_id": customer_id},
            },
        },
    )

What to alert on

• P1: pipeline failure during business hours; SLA breach.
• P2: quality trending toward SLA line; HITL queue overflow.
• P3: input or output drift; investigate next business day.

Embedded-ops handoff

During on-call handoff (FDE-DE Playbook §08), the customer's PagerDuty rotation picks up. Your alerts route to their rotation, following their escalation policy.

Splunk

What it is

Enterprise log management and SIEM. Common at large regulated industries (finance, healthcare, government).

How to integrate

HTTP Event Collector (HEC) for log forwarding. Datadog-like agents available. Splunk SPL (Search Processing Language) for queries.

Gotchas

• Customer's Splunk admin gates HEC tokens; provisioning takes time.
• Pricing model often based on ingest volume; controlling log volume matters.
• Customer's Splunk apps and dashboards have their own conventions; integrate gradually.

Grafana / Prometheus

What it is

Open-source metrics and dashboarding stack. Common at tech-forward customers, especially those running Kubernetes.

How to integrate

• Expose Prometheus-style metrics endpoint (/metrics) from your services.
• Customer's Prometheus scrapes; Grafana dashboards visualize.
• Alertmanager for alert routing.

Gotchas

• Self-hosted; customer's reliability depends on their own operations.
• Cardinality limits same as Datadog — tag carefully.

New Relic

Similar to Datadog. Less common at modern enterprises today; still present at customers who adopted it years ago. Integration patterns mirror Datadog's.

Embedded-ops patterns

Stage 1: FDE-primary on-call

During the first 8–10 weeks, alerts route to the FDE's own PagerDuty or Slack. Customer's on-call observes; doesn't get paged.

Stage 2: Joint on-call

Alerts route to both the FDE and the customer's data-team on-call. Customer's team triages first using runbooks; FDE backup.

Stage 3: Customer-primary on-call

Alerts route to the customer's PagerDuty rotation. FDE is on a separate escalation path only for platform-team issues. Standard production state post-handoff.

The dashboard that matters most

The single most-watched dashboard during a deployment is the customer-facing SLA dashboard. Build it in their BI tool (Looker / Power BI / Tableau) where they already look, not in Datadog where only the on-call sees it. Reserve Datadog for operational metrics — service health, pipeline throughput — that the engineering team cares about, not business outcomes.

Logging hygiene

Structured JSON logs with these fields on every log line:

• correlation_id: traces a single request across services.
• customer_id: which customer.
• document_id or extraction_id: which artifact (when applicable).
• model_version, pipeline_version: which code produced this log.
• level: INFO / WARN / ERROR / FATAL.

This shape makes the logs searchable in any tool. The customer's incident response runs faster when their on-call can grep for a specific document_id and see the full pipeline trace.