Data & Pipelines

The telemetry pipeline

The basic shape, drawn left to right:

GPU nodes ──┐
            │  DCGM exporter (DaemonSet)        Prometheus / Mimir / Cortex
            ├──────────────────────────────────▶ scrape /metrics ──▶ TSDB
Serving pods│  /metrics from vLLM, Triton                                    ─▶ Grafana
            └──────────────────────────────────▶                              ─▶ Alertmanager
                                                                              ─▶ Cost ETL ──▶ data warehouse

Logs:    pods ──▶ fluent-bit ──▶ Loki / ELK / Datadog
Traces:  pods (OTel SDK) ──▶ OTel collector ──▶ Tempo / Jaeger / Honeycomb
  

You don't need to use these specific tools; you need to know what each layer does and what failure modes each has.

DCGM exporter — what you actually scrape

NVIDIA's DCGM (Data Center GPU Manager) reads telemetry directly from the GPU. The dcgm-exporter exposes it as Prometheus metrics. Default fields aren't enough; you'll add profiling fields.

Prometheus shape & scaling

Prometheus is the de facto metrics backbone. For a fleet:

• Cardinality is the enemy. Labels with high cardinality (request_id, prompt_hash) will balloon the index. Stay at: node, gpu_index, model, namespace, deployment, instance.
• Federation or remote-write to a long-term store (Mimir, Cortex, Thanos, VictoriaMetrics) for cross-cluster queries and retention.
• Recording rules precompute heavy expressions (p95 latency over 5m windowed) so dashboards are snappy.
• Histograms over averages for any latency metric — histogram_quantile on a _bucket series.

# A representative ServiceMonitor (Prometheus Operator)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  labels: { release: prometheus }
spec:
  selector:
    matchLabels: { app: dcgm-exporter }
  endpoints:
    - port: metrics
      interval: 5s
      scrapeTimeout: 4s
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          targetLabel: node

Logs from inference servers

Don't log the prompt body by default. Do log:

• Request ID (correlatable across services).
• Tenant / API key hash.
• Model and version.
• Prompt token count, response token count.
• TTFT, total latency.
• Cache hit indicator (prefix-cache reused or not).
• Outcome: success / error / aborted by client / timeout.
• Error class if applicable (OOM, NCCL, downstream).

Structured JSON. Fluent-bit (lightweight) or Vector to a backend (Loki, Elastic, Datadog). Retention sized to the regulatory window for any audit-relevant logs.

Tracing across model invocations

For complex pipelines (RAG, agents, multi-stage) traces beat logs. OpenTelemetry is the default standard.

• Gateway opens the root span on a request.
• Each downstream — retriever, embedding service, reranker, LLM — opens a child span.
• Spans carry attributes: model, tokens, cache hit, GPU node.
• Sampling is essential. Tail-based sampling (keep all traces with errors or high latency) gives the best signal-to-cost.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("inference-gateway")

async def handle(request):
    with tracer.start_as_current_span("infer") as span:
        span.set_attribute("model", request.model)
        span.set_attribute("prompt.tokens", request.prompt_tokens)
        try:
            with tracer.start_as_current_span("backend.vllm") as backend_span:
                resp = await call_backend(request)
                backend_span.set_attribute("response.tokens", resp.tokens)
            return resp
        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            raise

The metrics catalog — what every service should expose

If you build a serving service, expose this minimum metric set. vLLM and Triton ship most of these by default.

• {service}_requests_total{status, model} — counter.
• {service}_request_duration_seconds_bucket{model, le} — histogram for percentiles.
• {service}_time_to_first_token_seconds_bucket{model} — histogram.
• {service}_tokens_generated_total{model} — counter.
• {service}_tokens_prompt_total{model} — counter.
• {service}_num_requests_running{model} — gauge.
• {service}_num_requests_waiting{model} — gauge.
• {service}_kv_cache_usage_perc{model} — gauge.
• {service}_prefix_cache_hit_ratio{model} — gauge.

The cost pipeline

Cost is a derived metric. The pipeline:

1. Source: Prometheus GPU-hour-equivalent usage per pod (sum of allocated GPUs × time).
2. Source: cloud billing API (or on-prem amortization table) giving $/GPU-hour by instance type, including the team's reservation/spot mix.
3. ETL: nightly job joins usage × unit cost, attributes via labels (team, namespace, model).
4. Sink: data warehouse (BigQuery, Snowflake, Redshift) with daily granularity.
5. Surface: Grafana dashboards, Slack weekly digest, internal cost portal.

Tools to know: Kubecost / OpenCost (Kubernetes-native), AWS Cost & Usage Reports, GCP Billing Export to BigQuery, Cloudability. None replaces a custom join when you need cross-cloud or on-prem attribution.

Training-side data flows

This role probably won't own training data quality, but you'll own its plumbing.

• Dataset storage — Parquet on S3 with manifest files. Petabyte-class for serious training.
• Data loaders — feeding the GPUs is its own performance problem. PyTorch DataLoader workers, prefetch, pin_memory, IterableDataset for streaming.
• Throughput target — keep the GPU saturated. The "GPU starved on I/O" pathology is real; profile data-loader idle time as part of training health.
• Filesystem — for HPC-style training, Lustre/Weka/FSx-for-Lustre over IB. For lighter training, S3 with a local cache (e.g. s3fs, mountpoint-s3, or pre-staged copies).
• Sharding strategy — pre-shard datasets by rank so workers don't all read the same files.

Model weight pipeline

Weights flow from training to serving. The pipeline you build:

1. Source of truth — Hugging Face Hub mirror or internal model registry (e.g. MLflow, custom artifact store backed by S3 with signed manifests).
2. Versioning — every checkpoint has an immutable ID. Tags like llama3-70b-finetune-2026-05-08 resolve to a hash.
3. Quantization step — produce FP8/INT4 derivatives, store them as siblings of the source FP16.
4. Conversion to serving format — TensorRT-LLM engine builds happen on the target GPU type; safetensors is the source format for vLLM.
5. Distribution — pre-stage to local NVMe on every relevant node; cache for re-use across pod restarts.
6. Garbage collection — old checkpoints are big; lifecycle rules age them out of hot storage to glacier.