Data & Pipelines

The telemetry pipeline

The basic shape, drawn left to right:

GPU nodes ──┐
            │  DCGM exporter (DaemonSet)        Prometheus / Mimir / Cortex
            ├──────────────────────────────────▶ scrape /metrics ──▶ TSDB
Serving pods│  /metrics from vLLM, Triton                                    ─▶ Grafana
            └──────────────────────────────────▶                              ─▶ Alertmanager
                                                                              ─▶ Cost ETL ──▶ data warehouse

Logs:    pods ──▶ fluent-bit ──▶ Loki / ELK / Datadog
Traces:  pods (OTel SDK) ──▶ OTel collector ──▶ Tempo / Jaeger / Honeycomb
  

You don't need to use these specific tools; you need to know what each layer does and what failure modes each has.

DCGM exporter — what you actually scrape

NVIDIA's DCGM (Data Center GPU Manager) reads telemetry directly from the GPU. The dcgm-exporter exposes it as Prometheus metrics. Default fields aren't enough; you'll add profiling fields.

Prometheus shape & scaling

Prometheus is the de facto metrics backbone. For a fleet:

• Cardinality is the enemy. Labels with high cardinality (request_id, prompt_hash) will balloon the index. Stay at: node, gpu_index, model, namespace, deployment, instance.
• Federation or remote-write to a long-term store (Mimir, Cortex, Thanos, VictoriaMetrics) for cross-cluster queries and retention.
• Recording rules precompute heavy expressions (p95 latency over 5m windowed) so dashboards are snappy.
• Histograms over averages for any latency metric — histogram_quantile on a _bucket series.

# A representative ServiceMonitor (Prometheus Operator)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  labels: { release: prometheus }
spec:
  selector:
    matchLabels: { app: dcgm-exporter }
  endpoints:
    - port: metrics
      interval: 5s
      scrapeTimeout: 4s
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          targetLabel: node

Logs from inference servers

Don't log the prompt body by default. Do log:

• Request ID (correlatable across services).
• Tenant / API key hash.
• Model and version.
• Prompt token count, response token count.
• TTFT, total latency.
• Cache hit indicator (prefix-cache reused or not).
• Outcome: success / error / aborted by client / timeout.
• Error class if applicable (OOM, NCCL, downstream).

Structured JSON. Fluent-bit (lightweight) or Vector to a backend (Loki, Elastic, Datadog). Retention sized to the regulatory window for any audit-relevant logs.

Tracing across model invocations

For complex pipelines (RAG, agents, multi-stage) traces beat logs. OpenTelemetry is the default standard.

• Gateway opens the root span on a request.
• Each downstream — retriever, embedding service, reranker, LLM — opens a child span.
• Spans carry attributes: model, tokens, cache hit, GPU node.
• Sampling is essential. Tail-based sampling (keep all traces with errors or high latency) gives the best signal-to-cost.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("inference-gateway")

async def handle(request):
    with tracer.start_as_current_span("infer") as span:
        span.set_attribute("model", request.model)
        span.set_attribute("prompt.tokens", request.prompt_tokens)
        try:
            with tracer.start_as_current_span("backend.vllm") as backend_span:
                resp = await call_backend(request)
                backend_span.set_attribute("response.tokens", resp.tokens)
            return resp
        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            raise

The metrics catalog — what every service should expose

If you build a serving service, expose this minimum metric set. vLLM and Triton ship most of these by default.

• {service}_requests_total{status, model} — counter.
• {service}_request_duration_seconds_bucket{model, le} — histogram for percentiles.
• {service}_time_to_first_token_seconds_bucket{model} — histogram.
• {service}_tokens_generated_total{model} — counter.
• {service}_tokens_prompt_total{model} — counter.
• {service}_num_requests_running{model} — gauge.
• {service}_num_requests_waiting{model} — gauge.
• {service}_kv_cache_usage_perc{model} — gauge.
• {service}_prefix_cache_hit_ratio{model} — gauge.

The cost pipeline

Cost is a derived metric. The pipeline:

1. Source: Prometheus GPU-hour-equivalent usage per pod (sum of allocated GPUs × time).
2. Source: cloud billing API (or on-prem amortization table) giving $/GPU-hour by instance type, including the team's reservation/spot mix.
3. ETL: nightly job joins usage × unit cost, attributes via labels (team, namespace, model).
4. Sink: data warehouse (BigQuery, Snowflake, Redshift) with daily granularity.
5. Surface: Grafana dashboards, Slack weekly digest, internal cost portal.

Tools to know: Kubecost / OpenCost (Kubernetes-native), AWS Cost & Usage Reports, GCP Billing Export to BigQuery, Cloudability. None replaces a custom join when you need cross-cloud or on-prem attribution.

Training-side data flows

This role probably won't own training data quality, but you'll own its plumbing.

• Dataset storage — Parquet on S3 with manifest files. Petabyte-class for serious training.
• Data loaders — feeding the GPUs is its own performance problem. PyTorch DataLoader workers, prefetch, pin_memory, IterableDataset for streaming.
• Throughput target — keep the GPU saturated. The "GPU starved on I/O" pathology is real; profile data-loader idle time as part of training health.
• Filesystem — for HPC-style training, Lustre/Weka/FSx-for-Lustre over IB. For lighter training, S3 with a local cache (e.g. s3fs, mountpoint-s3, or pre-staged copies).
• Sharding strategy — pre-shard datasets by rank so workers don't all read the same files.

Model weight pipeline

Weights flow from training to serving. The pipeline you build:

1. Source of truth — Hugging Face Hub mirror or internal model registry (e.g. MLflow, custom artifact store backed by S3 with signed manifests).
2. Versioning — every checkpoint has an immutable ID. Tags like llama3-70b-finetune-2026-05-08 resolve to a hash.
3. Quantization step — produce FP8/INT4 derivatives, store them as siblings of the source FP16.
4. Conversion to serving format — TensorRT-LLM engine builds happen on the target GPU type; safetensors is the source format for vLLM.
5. Distribution — pre-stage to local NVMe on every relevant node; cache for re-use across pod restarts.
6. Garbage collection — old checkpoints are big; lifecycle rules age them out of hot storage to glacier.

✓ Check yourself

Answer from memory before opening each one.

1. A dashboard from default exporter fields shows GPU utilisation at 95 percent, yet users report slow responses. Which fields do you add, and what do they cost?

   Show answer

   Add the profiling fields: SM-active, SM-occupancy, tensor pipe activity and DRAM activity. They show how much of the hardware did work, while the default utilisation field only shows that something ran. They use the GPU's profiling counters, so they add small overhead and can conflict with a profiler, which is why you sample every one to five seconds. See the DCGM exporter.

2. An engineer adds a request ID label to the latency histogram so they can find slow requests. What happens, and what do you suggest instead?

   Show answer

   Every distinct ID creates a new time series per bucket, so the index grows without bound and queries and memory fail. Keep labels to node, GPU index, model, namespace and deployment, and find individual slow requests through traces or logs that carry the request ID. See Prometheus shape and scaling.

3. About one request in a hundred errors out in a RAG pipeline. Why do traces help, and which sampling method keeps those failures?

   Show answer

   A trace joins the retriever, reranker and model calls under one request, so you can see which hop was slow or failed. Sampling one percent up front keeps almost none of the failures. Tail-based sampling decides after the trace completes and keeps errors and slow requests. See tracing across model invocations.

4. p99 latency doubled while GPU utilisation looks normal. Which service metrics do you look at first, and what does each tell you?

   Show answer

   The number of waiting requests shows whether load is queueing, KV-cache usage shows whether memory pressure is limiting concurrency, and tokens generated against the p95 latency shows whether throughput changed. Together these cover most serving incidents. See the metrics catalog.

5. A team reserved eight GPUs and keeps SM-active near 10 percent. What does the cost pipeline bill them, and what does that reveal?

   Show answer

   It bills allocated GPU-hours at the blended unit price, whether or not the GPUs were busy. The gap between that bill and the utilisation shows over-provisioning, which is the saving to chase. Join usage from Prometheus with billing data and attribute by label. See the cost pipeline.

6. Pods pull model weights from a public hub on every restart. Name the problems and the replacement design.

   Show answer

   It is slow, it costs egress and rate limits at scale, and the upstream content can change or be compromised. Mirror weights into an internal store, version them with immutable IDs, pre-stage them to local NVMe on nodes, and pin by digest. See the model weight pipeline.

Takeaway

• The pipeline has four signals: metrics, logs, traces and cost. Know what each layer does and how it fails.
• Scrape DCGM profiling fields as well as defaults, and sample them every one to five seconds.
• Keep Prometheus label cardinality low, use histograms for latency, and use recording rules for heavy queries.
• Log request metadata, not prompt bodies, and use tail-based sampling for traces so errors and slow requests survive.
• Every serving service should expose requests waiting, request duration, tokens generated and KV-cache usage.
• Cost is a derived metric: allocated GPU-hours times blended unit price, attributed by label.
• Weights need a mirrored store, immutable versions, pre-staging to NVMe and pinning by digest.

Next the guide turns the pipeline into practice: how the fleet ships and how the team runs incidents. → Deployment & Ops

Further reading

• DCGM-Exporter — NVIDIA docs. Deploying the exporter and choosing its metric set.
• DCGM field identifiers — NVIDIA docs. The authoritative list of field IDs and what each measures.
• Histograms and summaries — Prometheus docs. Why histograms suit latency and how quantile estimation works.
• Instrumentation: do not overuse labels — Prometheus docs. The cardinality guidance behind a small label set.
• Sampling — OpenTelemetry docs. Head and tail sampling and their trade-offs.
• OpenCost documentation — OpenCost project. Allocating Kubernetes spend to teams and workloads.