Section C · DE inside a neocloud

Fleet Telemetry

The first thing a neocloud DE team builds. GPU-level, node-level, and cluster-level metrics flowing into hot-path operations and into the warehouse for analytics. Everything downstream — billing, capacity planning, fraud detection — depends on this.

Why telemetry is foundational

A neocloud's data systems are built on a foundation of telemetry. Without trustworthy GPU-level and node-level metrics, the company can't:

  • Bill customers accurately.
  • Diagnose problems with specific hardware or jobs.
  • Plan capacity expansion.
  • Detect fraud or abuse.
  • Improve operational efficiency.
  • Build the customer-facing analytics that retain accounts.

The DE team responsible for telemetry sits at the foundation of every other data initiative. Get this wrong and the next 18 months of analytics work is built on sand.

Metric sources

Telemetry comes from many layers:

GPU-level

  • Utilization percentage (SM activity).
  • Memory used / free / fragmentation.
  • Temperature, power draw, fan speed.
  • Clock speeds (boost vs base).
  • ECC error counts.
  • PCIe link state.

Standard source: nvidia-smi, NVML, DCGM (NVIDIA Data Center GPU Manager). DCGM is the production-grade choice.

Node-level

  • CPU utilization, memory, network throughput.
  • Disk I/O and capacity.
  • Process-level metrics.
  • Kernel events.
  • Container / job lifecycle events.

Sources: node_exporter (Prometheus), custom agents, container runtime hooks.

Cluster-level

  • InfiniBand link state, congestion, errors.
  • Network fabric throughput.
  • Storage system performance.
  • Cluster scheduler state.

Job-level (customer-facing)

  • Submitted / started / completed / failed.
  • Resources requested vs used.
  • Job duration.
  • Termination reason.

Customer-API-level

  • API calls (auth, request type, response time).
  • Rate-limit events.
  • Error responses.

Collection

Collection architecture typically blends:

  • Push from node agents. Lightweight agents on each node push metrics to a regional collector.
  • Pull from a metrics endpoint. Prometheus-style scraping from a managed endpoint.
  • Event streams. Job and API events flow through Kafka / equivalent for processing.
  • Batch exports. Periodic batch extracts from systems that don't stream natively (sometimes including the orchestrator).

The collection layer has its own DE concerns — backpressure, agent failure handling, schema evolution, time-zone handling, clock drift. Skipping these creates debug nightmares later.

Hot-path vs warehouse

Telemetry serves two different downstream needs:

Hot path (operations)

Real-time or near-real-time. Powers dashboards on-call uses, alerting, and the customer-facing job-status APIs. Latency requirement: seconds to low minutes.

Typical stack: Prometheus / VictoriaMetrics / similar time-series DB for metrics; Loki / Elasticsearch for logs; Kafka for events. Visualized with Grafana.

Warehouse path (analytics)

Hours-of-lag is acceptable. Powers billing, capacity planning, churn analysis, financial reporting. Latency requirement: hours.

Typical stack: events stream into a warehouse (Snowflake, BigQuery, Iceberg lakehouse); modeled with dbt or equivalent; consumed by BI tools.

Both paths need the same source data but with different SLAs and modeling. The DE team typically maintains the dual-path architecture.

Aggregations & rollups

Raw GPU-level metrics at second-level resolution would overwhelm storage. Rollup strategy:

  • 1-second raw retention: Few days to weeks for debug purposes.
  • 1-minute aggregates: Several months. Powers most operational queries.
  • 1-hour aggregates: Multi-year. Powers analytics and reporting.
  • Daily summaries: Indefinite. Powers historical comparisons.

The downsampling pipeline runs continuously. The DE team owns it. Mistakes here lose data you can't recover.

Downstream use cases

Each downstream use case has different requirements:

  • Billing needs accurate per-second GPU usage per customer. Auditable, append-only, idempotent.
  • Customer-facing dashboards need recent activity per customer, fast.
  • On-call dashboards need fleet-wide health visible at a glance.
  • Capacity planning needs historical utilization to forecast.
  • Quality / reliability needs per-node failure rates and trends.
  • Pricing analytics needs cross-customer aggregates without exposing individual data.
  • Fraud detection needs anomaly inputs derived from telemetry patterns.

Each use case is a downstream consumer. The DE team builds shared upstream sources and lets the downstream teams shape their consumption.

Takeaway

Fleet telemetry is the data backbone of any neocloud. The DE team that builds it well unlocks every other data investment the company makes. The next chapter examines the most consequential downstream consumer: billing.