Section C · DE inside a neocloud

Fleet Telemetry

The first thing a neocloud DE team builds. GPU-level, node-level, and cluster-level metrics flowing into hot-path operations and into the warehouse for analytics. Everything downstream — billing, capacity planning, fraud detection — depends on this.

Why telemetry is foundational

A neocloud's data systems are built on a foundation of telemetry. Without trustworthy GPU-level and node-level metrics, the company can't:

Bill customers accurately.
Diagnose problems with specific hardware or jobs.
Plan capacity expansion.
Detect fraud or abuse.
Improve operational efficiency.
Build the customer-facing analytics that retain accounts.

The DE team responsible for telemetry sits at the foundation of every other data initiative. Get this wrong and the next 18 months of analytics work is built on sand.

Metric sources

Telemetry comes from many layers:

GPU-level

Utilization percentage (SM activity).
Memory used / free / fragmentation.
Temperature, power draw, fan speed.
Clock speeds (boost vs base).
ECC error counts.
PCIe link state.

Standard source: nvidia-smi, NVML, DCGM (NVIDIA Data Center GPU Manager). DCGM is the production-grade choice.

Node-level

CPU utilization, memory, network throughput.
Disk I/O and capacity.
Process-level metrics.
Kernel events.
Container / job lifecycle events.

Sources: node_exporter (Prometheus), custom agents, container runtime hooks.

Cluster-level

InfiniBand link state, congestion, errors.
Network fabric throughput.
Storage system performance.
Cluster scheduler state.

Job-level (customer-facing)

Submitted / started / completed / failed.
Resources requested vs used.
Job duration.
Termination reason.

Customer-API-level

API calls (auth, request type, response time).
Rate-limit events.
Error responses.

Collection

Collection architecture typically blends:

Push from node agents. Lightweight agents on each node push metrics to a regional collector.
Pull from a metrics endpoint. Prometheus-style scraping from a managed endpoint.
Event streams. Job and API events flow through Kafka / equivalent for processing.
Batch exports. Periodic batch extracts from systems that don't stream natively (sometimes including the orchestrator).

The collection layer has its own DE concerns — backpressure, agent failure handling, schema evolution, time-zone handling, clock drift. Skipping these creates debug nightmares later.

Hot-path vs warehouse

Telemetry serves two different downstream needs:

Hot path (operations)

Real-time or near-real-time. Powers dashboards on-call uses, alerting, and the customer-facing job-status APIs. Latency requirement: seconds to low minutes.

Typical stack: Prometheus / VictoriaMetrics / similar time-series DB for metrics; Loki / Elasticsearch for logs; Kafka for events. Visualized with Grafana.

Warehouse path (analytics)

Hours-of-lag is acceptable. Powers billing, capacity planning, churn analysis, financial reporting. Latency requirement: hours.

Typical stack: events stream into a warehouse (Snowflake, BigQuery, Iceberg lakehouse); modeled with dbt or equivalent; consumed by BI tools.

Both paths need the same source data but with different SLAs and modeling. The DE team typically maintains the dual-path architecture.

Aggregations & rollups

Raw GPU-level metrics at second-level resolution would overwhelm storage. Rollup strategy:

1-second raw retention: Few days to weeks for debug purposes.
1-minute aggregates: Several months. Powers most operational queries.
1-hour aggregates: Multi-year. Powers analytics and reporting.
Daily summaries: Indefinite. Powers historical comparisons.

The downsampling pipeline runs continuously. The DE team owns it. Mistakes here lose data you can't recover.

Downstream use cases

Each downstream use case has different requirements:

Billing needs accurate per-second GPU usage per customer. Auditable, append-only, idempotent.
Customer-facing dashboards need recent activity per customer, fast.
On-call dashboards need fleet-wide health visible at a glance.
Capacity planning needs historical utilization to forecast.
Quality / reliability needs per-node failure rates and trends.
Pricing analytics needs cross-customer aggregates without exposing individual data.
Fraud detection needs anomaly inputs derived from telemetry patterns.

Each use case is a downstream consumer. The DE team builds shared upstream sources and lets the downstream teams shape their consumption.

✓ Check yourself

Answer from memory before opening each one.

A customer disputes a charge and your only record is one-minute utilization averages. Why is that not enough, and what should billing read instead?

Show answer
Billing needs accurate per-second usage per customer, which comes from instance lifecycle events and heartbeats, not from rolled-up averages. Rollups are for operations and analytics. Keep billing's source auditable, append-only and idempotent. See downstream use cases.
Why does a neocloud keep two paths for the same telemetry?

Show answer
The consumers need different latency and modeling. On-call dashboards, alerts and job-status APIs need seconds to minutes, while billing, capacity planning and finance accept hours and need modeled tables. One path cannot serve both without compromising one. See hot path vs warehouse.
One node agent's clock runs 90 seconds fast. Which downstream results go wrong?

Show answer
Events arrive out of order, windows assign data to the wrong period, and usage near a billing boundary lands in the wrong month. Collection needs clock-drift handling alongside backpressure and schema evolution. See collection.
You downsample one-second metrics to one-minute averages after two weeks. A month later someone asks for the worst power-draw spike on a node. Why can you not answer?

Show answer
An average hides peaks, and the raw data is gone. Keep min, max and percentiles beside the mean in rollups, and be sure of retention before deleting, since downsampling mistakes lose data you cannot recover. See aggregations and rollups.
You want to catch failing GPUs before customers do. Which signals from which source would you collect first?

Show answer
Use DCGM for GPU health: ECC error counts, PCIe link state, clock behavior, temperature and power. Use node-level agents for host faults and the fabric counters for InfiniBand errors. DCGM is the production-grade source, not ad hoc nvidia-smi calls. See metric sources.

Takeaway

Telemetry underlies billing, diagnosis, capacity planning, fraud detection and customer analytics, so errors here spread everywhere.
Sources span GPU, node, cluster, job and API layers, and DCGM is the production-grade GPU source.
Collection mixes agent push, Prometheus-style pull, event streams and batch exports, and needs backpressure, schema evolution and clock handling.
A hot path serves operations in seconds and a warehouse path serves billing and planning in hours, both fed from the same sources.
Rollups must keep what later questions need, and a bad downsampling job loses data permanently.
Billing reads lifecycle events, not rollups, so it stays auditable and idempotent.

The most consequential consumer of that telemetry is billing. → Billing & Metering