AI Compute Platform Data

What these companies do

The "AI compute" category has a few distinct business models. Many companies span multiple:

• Inference cloud — host open models (Llama, Mistral, etc.) on managed GPUs, charge per token. Examples: Together, Fireworks, Replicate, Anyscale, Lepton.
• GPU marketplace — match third-party GPU providers (cloud providers, decentralized operators) with customers who need compute. Often by-the-hour rentals or on-demand. Examples: Vast.ai, RunPod, Akash.
• Training compute — sell access to multi-GPU clusters for model training. Per-hour, per-cluster.
• Fine-tuning platforms — take a base model + customer data, produce a fine-tuned model, serve it.
• Foundation model providers — train their own large models, serve via API (OpenAI, Anthropic, Mistral). Different business model — more like B2B SaaS than infra.

The data analytics work is more interesting at the infrastructure companies (the first three) — there's real unit economics and operational complexity. Foundation model providers have simpler unit economics but more complex training-time analytics.

The data shape

AI compute platforms produce unusual data compared to a classic SaaS:

• Very high volume telemetry — every GPU emits utilization, memory, temperature metrics every few seconds. Thousands of GPUs × seconds = billions of rows fast.
• Inference request logs at scale — every API call has latency, tokens, cost, customer, model, region. Petabyte scale is realistic.
• Time-series flavor — much of the data is time-indexed metrics, not classic transactional data. ClickHouse, Druid, or warehouse time-series tables matter.
• Multi-tenant with stark unit economics — every customer's behavior directly translates to cost. Margin is computable per-customer, per-model, per-region.
• Real-time-ish needs — billing, capacity planning, alerting. Pure daily-batch isn't enough.
• Two-sided marketplace data (for marketplaces) — supply side (providers, their availability and reliability) and demand side (customers, their consumption).

Unit economics — the core analytical problem

The single most-asked-about analytical question at AI compute companies: "are we making money on this customer / this model / this region?"

The basic equation

The leverage

Unit margin scales with two things:

1. Utilization — a GPU rented at $X/hour serving 100 requests/hour costs $X/100 per request. The same GPU serving 1000 requests/hour costs $X/1000. Utilization is the dominant lever on margin.
2. Throughput per dollar — better batching, better inference servers (vLLM, TensorRT-LLM), better quantization push throughput up at the same hardware cost.

Analytics work follows: utilization dashboards, throughput tracking, per-customer-per-model margin reports.

GPU utilization — the load-bearing metric

"Utilization" is overloaded — make sure you know which one is being asked:

The modeling layer

Periodic snapshot fact at hourly grain is standard:

CREATE TABLE fct_gpu_hourly_utilization (
  gpu_id            STRING,
  hour_ts           TIMESTAMP,
  utilization_pct   FLOAT,    -- compute, averaged over the hour
  memory_used_gb    FLOAT,    -- VRAM, averaged
  assigned_minutes  INT,      -- how many minutes was the GPU allocated to a customer
  requests_served   INT,      -- inference requests handled
  revenue_usd       FLOAT,    -- revenue attributed to this GPU this hour
  cost_usd          FLOAT,    -- provider payout or amortized hardware cost
  current_customer_id STRING, -- if single-tenant during this hour
  current_model_id  STRING,   -- model loaded if applicable
  PRIMARY KEY (gpu_id, hour_ts)
);

From this single fact you can answer:

• Fleet-wide utilization by region / provider / GPU type.
• Idle GPU-hours (assigned_minutes < 60).
• Revenue per GPU per day / week / month.
• Margin per GPU (revenue - cost).

Inference logs — petabyte territory

Every API request to an inference platform produces a log entry. At scale this is the biggest dataset in the warehouse.

Schema

CREATE TABLE fct_inference_requests (
  request_id        STRING,
  customer_id       STRING,
  api_key_id        STRING,        -- which key, in case customer has many
  model_id          STRING,
  gpu_id            STRING,        -- which GPU handled it
  region_id         STRING,
  occurred_at       TIMESTAMP,
  date_key          DATE,          -- for partitioning
  latency_ms        INT,           -- end-to-end
  ttft_ms           INT,           -- time to first token (streaming)
  input_tokens      INT,
  output_tokens     INT,
  cost_usd          DECIMAL(18,8), -- compute cost
  revenue_usd       DECIMAL(18,8), -- billed to customer
  status            STRING,        -- 'success' | 'timeout' | 'error' | 'rate_limited'
  error_code        STRING,
  request_metadata  VARIANT/JSON   -- temperature, max_tokens, system_prompt_hash, etc.
);
-- Partition by date_key, cluster by (customer_id, model_id)

Aggregation strategy

Don't serve this raw to analysts. Build a layered aggregation:

Most queries hit the hourly or daily aggs. Raw is for spot debugging.

The token-counting gotcha

Two ways to count tokens differ by ~5-15%:

• Tokenizer output — what the model actually consumed.
• Billable tokens — what you charge for (may round, may have a minimum).

Be explicit about which one a metric is using. "Revenue uses billable_tokens; throughput uses tokenizer_tokens."

Marketplace dynamics (if applicable)

If the company is a two-sided marketplace (GPU providers → customers), the data has supply-side and demand-side dimensions.

Supply side

• Providers — who supplies GPUs. Their reliability, payout terms, geographic distribution.
• GPU inventory — what's available, what's reserved, what's offline.
• Provider utilization — are providers' GPUs being used? Idle GPUs = wasted supply.
• Provider payouts — billing them, attributing revenue.
• Reliability metrics — uptime, failure rates, response time.

Demand side

• Customer onboarding funnel — signup → first request → meaningful usage.
• Cohort retention — do customers stay or churn?
• Concentration risk — top 5 customers' share of revenue.
• Model preferences — which models drive demand.

Marketplace-specific metrics

• Take rate — % of GMV the platform keeps.
• Match quality — how well supply meets demand (queue depth, request failures due to no available GPU).
• Geographic balance — is supply in the same regions as demand?
• Liquidity — for new entrants on either side, how long until they get utilization / get matched.

Billing & invoicing data

Billing data is the most-scrutinized data at an infra company. Finance audits it; customers dispute it; SOC2 wants tamper-evidence. Special considerations:

• Idempotency is non-negotiable — billing the same request twice gets you sued. Every billing event needs an idempotency key.
• Append-only history — never update a billing record in place. Append a correction event.
• Currency / FX — bill in customer's currency, account in USD, store rate at time-of-transaction.
• Pro-rating — partial months, plan changes mid-cycle.
• Credits and refunds — model these as separate event types with explicit references to the original charge.
• Latency tolerance — billing data must be eventually accurate, often within hours. Real-time isn't required, but reconciliation matters.

Multi-tenant analytics

Two distinct multi-tenant questions:

1. Customer-facing analytics (the data product)

If the platform shows each customer their own usage dashboard, you're effectively building an SaaS analytics product. Concerns:

• Row-level security / tenant isolation — customer A must never see customer B's data.
• Latency requirements — customer-facing dashboards expect sub-second response. Pre-aggregate + cache.
• Custom rollups — customers may want to slice by their own tags / API keys.
• Embedded BI — Looker, Cube, ClickHouse + a thin app — pick a serving pattern early.

2. Internal cross-tenant analytics (the data team's work)

Finance / product / ops need cross-tenant analyses: "top 10 customers by usage", "new-customer cohort retention by signup month." Concerns:

• Access controls — who internally can query PII vs aggregated?
• Audit logs — who queried what when.
• Tokenization where customer identities aren't needed.

Key metrics to be conversational about

Revenue / customer

• ARR / MRR — but careful, usage-based pricing makes ARR a derived projection.
• Revenue per customer per month — clear, used everywhere.
• NRR (Net Revenue Retention) — same-customer revenue growth, including churn.
• GRR (Gross Revenue Retention) — same-customer retention without expansion. Floors NRR.
• Concentration — top-N customers' share of revenue.

Cost / infrastructure

• COGS — cost of goods sold; for infra companies, mostly GPU costs.
• Gross margin — revenue minus COGS.
• GPU utilization rate — fleet-wide, by region, by GPU type.
• Cost per token / per request / per GPU-hour.

Reliability / ops

• p50 / p95 / p99 latency — by model, by region.
• TTFT (time to first token) — for streaming inference.
• Success rate / error rate — overall and per-error-class.
• GPU failure rate — per provider.
• Queue depth / wait time — when demand exceeds supply.

Customer behavior

• Activation — first meaningful request after signup.
• Time-to-first-value — signup to first successful production-volume usage.
• Cohort retention — % of signup cohort still active at month N.
• Model adoption — share of usage by model.
• Workload mix — chat vs batch vs streaming vs fine-tuning.

Interview talking points

✓ Check yourself

Answer from memory before opening each one.

1. A GPU rented at 2 dollars an hour serves 100 requests an hour in one scenario and 1,000 in another. What is the GPU cost per request in each, and what does it tell you about margin?

   Show answer

   Two cents per request, then two tenths of a cent. Utilization is the dominant lever on unit margin, since the cost of the hour is fixed while revenue scales with requests served. See unit economics.

2. A dashboard says fleet utilization is 85%, but finance says the fleet is losing money. Which definitions might explain the gap, and what do you ask?

   Show answer

   "Utilization" may mean compute use, memory use, time assigned to a customer, or revenue per GPU-hour. A GPU can be assigned all hour yet barely computing, so one number looks good while revenue per GPU-hour is poor. Ask which definition each side means. See GPU utilization.

3. You roll hourly GPU rows up to a daily number per region. What do you do with utilization_pct, and what with revenue_usd?

   Show answer

   Average utilization, weighted by the minutes each row covers, since a ratio cannot be summed across hours or GPUs. Sum revenue, which is additive. A dashboard that sums eight GPUs shows 560%. See the modeling layer.

4. A billing consumer replays a message partition after an outage and customers are charged twice. Which design rule was missing?

   Show answer

   An idempotency key on every billing event, so a replay is recognised and skipped. Billing records should also be append-only, with corrections as new events that reference the original. See billing and invoicing data.

5. Analysts query a 10-billion-row monthly inference table directly for customer revenue and the warehouse bill spikes. How do you redesign?

   Show answer

   Layer aggregates: five-minute for operations, hourly for cost analysis, daily for finance and retention. Keep raw logs partitioned by date in cheap storage for spot debugging with strict partition filters. Most queries then hit the aggregates. See inference logs.

6. Revenue per token on a dashboard disagrees with throughput per token by about ten percent. What is the likely cause, and what is the rule?

   Show answer

   The two use different counts: billable tokens, which can round or carry minimums, versus tokenizer tokens, which the model consumed. State which count each metric uses and never mix them. See the token-counting gotcha.

Takeaway

• Unit economics is the core problem: revenue per request from tokens minus GPU time and overhead, summed per customer, model and region.
• Utilization drives margin, and the word is overloaded: compute, memory, time and revenue utilization each answer a different question.
• An hourly periodic snapshot fact per GPU is the standard shape; ratios such as utilization are averaged, never summed.
• Inference logs are too large to serve raw, so layer five-minute, hourly and daily aggregates and keep raw logs partitioned in cheap storage.
• State whether a metric uses billable tokens or tokenizer tokens, and never mix the two.
• Billing data needs idempotency keys, append-only corrections, stored exchange rates and reconciliation.
• Customer-facing analytics needs tenant isolation and fast pre-aggregated reads; internal cross-tenant analytics needs access control and audit logs.

The next chapter drills all of this with practice interview questions. → Practice Interview Questions

Further reading

• PaLM: Scaling Language Modeling with Pathways — Chowdhery et al., 2022. Where model FLOPs utilization is defined and reported.
• Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023. Why serving throughput per GPU depends on memory management; the paper behind vLLM.
• Pricing — Anthropic docs, accessed 2026. A worked example of per-token pricing with separate input and output rates.
• Basic usage-based billing — Stripe documentation, accessed 2026. How metered usage becomes invoices.
• Idempotent requests — Stripe API reference, accessed 2026. The idempotency-key pattern for safe retries.