Section B · DE using neoclouds

Cost Engineering

Compute cost is the dominant line item for serious AI workloads. The DE function owns the pipelines that make cost visible, attributable, and actionable. Without these pipelines, leadership can't reason about ROI and engineers can't reason about choices.

Why this is DE work

Cost engineering for GPU compute is data engineering because the inputs (billing exports, usage metrics, allocation tags) live in different systems with different schemas and need to be joined, modeled, and exposed via warehouses and dashboards. Finance teams can't do this work; engineering teams need the output to make decisions.

At smaller AI companies, the DE function builds the cost pipelines as a side project. At larger ones, a FinOps or platform-analytics team owns it. Either way, the work is DE in nature.

Tagging discipline

Cost attribution requires tagging compute jobs with metadata that lets you slice spend later. Common tags:

Team / cost center. Who owns the bill.
Project / experiment. What the work was for.
Job type. Training, fine-tuning, inference, evaluation, exploration.
Workload class. Production vs research.
Provider region. Where the GPUs lived.
SKU. What hardware was used.
Run / experiment ID. Unique identifier for downstream joining.

Tags must be enforced at job submission time. The orchestrator (or job-submission layer) refuses jobs missing required tags. Untagged compute is the original sin of FinOps.

Allocation pipelines

The allocation pipeline takes billing data plus tag data and produces cost-per-thing tables. Stages:

Ingest billing exports. Each provider has different billing data formats. Normalize.
Join with job metadata. Match billing line items to job tags using job IDs.
Resolve unallocated. Some line items won't match a tag — storage overhead, idle instances, etc. Attribute to a default bucket.
Build cost-per-X tables. Cost per team, cost per project, cost per experiment, cost per GB of training data.
Publish to BI. Dashboards that engineers and leadership can self-serve.

This pipeline runs daily. Cost data has 1-2 day lag because providers' billing exports aren't real-time.

Unit cost analysis

Beyond aggregated spend, leadership wants unit cost — cost per training run, cost per million inference tokens, cost per customer.

Unit cost analysis is hard because:

Fixed costs (reserved capacity, cluster bring-up) have to be allocated to specific work.
Some workloads are joint (a shared inference cluster serves multiple customers).
Different unit metrics (queries, tokens, training steps) need different unit-cost calculations.

The DE work here is building the data model that supports these calculations and the SQL or transformation logic that materializes the unit-cost marts. Once available, product / finance / leadership can answer questions they couldn't before.

Budget guardrails

Even good cost reporting is reactive. Guardrails prevent overspend before it happens:

Budget alerts. When team / project spend exceeds threshold, alert.
Auto-pause runaway jobs. Jobs that exceed expected compute by threshold get killed or paused for review.
Quota at job submission. Hard limits on concurrent jobs per team.
Cost preview at submission. The job-submission layer shows expected cost before launching.
Approval workflows. Above-threshold jobs require approval.

Guardrails are a real engineering investment. The DE who builds them protects the company from the next "we accidentally ran 1000 A100s for 48 hours" incident.

Optimization patterns

Once visibility exists, optimization opportunities surface:

Right-sizing. Jobs that don't need an H100 should run on cheaper hardware. The data tells you which.
Spot adoption. Resumable workloads should run on spot. The data shows you what's not yet on spot.
Reservation matching. Reserved capacity is wasted if utilization is below commitment. The data shows the gap.
Idle teardown. Long-running interactive instances accumulate cost. Auto-teardown after inactivity.
Cross-provider arbitrage. Same workload at different providers at different prices. Move workloads.
Model / data version dedup. Multiple teams training near-duplicates. Consolidate.
Storage tiering. Old training data on cold storage; hot only for active work.

The DE function delivers each of these as a "here's the data; here's the recommendation" report. The engineering and ML teams execute.

✓ Check yourself

Answer from memory before opening each one.

The month's billing export has $48,000 of GPU line items, about 15 percent of spend, that match no job ID. Where do you look, and what do you change?

Show answer
Look for untagged jobs, idle instances and storage overhead, and check where the submission path lets jobs through without tags. Enforce required tags at submission so the orchestrator refuses untagged work, and keep the unallocated bucket small and visible. See tagging and allocation.
Why can the dashboard not show yesterday's spend, and what does that mean for your guardrails?

Show answer
Provider billing exports lag by one to two days. Guardrails that depend on billing data act too late, so enforce quotas and cost previews at submission and pause runaway jobs from your own usage metrics. See allocation pipelines and guardrails.
One reserved inference cluster at a flat monthly fee serves three customers. Finance wants cost per million tokens per customer. How do you build it?

Show answer
Pick an allocation driver, such as metered tokens or GPU-seconds per customer, and spread the fixed cost by each customer's share. Decide and document where idle capacity goes, because that choice moves each customer's unit cost. See unit cost analysis.
You reserved 100 GPUs and the data shows 60 percent average utilization for the month. What does it say, and what do you do?

Show answer
Forty GPUs' worth of the commitment is paid for and idle. Move work onto the reservation that currently runs on-demand elsewhere, such as research jobs, and size the next renewal to the observed floor. See optimization patterns.
Should a budget breach block new jobs or only alert? Make the call.

Show answer
There is no universal answer. Hard blocks stop runaway spend but can halt a critical run. A workable split is an alert at a threshold, approval above a larger one, and auto-pause only for non-production classes. The workload class tag decides which rule applies. See budget guardrails.

Takeaway

Cost engineering is data engineering because billing exports, usage metrics and tags live in different systems and must be joined.
Tags are enforced at job submission, since untagged compute cannot be attributed afterward.
The allocation pipeline normalizes provider billing, joins on job ID, resolves unallocated lines into a visible default bucket, and publishes cost-per-X tables daily.
Unit cost needs a data model that allocates fixed and shared costs by a chosen driver.
Guardrails act before the bill arrives, because billing data lags by a day or two.
Right-sizing, spot adoption, reservation matching and idle teardown all start from the same visibility.

The next section flips the view to the provider's side, where the same data is built from telemetry. → Fleet Telemetry