Section B · DE using neoclouds

Cost Engineering

Compute cost is the dominant line item for serious AI workloads. The DE function owns the pipelines that make cost visible, attributable, and actionable. Without these pipelines, leadership can't reason about ROI and engineers can't reason about choices.

Why this is DE work

Cost engineering for GPU compute is data engineering because the inputs (billing exports, usage metrics, allocation tags) live in different systems with different schemas and need to be joined, modeled, and exposed via warehouses and dashboards. Finance teams can't do this work; engineering teams need the output to make decisions.

At smaller AI companies, the DE function builds the cost pipelines as a side project. At larger ones, a FinOps or platform-analytics team owns it. Either way, the work is DE in nature.

Tagging discipline

Cost attribution requires tagging compute jobs with metadata that lets you slice spend later. Common tags:

  • Team / cost center. Who owns the bill.
  • Project / experiment. What the work was for.
  • Job type. Training, fine-tuning, inference, evaluation, exploration.
  • Workload class. Production vs research.
  • Provider region. Where the GPUs lived.
  • SKU. What hardware was used.
  • Run / experiment ID. Unique identifier for downstream joining.

Tags must be enforced at job submission time. The orchestrator (or job-submission layer) refuses jobs missing required tags. Untagged compute is the original sin of FinOps.

Allocation pipelines

The allocation pipeline takes billing data plus tag data and produces cost-per-thing tables. Stages:

  1. Ingest billing exports. Each provider has different billing data formats. Normalize.
  2. Join with job metadata. Match billing line items to job tags using job IDs.
  3. Resolve unallocated. Some line items won't match a tag — storage overhead, idle instances, etc. Attribute to a default bucket.
  4. Build cost-per-X tables. Cost per team, cost per project, cost per experiment, cost per GB of training data.
  5. Publish to BI. Dashboards that engineers and leadership can self-serve.

This pipeline runs daily. Cost data has 1-2 day lag because providers' billing exports aren't real-time.

Unit cost analysis

Beyond aggregated spend, leadership wants unit cost — cost per training run, cost per million inference tokens, cost per customer.

Unit cost analysis is hard because:

  • Fixed costs (reserved capacity, cluster bring-up) have to be allocated to specific work.
  • Some workloads are joint (a shared inference cluster serves multiple customers).
  • Different unit metrics (queries, tokens, training steps) need different unit-cost calculations.

The DE work here is building the data model that supports these calculations and the SQL or transformation logic that materializes the unit-cost marts. Once available, product / finance / leadership can answer questions they couldn't before.

Budget guardrails

Even good cost reporting is reactive. Guardrails prevent overspend before it happens:

  • Budget alerts. When team / project spend exceeds threshold, alert.
  • Auto-pause runaway jobs. Jobs that exceed expected compute by threshold get killed or paused for review.
  • Quota at job submission. Hard limits on concurrent jobs per team.
  • Cost preview at submission. The job-submission layer shows expected cost before launching.
  • Approval workflows. Above-threshold jobs require approval.

Guardrails are a real engineering investment. The DE who builds them protects the company from the next "we accidentally ran 1000 A100s for 48 hours" incident.

Optimization patterns

Once visibility exists, optimization opportunities surface:

  • Right-sizing. Jobs that don't need an H100 should run on cheaper hardware. The data tells you which.
  • Spot adoption. Resumable workloads should run on spot. The data shows you what's not yet on spot.
  • Reservation matching. Reserved capacity is wasted if utilization is below commitment. The data shows the gap.
  • Idle teardown. Long-running interactive instances accumulate cost. Auto-teardown after inactivity.
  • Cross-provider arbitrage. Same workload at different providers at different prices. Move workloads.
  • Model / data version dedup. Multiple teams training near-duplicates. Consolidate.
  • Storage tiering. Old training data on cold storage; hot only for active work.

The DE function delivers each of these as a "here's the data; here's the recommendation" report. The engineering and ML teams execute.

Takeaway

Cost engineering is the highest-impact DE work in an AI organization, full stop. The pipelines that make compute visible, attributable, and controllable typically pay for themselves dozens of times over. The next section flips the perspective to data engineering inside a neocloud company.