GPU Compute Domain
The preferred-qualification edge. Enough vocabulary about GPU marketplaces, inference economics, and supply/demand metrics to sound credible with PhD founders.
Why this chapter
The JD's preferred qualifications list "Experience in GPU compute infrastructure or cloud computing space" and "Background in AI infrastructure or related technical domains." Most candidates won't have this. The differentiator is not pretending to — it's knowing enough vocabulary to engage thoughtfully and ask good questions back. This chapter is that vocabulary stack.
The GPU cloud landscape
Three categories of provider:
- Hyperscalers — AWS (P-series, G-series), GCP (A2/A3, G2), Azure (NC-series). Expensive on-demand, deep enterprise sales, opinionated reservation pricing.
- Specialty GPU clouds — Lambda Labs, CoreWeave, Crusoe, Together.ai. Focused on AI workloads, often cheaper per GPU-hour than hyperscalers.
- Decentralized / marketplace — RunPod, Vast.ai, Akash. Aggregate GPU supply from many providers (including consumer-grade and excess capacity), serve developers at lower prices.
Open-access marketplaces play in the third category, plus offers managed inference (similar to Together.ai, Anyscale, Replicate).
GPU hardware tiers
The names that come up in conversation:
| GPU | Memory | Use case | Approx. on-demand price (hyperscaler) |
|---|---|---|---|
| H100 | 80 GB HBM3 | Training, large-model inference | $4–8/hr |
| H200 | 141 GB HBM3e | Larger context inference, training | $5–10/hr |
| A100 | 40 / 80 GB | Training, inference workhorse | $2–4/hr |
| L40 / L40S | 48 GB | Inference-optimized, mid-size models | $1–2/hr |
| A10 / A10G | 24 GB | Smaller inference, dev workloads | $1/hr |
| RTX 4090 / 3090 | 24 GB | Consumer / hobbyist inference, fine-tuning | $0.20–0.50/hr (marketplace) |
Marketplace prices are typically 30–70% off hyperscaler list. The company's pitch: "much lower than hyperscalers."
The vocabulary worth knowing
- HBM / HBM3 / HBM3e — High-Bandwidth Memory. The GPU memory tier; matters for whether a model fits and how fast it runs.
- NVLink / NVSwitch — fast GPU-to-GPU interconnect. Required for multi-GPU model training.
- Tensor cores — specialized matmul units. Modern AI runs on these.
- FLOPs — floating-point operations per second. Marketing number; less meaningful than memory bandwidth in practice for inference.
- FP16 / BF16 / FP8 — reduced-precision formats that trade accuracy for speed/memory. Inference often runs in BF16 or FP8.
Two-sided marketplace economics
This is a two-sided marketplace:
- Supply side: GPU providers list their machines. Could be datacenter operators, smaller clouds, or even consumer GPU owners.
- Demand side: AI developers rent GPU-hours at marketplace prices.
- The marketplace: matches supply to demand, takes a margin.
The classic marketplace metrics
- Take rate: the marketplace's revenue ÷ gross merchandise value (total dollars flowing through). 10–30% is common in marketplaces.
- Utilization: fraction of available GPU-hours actually rented. Below 50% means too much supply / not enough demand at current price.
- Liquidity: how quickly demand finds matching supply. "Time to fulfill" is a proxy.
- Cohort retention: both sides. Supplier retention says the economics work for providers; demand retention says the product is good for developers.
- Cross-side network effects: more suppliers → more variety + lower prices → more demand → more revenue per supplier → more suppliers. The flywheel.
The hard analytical questions
- How do you set price? Auction? Fixed? Per-tier?
- How do you handle elastic supply (suppliers come and go)?
- What's the right level of GPU-tier granularity for the product UX vs the price model?
- How do you measure marketplace health when supply and demand are growing simultaneously (you can't see liquidity drag because raw volume hides it)?
Metric vocabulary
What to track at a GPU marketplace, by function:
Growth
- New customer signups per week
- New supplier signups per week
- First-session conversion (signup → first GPU-hour)
- Activation rate (signup → meaningful usage threshold)
Engagement / revenue
- GPU-hours billed per week, split by tier
- Revenue per customer per week (and per cohort)
- Per-customer retention curves
- Whale concentration: % of revenue from top 1% / 5% / 10% of customers
Marketplace health
- Utilization by tier (% of listed GPU-hours actually rented)
- Median fulfillment time per tier (how long until a request finds matching supply)
- Price spread by tier (high spread = thin liquidity)
- Supplier retention curves
Unit economics
- Gross margin per GPU-hour by tier (revenue − cost-of-supply)
- CAC per customer (acquisition cost) vs LTV (lifetime value)
- Customer payback period
Inference economics
Marketplaces in this space often offer managed inference on top of the marketplace. The economics differ from raw GPU rental:
- Per-token pricing. Inference is usually priced per input/output token, like Anthropic/OpenAI. Customers don't see GPUs at all.
- Batching. Inference servers batch multiple requests onto one GPU pass to improve utilization. A well-batched H100 can serve 100+ concurrent inference requests for a small model.
- Latency vs throughput. Bigger batches = higher throughput but higher latency. Customers care about both; pricing tiers often vary by SLA.
- Cold start. Loading a model into GPU memory takes 5–60 seconds. For rarely-used models, the first request is slow; for popular models, the model stays loaded.
- Spot vs reserved supply. Inference-optimal supply is reserved (predictable cost). Spot supply is cheap but can vanish; less ideal for serving SLAs.
The analytical questions for inference
- What's our cost per million tokens, by model? How does it vary with batch size?
- Which models drive most revenue? Which lose money?
- What's our p95 / p99 latency? How does that correlate with churn?
- How does our pricing compare to Together.ai, Anyscale, Replicate per million tokens?
Supply-side considerations
Often overlooked by candidates. A marketplace must keep suppliers happy or the marketplace starves.
- Supplier earnings stability: are providers getting enough utilization to justify staying?
- Supplier churn: same metric, inverted. Hard to model when supply is dynamic.
- Tier mix: are we over-supplied in low-end GPUs while H100 demand is unmet? Mix optimization is real product work.
- Geographic distribution: latency-sensitive customers need supply near them.
- Quality signals: which suppliers have high uptime, low latency, no DOA hardware? Trust metrics matter for routing.
Interview probes
Show probe 1: "What's the most important metric for a GPU marketplace?"
Trick question — there is no single one. But the answer that reads well: "Utilization by tier, weighted by tier revenue. Low utilization means either too much supply or under-pricing; high utilization means we're leaving demand on the table. It's the metric that flags both supply and demand health in one number — and you'd pair it with fulfillment time as the leading indicator."
Show probe 2: "Why is H100 supply different from A100 supply analytically?"
H100s are scarcer and more expensive — fewer providers can list them, and the demand-side is much more price-sensitive at the higher tier. The analytical implications: H100 utilization should run higher than A100 (less slack); H100 churn (suppliers leaving) is more material because each supplier represents more capacity; price elasticity is different. The metric system that treats H100 and A100 identically misses where the levers are.
Show probe 3: "Walk me through how you'd model GPU cohort retention."
Two sides. (1) Demand-side cohort: customers grouped by signup week, measured by GPU-hours or revenue per subsequent week. Standard cohort curve. (2) Supply-side cohort: providers grouped by listing week, measured by GPU-hours rented or earnings per subsequent week. Compare retention curves across tiers — patterns differ a lot between H100 providers (sticky, high earnings) and consumer-grade providers (transient).
Show probe 4: "What's per-token pricing's biggest analytical wrinkle?"
Input tokens and output tokens cost differently because output requires sequential generation while input is parallelized. Most providers charge 3–5× more per output token. The analytical wrinkle: a customer's "tokens used" is meaningless without the input/output split. The metric you actually care about is cost per request weighted by typical input/output ratio for their workload.
Show probe 5: "What's batching, and why does it matter?"
Inference servers process multiple concurrent requests in one GPU forward pass — the GPU utilizes parallelism much better with batches. A batch size of 16 vs 1 can mean 5–10× higher throughput per GPU at the cost of slightly higher per-request latency. Analytical implications: cost per request drops sharply with utilization, so two customers running at low concurrency are much more expensive than one running at high. Pricing should reflect that.