Section D · Domain

GPU Compute Domain

The preferred-qualification edge. Enough vocabulary about GPU marketplaces, inference economics, and supply/demand metrics to sound credible with PhD founders.

Why this chapter

The JD's preferred qualifications list "Experience in GPU compute infrastructure or cloud computing space" and "Background in AI infrastructure or related technical domains." Most candidates won't have this. The differentiator is not pretending to — it's knowing enough vocabulary to engage thoughtfully and ask good questions back. This chapter is that vocabulary stack.

The GPU cloud landscape

Three categories of provider:

Hyperscalers — AWS (P-series, G-series), GCP (A2/A3, G2), Azure (NC-series). Expensive on-demand, deep enterprise sales, opinionated reservation pricing.
Specialty GPU clouds — Lambda Labs, CoreWeave, Crusoe, Together.ai. Focused on AI workloads, often cheaper per GPU-hour than hyperscalers.
Decentralized / marketplace — RunPod, Vast.ai, Akash. Aggregate GPU supply from many providers (including consumer-grade and excess capacity), serve developers at lower prices.

Open-access marketplaces play in the third category, plus offers managed inference (similar to Together.ai, Anyscale, Replicate).

GPU hardware tiers

The names that come up in conversation:

GPU	Memory	Use case	Approx. on-demand price (hyperscaler)
H100	80 GB HBM3	Training, large-model inference	$4–8/hr
H200	141 GB HBM3e	Larger context inference, training	$5–10/hr
A100	40 / 80 GB	Training, inference workhorse	$2–4/hr
L40 / L40S	48 GB	Inference-optimized, mid-size models	$1–2/hr
A10 / A10G	24 GB	Smaller inference, dev workloads	$1/hr
RTX 4090 / 3090	24 GB	Consumer / hobbyist inference, fine-tuning	$0.20–0.50/hr (marketplace)

Marketplace prices are typically 30–70% off hyperscaler list. The company's pitch: "much lower than hyperscalers."

The vocabulary worth knowing

HBM / HBM3 / HBM3e — High-Bandwidth Memory. The GPU memory tier; matters for whether a model fits and how fast it runs.
NVLink / NVSwitch — fast GPU-to-GPU interconnect. Required for multi-GPU model training.
Tensor cores — specialized matmul units. Modern AI runs on these.
FLOPs — floating-point operations per second. Marketing number; less meaningful than memory bandwidth in practice for inference.
FP16 / BF16 / FP8 — reduced-precision formats that trade accuracy for speed/memory. Inference often runs in BF16 or FP8.

Two-sided marketplace economics

This is a two-sided marketplace:

Supply side: GPU providers list their machines. Could be datacenter operators, smaller clouds, or even consumer GPU owners.
Demand side: AI developers rent GPU-hours at marketplace prices.
The marketplace: matches supply to demand, takes a margin.

The classic marketplace metrics

Take rate: the marketplace's revenue ÷ gross merchandise value (total dollars flowing through). 10–30% is common in marketplaces.
Utilization: fraction of available GPU-hours actually rented. Below 50% means too much supply / not enough demand at current price.
Liquidity: how quickly demand finds matching supply. "Time to fulfill" is a proxy.
Cohort retention: both sides. Supplier retention says the economics work for providers; demand retention says the product is good for developers.
Cross-side network effects: more suppliers → more variety + lower prices → more demand → more revenue per supplier → more suppliers. The flywheel.

The hard analytical questions

How do you set price? Auction? Fixed? Per-tier?
How do you handle elastic supply (suppliers come and go)?
What's the right level of GPU-tier granularity for the product UX vs the price model?
How do you measure marketplace health when supply and demand are growing simultaneously (you can't see liquidity drag because raw volume hides it)?

Metric vocabulary

What to track at a GPU marketplace, by function:

Growth

New customer signups per week
New supplier signups per week
First-session conversion (signup → first GPU-hour)
Activation rate (signup → meaningful usage threshold)

Engagement / revenue

GPU-hours billed per week, split by tier
Revenue per customer per week (and per cohort)
Per-customer retention curves
Whale concentration: % of revenue from top 1% / 5% / 10% of customers

Marketplace health

Utilization by tier (% of listed GPU-hours actually rented)
Median fulfillment time per tier (how long until a request finds matching supply)
Price spread by tier (high spread = thin liquidity)
Supplier retention curves

Unit economics

Gross margin per GPU-hour by tier (revenue − cost-of-supply)
CAC per customer (acquisition cost) vs LTV (lifetime value)
Customer payback period

Inference economics

Marketplaces in this space often offer managed inference on top of the marketplace. The economics differ from raw GPU rental:

Per-token pricing. Inference is usually priced per input/output token, like Anthropic/OpenAI. Customers don't see GPUs at all.
Batching. Inference servers batch multiple requests onto one GPU pass to improve utilization. A well-batched H100 can serve 100+ concurrent inference requests for a small model.
Latency vs throughput. Bigger batches = higher throughput but higher latency. Customers care about both; pricing tiers often vary by SLA.
Cold start. Loading a model into GPU memory takes 5–60 seconds. For rarely-used models, the first request is slow; for popular models, the model stays loaded.
Spot vs reserved supply. Inference-optimal supply is reserved (predictable cost). Spot supply is cheap but can vanish; less ideal for serving SLAs.

The analytical questions for inference

What's our cost per million tokens, by model? How does it vary with batch size?
Which models drive most revenue? Which lose money?
What's our p95 / p99 latency? How does that correlate with churn?
How does our pricing compare to Together.ai, Anyscale, Replicate per million tokens?

Supply-side considerations

Often overlooked by candidates. A marketplace must keep suppliers happy or the marketplace starves.

Supplier earnings stability: are providers getting enough utilization to justify staying?
Supplier churn: same metric, inverted. Hard to model when supply is dynamic.
Tier mix: are we over-supplied in low-end GPUs while H100 demand is unmet? Mix optimization is real product work.
Geographic distribution: latency-sensitive customers need supply near them.
Quality signals: which suppliers have high uptime, low latency, no DOA hardware? Trust metrics matter for routing.

Interview probes

Show probe 1: "What's the most important metric for a GPU marketplace?"

Trick question — there is no single one. But the answer that reads well: "Utilization by tier, weighted by tier revenue. Low utilization means either too much supply or under-pricing; high utilization means we're leaving demand on the table. It's the metric that flags both supply and demand health in one number — and you'd pair it with fulfillment time as the leading indicator."

Show probe 2: "Why is H100 supply different from A100 supply analytically?"

H100s are scarcer and more expensive — fewer providers can list them, and the demand-side is much more price-sensitive at the higher tier. The analytical implications: H100 utilization should run higher than A100 (less slack); H100 churn (suppliers leaving) is more material because each supplier represents more capacity; price elasticity is different. The metric system that treats H100 and A100 identically misses where the levers are.

Show probe 3: "Walk me through how you'd model GPU cohort retention."

Two sides. (1) Demand-side cohort: customers grouped by signup week, measured by GPU-hours or revenue per subsequent week. Standard cohort curve. (2) Supply-side cohort: providers grouped by listing week, measured by GPU-hours rented or earnings per subsequent week. Compare retention curves across tiers — patterns differ a lot between H100 providers (sticky, high earnings) and consumer-grade providers (transient).

Show probe 4: "What's per-token pricing's biggest analytical wrinkle?"

Input tokens and output tokens cost differently because output requires sequential generation while input is parallelized. Most providers charge 3–5× more per output token. The analytical wrinkle: a customer's "tokens used" is meaningless without the input/output split. The metric you actually care about is cost per request weighted by typical input/output ratio for their workload.

Show probe 5: "What's batching, and why does it matter?"

Inference servers process multiple concurrent requests in one GPU forward pass — the GPU utilizes parallelism much better with batches. A batch size of 16 vs 1 can mean 5–10× higher throughput per GPU at the cost of slightly higher per-request latency. Analytical implications: cost per request drops sharply with utilization, so two customers running at low concurrency are much more expensive than one running at high. Pricing should reflect that.

✓ Check yourself

Answer from memory before opening each one.

Utilization is 38% on H100s and 71% on RTX 4090s. What does each number suggest, and what do you look at next?

Show answer
Low H100 utilization suggests too much supply or a price above what demand accepts. Higher 4090 utilization suggests the tier is closer to balanced or short of supply. Next look at fulfillment time and price spread by tier, plus supplier counts and earnings, before changing price. See marketplace economics.
One customer sends 9M input and 1M output tokens, and another sends 1M input and 9M output. Output costs 5 times input. Compare their costs, and say what this does to a metric called tokens used.

Show answer
In input-price units the first costs 9 plus 5, which is 14, and the second costs 1 plus 45, which is 46. Total tokens are the same, so tokens used hides a threefold cost gap. Track the input and output split. See inference economics.
Why does cost per request fall as concurrency rises on an inference server, and what does the customer give up?

Show answer
A GPU forward pass handles several requests at once, so the fixed cost of loading weights is shared. Larger batches raise throughput but increase per-request latency. Customers who need tight latency pay for lower batching. See inference economics.
Supply and demand are both growing 30% a month and a board slide shows total GPU-hours rising. Why is that weak evidence of marketplace health?

Show answer
Growth on both sides can hide poor matching, since raw volume rises while fulfillment time lengthens or the price spread widens. Report utilization and fulfillment time by tier, and retention on both sides. See marketplace economics.
Your metric system treats H100 and A100 identically. What does it miss?

Show answer
H100s are scarcer and more expensive, so supplier churn is more material, utilization should run higher and price sensitivity differs. A blended metric averages away those differences. Slice metrics by tier. See interview probes.

Takeaway

There are three kinds of GPU provider: hyperscalers, specialty GPU clouds and marketplaces, and the company sits in the third with managed inference on top.
The core marketplace metrics are take rate, utilization, liquidity and retention on both sides.
Utilization by tier is the closest single health number, but it needs fulfillment time beside it.
Inference is priced per token, and input and output tokens are priced differently, so a total token count says little about cost.
Batching trades per-request latency for throughput, so cost per request depends on concurrency.
Supply-side health, such as supplier retention, tier mix and trust signals, matters as much as demand.

With the vocabulary in place, the next chapter is a drill across all of it. → Practice Interview Questions