Deep Dive — GPU Scheduling & Orchestration

The scheduling problem, in one paragraph

You have a heterogeneous fleet — H100s, A100s, maybe L40s, with different memory sizes and topology. You have multiple tenants — research training, online inference, batch evaluation, dev/notebooks. Their workloads have different shapes: long jobs with strict topology requirements (training), short jobs with strict latency requirements (online inference), bursty jobs with elastic needs (batch). Fair use across teams matters; so does maximizing utilization; so does respecting priority for the production path. The "scheduler" you build is the answer to "who gets which GPU, when, and what happens when the answer changes."

Kubernetes device plugin

K8s itself only knows about CPU, memory, and ephemeral storage. The NVIDIA device plugin (a DaemonSet) introduces nvidia.com/gpu as a schedulable resource. The flow:

1. Plugin starts on each GPU node, discovers GPUs via NVML.
2. Registers with kubelet over a Unix socket.
3. Kubelet advertises N GPUs as allocatable to the API server.
4. Scheduler sees pod request nvidia.com/gpu: 4, finds a node with ≥4 free, binds.
5. Kubelet calls plugin's Allocate(), which returns env vars and device mounts.
6. Container starts with NVIDIA_VISIBLE_DEVICES set; toolkit hook bind-mounts devices.

The plugin is also where you configure sharing modes: exclusive, time-slicing, MPS, or MIG.

# NVIDIA device plugin config — time-slicing example
version: v1
flags:
  migStrategy: "single"        # or "mixed" or "none"
sharing:
  timeSlicing:
    resources:
      - name: nvidia.com/gpu
        replicas: 4            # 4 virtual GPUs share each physical

The NVIDIA GPU Operator ships the device plugin together with driver containers, container toolkit, DCGM exporter, and MIG manager — strongly recommended over rolling your own.

MIG — Multi-Instance GPU

MIG (A100/H100/B100) partitions a single GPU into up to 7 hardware-isolated instances. Each instance gets dedicated SM partitions, L2 slices, and memory. This is real isolation — separate fault domains, separate memory bandwidth, no noisy neighbor.

Tradeoffs:

• Hardware isolation — predictable performance under multi-tenancy. Great for shared inference platforms.
• Static partitioning — you choose profiles at boot; changing requires a workload drain. Mixed-profile nodes complicate scheduling.
• No NVLink between MIG instances — multi-MIG-instance models lose collectives. MIG is a small-job platform.
• No P2P beyond the GPU: driver R570 allows it only between MIG instances on the same GPU. MPS does run on top of MIG, with the 48-client limit scaled down to the compute-instance size.

MPS — Multi-Process Service

MPS lets multiple CUDA processes share a single GPU's SMs concurrently via a server proxy. Unlike default context-switching (which serializes work), MPS interleaves kernels from different processes on the SMs.

• Better utilization when each tenant is too small to saturate the GPU.
• Soft isolation only — one process's OOM can affect neighbors; one process's bug can corrupt others (less so with Volta+ MPS, but still weaker than MIG).
• Per-process SM percentage caps available; per-process memory caps available on Volta+.
• Latency improves for many concurrent small workloads compared to default time-slicing.

MPS is the right choice when you have many small co-trusted clients (e.g. several inference replicas sharing a node) and you've accepted the weaker isolation in exchange for better utilization.

The three sharing modes, side by side

Gang scheduling

Distributed training needs all N pods to start at the same time. If only 6 of 8 pods land and the other 2 are pending, the 6 idle GPUs are useless and may block someone else. Vanilla Kubernetes will happily start the 6.

Solutions:

• Volcano — CNCF project (Apache 2.0 licence), full batch scheduler for K8s with gang scheduling, queues, fair share. Common in production AI fleets.
• Kueue — Kubernetes-native job queueing built by SIG-Scheduling. Newer, simpler, increasingly the default.
• Yunikorn — Apache, started at Lyft / Cloudera, hierarchical queues.
• PodGroup (Volcano) / Workload (Kueue) — the CRDs that express "schedule all N or none."

# Kueue: a ClusterQueue with cohort-level borrowing
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-h100
spec:
  cohort: gpu-fleet
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: h100-sxm
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 32
              borrowingLimit: 16   # can borrow from cohort up to 16 more
            - name: cpu
              nominalQuota: 384
            - name: memory
              nominalQuota: 4Ti

Priority & preemption

Production inference must beat researcher fine-tuning when capacity is tight. K8s gives you PriorityClass:

• inference-online — highest priority, can preempt others.
• training-prod — high but below online.
• research-batch — low, preemptible.
• opportunistic — negative priority, runs only if nothing else wants the resource.

The interesting part is the preemption policy:

• Graceful preemption — send SIGTERM, give a budget (e.g. 60s) to checkpoint, then SIGKILL.
• Hard preemption — kill now. Used when the system is in trouble.
• Preemption avoidance — only preempt if no other node fits; bin-packing-aware.

Quotas & fair share

The company has teams; teams have budgets. The scheduler enforces them.

• ResourceQuota per namespace — hard caps on requests.nvidia.com/gpu.
• LimitRange — per-pod defaults and maxes.
• Hierarchical queues (Volcano, Yunikorn) — "research" gets 60%, "product" gets 40%, with borrowing when one is idle.
• Fair-share / DRF (Dominant Resource Fairness) — when both demand more than their share, allocate proportionally.
• Backfill — small jobs slip into gaps left by large gang jobs waiting for placement.

Quota is also a cost mechanism: a team that consistently uses 100% of its quota is the signal that you should resize or renegotiate.

Node configuration that matters

The "boring" host config is where many GPU perf bugs hide.

• NUMA pinning — pin pod CPUs to the NUMA node where the GPUs physically attach. K8s topology manager with single-numa-node policy.
• CPU pinning & isolation — exclusive CPU sets via the cpu manager. Inference latency is sensitive to scheduler jitter.
• Hugepages — enable 2 MiB or 1 GiB hugepages for RDMA-heavy paths.
• IRQ affinity — NIC interrupts on the right NUMA node.
• CPU governor — performance, not powersave. (Yes, this is a real production gotcha.)
• Persistence mode — nvidia-smi -pm 1 or run nvidia-persistenced so the driver doesn't unload between processes.
• Fabric Manager — required for NVSwitch nodes (DGX-class, HGX).
• ECC settings, GPU clocks, application clocks — usually leave at default; document if you don't.

# kubelet config: NUMA-aware GPU placement
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
topologyManagerPolicy: single-numa-node
reservedSystemCPUs: "0,1,48,49"
featureGates:
  CPUManager: true
  TopologyManager: true

Topology-aware scheduling

A pod requesting 4 GPUs on an 8-GPU H100 node should get 4 GPUs connected by NVLink as a single NVSwitch group, not a random 4 of 8. The NVIDIA device plugin supports topology-aware allocation:

• Best-effort mode prefers same-NVLink-domain GPUs but won't fail if unavailable.
• Strict mode refuses to allocate if topology can't be satisfied.

Across nodes: place gang-scheduled pods on nodes connected to the same rail of the IB fabric. Tools like Kueue's flavor fungibility, kube-scheduler-extender, or vendor schedulers (Run:AI, etc.) handle this.

Workload isolation — what guarantees can you actually make?

Be precise here in interviews. Different mechanisms give different guarantees.

Queue depth & saturation signals

How do you know your scheduler is failing? Watch:

• Pending pod count + age per priority class. Online inference pending > 0 is an immediate page.
• Queue depth per Kueue/Volcano queue.
• Time-to-bind p95 for new pods.
• GPU allocatable vs available vs SM-active — three different things; the gap between them is where waste hides.
• Preemption rate — sudden spikes mean priority is being abused or capacity has shifted.
• Fragmentation — number of nodes with idle GPUs that no pending pod can use (wrong topology, wrong taints).

The dashboard you build for this is half the role's day-to-day. See chapter 12 for the telemetry pipeline.

✓ Check yourself

Answer from memory before opening each one.

1. A pod asks for four GPUs. Trace what happens from the request until the container sees its devices.

   Show answer

   The device plugin discovers GPUs through NVML and registers with the kubelet, which advertises them as allocatable. The scheduler binds the pod to a node with enough free GPUs. The kubelet calls Allocate on the plugin, which returns environment variables and device mounts, and the toolkit hook mounts the devices at container start. See the device plugin.

2. An eight-pod training job starts with six pods running and two pending. What is wrong, and what does it cost?

   Show answer

   The default scheduler places pods one at a time, so a partial start is legal. Six GPUs sit idle waiting for the rest and may block other work, and the job makes no progress. Gang scheduling through Kueue, Volcano or YuniKorn admits the job only when all pods can be placed. See gang scheduling.

3. Sixty small inference services, each under 10 GB with a p99 target, will share GPUs. Do you choose MIG or MPS?

   Show answer

   MIG when the p99 targets must hold, because instances have separate SMs, memory and bandwidth, so a neighbour cannot disturb them. MPS gives better utilisation but only soft isolation, which suits co-trusted replicas of one service. The cost of MIG is static partitions that need a drain to change. See MIG and the sharing modes.

4. A five-second inference request would preempt a six-hour fine-tuning run that is two minutes from finishing. What do you design?

   Show answer

   Combine priority with graceful preemption: send SIGTERM with a time budget to checkpoint, add a cooldown, and let the short request queue briefly rather than kill work at once. Preemption that ignores checkpoints and queue depth wastes far more GPU time than a short wait. See priority and preemption.

5. A tenant asks whether another tenant's job can ever slow theirs down. What do you promise under namespaces, MPS, MIG and exclusive allocation?

   Show answer

   Namespaces isolate the API surface and secrets but nothing on the GPU. MPS isolates address spaces but not performance. MIG isolates SMs, memory and bandwidth. Exclusive allocation isolates the device but not host CPU, NIC or NUMA noise. Ask what level the workload needs, then promise only that. See workload isolation.

6. A dashboard shows 64 GPUs allocatable, 60 allocated and average SM-active of 25 percent. What does each number say, and where is the waste?

   Show answer

   Allocated against allocatable says the scheduler is nearly full, so quotas look healthy. SM-active says the GPUs are mostly idle, so waste sits between allocation and use, in oversized requests and idle holders. Add fragmentation, meaning idle GPUs no pending pod can use. See queue depth and saturation signals.

Takeaway

• Scheduling answers who gets which GPU, when, and what happens when that answer changes.
• Kubernetes knows nothing about GPUs until the device plugin advertises them, and the GPU Operator is the supported way to ship plugin, driver and exporter together.
• MIG gives hardware isolation for many small SLO-bound workloads, MPS gives soft isolation for co-trusted replicas, and time-slicing gives sharing without isolation.
• Distributed training needs gang scheduling, because a partial start wastes GPUs and blocks others.
• Priority needs checkpointing, graceful preemption and cooldowns behind it, or preemption destroys more work than it protects.
• Quotas are a cost mechanism as well as a fairness one, and a team pinned at 100 percent of quota is a sizing signal.
• Promise only the isolation each mechanism provides, and track pending age, time-to-bind, preemption rate and fragmentation to see the scheduler failing.

Next the guide applies these pieces to concrete patterns you can describe at a whiteboard. → Applied Patterns

Further reading

• Device plugins — Kubernetes docs. The registration and Allocate flow that exposes GPUs to the scheduler.
• MIG User Guide — NVIDIA docs. Profiles, partitioning rules and what MIG does and does not isolate.
• Multi-Process Service — NVIDIA docs. How MPS shares a GPU between processes and the limits of its isolation.
• Kueue concepts — Kubernetes SIG Scheduling. ClusterQueues, cohorts, borrowing and preemption for batch admission.
• Volcano documentation — Volcano project. Gang scheduling, queues and fair share for batch workloads on Kubernetes.
• Pod priority and preemption — Kubernetes docs. Priority classes and how preemption selects victims.
• Control topology management policies on a node — Kubernetes docs. The single-numa-node policy and NUMA-aligned device allocation.