Deep Dive — GPU Scheduling & Orchestration

The scheduling problem, in one paragraph

You have a heterogeneous fleet — H100s, A100s, maybe L40s, with different memory sizes and topology. You have multiple tenants — research training, online inference, batch evaluation, dev/notebooks. Their workloads have different shapes: long jobs with strict topology requirements (training), short jobs with strict latency requirements (online inference), bursty jobs with elastic needs (batch). Fair use across teams matters; so does maximizing utilization; so does respecting priority for the production path. The "scheduler" you build is the answer to "who gets which GPU, when, and what happens when the answer changes."

Kubernetes device plugin

K8s itself only knows about CPU, memory, and ephemeral storage. The NVIDIA device plugin (a DaemonSet) introduces nvidia.com/gpu as a schedulable resource. The flow:

1. Plugin starts on each GPU node, discovers GPUs via NVML.
2. Registers with kubelet over a Unix socket.
3. Kubelet advertises N GPUs as allocatable to the API server.
4. Scheduler sees pod request nvidia.com/gpu: 4, finds a node with ≥4 free, binds.
5. Kubelet calls plugin's Allocate(), which returns env vars and device mounts.
6. Container starts with NVIDIA_VISIBLE_DEVICES set; toolkit hook bind-mounts devices.

The plugin is also where you configure sharing modes: exclusive, time-slicing, MPS, or MIG.

# NVIDIA device plugin config — time-slicing example
version: v1
flags:
  migStrategy: "single"        # or "mixed" or "none"
sharing:
  timeSlicing:
    resources:
      - name: nvidia.com/gpu
        replicas: 4            # 4 virtual GPUs share each physical

The NVIDIA GPU Operator ships the device plugin together with driver containers, container toolkit, DCGM exporter, and MIG manager — strongly recommended over rolling your own.

MIG — Multi-Instance GPU

MIG (A100/H100/B100) partitions a single GPU into up to 7 hardware-isolated instances. Each instance gets dedicated SM partitions, L2 slices, and memory. This is real isolation — separate fault domains, separate memory bandwidth, no noisy neighbor.

Tradeoffs:

• Hardware isolation — predictable performance under multi-tenancy. Great for shared inference platforms.
• Static partitioning — you choose profiles at boot; changing requires a workload drain. Mixed-profile nodes complicate scheduling.
• No NVLink between MIG instances — multi-MIG-instance models lose collectives. MIG is a small-job platform.
• No P2P, no MPS within a MIG instance — they're mutually exclusive.

MPS — Multi-Process Service

MPS lets multiple CUDA processes share a single GPU's SMs concurrently via a server proxy. Unlike default context-switching (which serializes work), MPS interleaves kernels from different processes on the SMs.

• Better utilization when each tenant is too small to saturate the GPU.
• Soft isolation only — one process's OOM can affect neighbors; one process's bug can corrupt others (less so with Volta+ MPS, but still weaker than MIG).
• Per-process SM percentage caps available; per-process memory caps available on Volta+.
• Latency improves for many concurrent small workloads compared to default time-slicing.

MPS is the right choice when you have many small co-trusted clients (e.g. several inference replicas sharing a node) and you've accepted the weaker isolation in exchange for better utilization.

The three sharing modes, side by side

Gang scheduling

Distributed training needs all N pods to start at the same time. If only 6 of 8 pods land and the other 2 are pending, the 6 idle GPUs are useless and may block someone else. Vanilla Kubernetes will happily start the 6.

Solutions:

• Volcano — Apache project, full batch scheduler for K8s with gang scheduling, queues, fair share. Common in production AI fleets.
• Kueue — Kubernetes-native job queueing built by SIG-Scheduling. Newer, simpler, increasingly the default.
• Yunikorn — Apache, started at Lyft / Cloudera, hierarchical queues.
• PodGroup (Volcano) / Workload (Kueue) — the CRDs that express "schedule all N or none."

# Kueue: a ClusterQueue with cohort-level borrowing
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-h100
spec:
  cohort: gpu-fleet
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: h100-sxm
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 32
              borrowingLimit: 16   # can borrow from cohort up to 16 more
            - name: cpu
              nominalQuota: 384
            - name: memory
              nominalQuota: 4Ti

Priority & preemption

Production inference must beat researcher fine-tuning when capacity is tight. K8s gives you PriorityClass:

• inference-online — highest priority, can preempt others.
• training-prod — high but below online.
• research-batch — low, preemptible.
• opportunistic — negative priority, runs only if nothing else wants the resource.

The interesting part is the preemption policy:

• Graceful preemption — send SIGTERM, give a budget (e.g. 60s) to checkpoint, then SIGKILL.
• Hard preemption — kill now. Used when the system is in trouble.
• Preemption avoidance — only preempt if no other node fits; bin-packing-aware.

Quotas & fair share

The company has teams; teams have budgets. The scheduler enforces them.

• ResourceQuota per namespace — hard caps on requests.nvidia.com/gpu.
• LimitRange — per-pod defaults and maxes.
• Hierarchical queues (Volcano, Yunikorn) — "research" gets 60%, "product" gets 40%, with borrowing when one is idle.
• Fair-share / DRF (Dominant Resource Fairness) — when both demand more than their share, allocate proportionally.
• Backfill — small jobs slip into gaps left by large gang jobs waiting for placement.

Quota is also a cost mechanism: a team that consistently uses 100% of its quota is the signal that you should resize or renegotiate.

Node configuration that matters

The "boring" host config is where many GPU perf bugs hide.

• NUMA pinning — pin pod CPUs to the NUMA node where the GPUs physically attach. K8s topology manager with single-numa-node policy.
• CPU pinning & isolation — exclusive CPU sets via the cpu manager. Inference latency is sensitive to scheduler jitter.
• Hugepages — enable 2 MiB or 1 GiB hugepages for RDMA-heavy paths.
• IRQ affinity — NIC interrupts on the right NUMA node.
• CPU governor — performance, not powersave. (Yes, this is a real production gotcha.)
• Persistence mode — nvidia-smi -pm 1 or run nvidia-persistenced so the driver doesn't unload between processes.
• Fabric Manager — required for NVSwitch nodes (DGX-class, HGX).
• ECC settings, GPU clocks, application clocks — usually leave at default; document if you don't.

# kubelet config: NUMA-aware GPU placement
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
topologyManagerPolicy: single-numa-node
reservedSystemCPUs: "0,1,48,49"
featureGates:
  CPUManager: true
  TopologyManager: true

Topology-aware scheduling

A pod requesting 4 GPUs on an 8-GPU H100 node should get 4 GPUs connected by NVLink as a single NVSwitch group, not a random 4 of 8. The NVIDIA device plugin supports topology-aware allocation:

• Best-effort mode prefers same-NVLink-domain GPUs but won't fail if unavailable.
• Strict mode refuses to allocate if topology can't be satisfied.

Across nodes: place gang-scheduled pods on nodes connected to the same rail of the IB fabric. Tools like Kueue's flavor fungibility, kube-scheduler-extender, or vendor schedulers (Run:AI, etc.) handle this.

Workload isolation — what guarantees can you actually make?

Be precise here in interviews. Different mechanisms give different guarantees.

Queue depth & saturation signals

How do you know your scheduler is failing? Watch:

• Pending pod count + age per priority class. Online inference pending > 0 is an immediate page.
• Queue depth per Kueue/Volcano queue.
• Time-to-bind p95 for new pods.
• GPU allocatable vs available vs SM-active — three different things; the gap between them is where waste hides.
• Preemption rate — sudden spikes mean priority is being abused or capacity has shifted.
• Fragmentation — number of nodes with idle GPUs that no pending pod can use (wrong topology, wrong taints).

The dashboard you build for this is half the role's day-to-day. See chapter 12 for the telemetry pipeline.