Workload scheduling, capacity management and GPU cluster operations

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Workload scheduling and capacity management · also covers Cluster operations, On-premises infrastructure, Data center infrastructure · Step 14 of 35 in MLOps / ML platform engineer · version 1

Practice this

On this page

A shared GPU cluster has more demand than capacity, so something must decide which jobs run, in what order, and on whose budget. The default Kubernetes scheduler places individual pods but has no notion of queues, team quotas or all-or-nothing job groups. Batch systems such as Kueue and Volcano add those on Kubernetes, while Slurm is the long-established scheduler on HPC-style clusters.

Read more

How it is done in practice

A platform team gives each group a LocalQueue backed by a ClusterQueue with a guaranteed GPU quota, places all queues in one cohort so idle capacity is borrowed, and defines a few priority classes such as production retraining, research and best-effort. Training job templates are created suspended with the queue label already set, and multi-worker jobs use all-or-nothing admission. Capacity planning runs on a regular cadence from metrics: pending time per queue, GPU allocation versus actual utilisation, and fragmentation across node types. Operations work includes cordoning and draining nodes for driver upgrades, handling failing GPUs, running reserved and spot capacity as separate flavors, and reviewing quota with team leads when wait times grow.

Technologies

KueueVolcanoSlurmkube-schedulerKubernetes ResourceQuotaKubernetes PriorityClassJobSetCluster AutoscalerPrometheus

Diagram

flowchart LR
  J[Job created suspended with queue label] --> LQ[LocalQueue in team namespace]
  LQ --> CQ[ClusterQueue with quota per ResourceFlavor]
  CQ <-->|borrow and lend| CO[Cohort of ClusterQueues]
  CQ -->|quota available: admit and unsuspend| JC[Job controller creates pods]
  JC --> KS[kube-scheduler filters and scores nodes]
  KS --> N[GPU nodes]
  CQ -->|no quota: wait or preempt lower priority| LQ
flowchart LR
  J[Job created suspended with queue label] --> LQ[LocalQueue in team namespace]
  LQ --> CQ[ClusterQueue with quota per ResourceFlavor]
  CQ <-->|borrow and lend| CO[Cohort of ClusterQueues]
  CQ -->|quota available: admit and unsuspend| JC[Job controller creates pods]
  JC --> KS[kube-scheduler filters and scores nodes]
  KS --> N[GPU nodes]
  CQ -->|no quota: wait or preempt lower priority| LQ

Code

yaml
apiVersion: batch/v1
kind: Job
metadata:
  generateName: finetune-
  namespace: team-nlp
  labels:
    kueue.x-k8s.io/queue-name: team-nlp-queue
spec:
  suspend: true          # Kueue unsuspends on admission
  parallelism: 1
  completions: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: registry.example/train:1.0
          resources:
            limits:
              nvidia.com/gpu: 4
bash
#!/bin/bash
#SBATCH --job-name=finetune
#SBATCH --partition=gpu
#SBATCH --nodes=2
#SBATCH --gres=gpu:4
#SBATCH --time=12:00:00
srun python train.py --epochs 3
# submit with: sbatch train.sbatch ; inspect with: squeue -u $USER

Subtopics

What kube-scheduler does and where it falls short for batch

Filter, score, bind

The Kubernetes scheduler watches for pods with no node assigned and places them one at a time. In the filtering phase it removes nodes that cannot run the pod because of insufficient resources, taints, affinity rules or other constraints. In the scoring phase it ranks the remaining feasible nodes and picks the highest, then binds the pod. This design is built for long-running services and works well for them. Its behaviour is extensible through scheduling framework plugins and profiles.

The gaps for training workloads

Three things are missing for batch. There is no job queue: a pod that does not fit simply stays pending, with no fair ordering between teams. There is no group awareness: a four-worker training job can get three pods placed and one stuck, holding three GPUs idle. And quota is a hard namespace cap through ResourceQuota, with no way to lend unused capacity to another team and take it back. Batch schedulers exist to fill these gaps.

Kueue: job queueing, quotas and fair sharing

Admission instead of placement

Kueue does not replace the scheduler. It decides when a job is allowed to start, then lets kube-scheduler place the pods. Jobs are created suspended; Kueue unsuspends them once quota is available. The main objects are the ResourceFlavor, which describes a kind of capacity such as a GPU model or spot versus on-demand nodes; the ClusterQueue, which holds quota per flavor and resource and the queueing and preemption policy; and the LocalQueue, a namespaced entry point that a team's jobs point to with a label. It works with batch Jobs, JobSet, Kubeflow jobs, Ray jobs and others.

Borrowing and preemption

ClusterQueues can be grouped in a cohort. A queue with unused nominal quota lends it to others in the cohort, and a queue under pressure can borrow up to a configured limit. When the owner needs its capacity back, preemption evicts borrowing workloads according to priority. This gives each team a guaranteed share while keeping GPUs busy. Workload priority classes decide ordering within a queue, and admission can be made all-or-nothing so a job starts only when every pod can be admitted.

Gang scheduling and Volcano

All or nothing placement

Gang scheduling means a group of pods is scheduled only if a minimum number of them can run at the same time. This is what distributed training and MPI jobs require, since a partial group does no useful work and wastes the GPUs it holds. Volcano is a batch scheduler for Kubernetes that implements this with a PodGroup object carrying a minimum member count. It runs as an additional scheduler that pods opt into by scheduler name, and it adds its own queues with weights and capacity, fair-share and priority policies, preemption and reclaim between queues, and bin-packing style placement that reduces GPU fragmentation. It integrates with frameworks such as Spark, PyTorch, TensorFlow, MPI and Ray. The choice between Volcano and Kueue often comes down to whether a team wants to replace placement logic or only add admission control on top of the default scheduler.

Slurm: the HPC scheduling model

Daemons, partitions and commands

Many large GPU clusters run Slurm rather than Kubernetes. A central controller daemon, slurmctld, tracks resources and the job queue, and a slurmd daemon on each compute node launches work. Nodes are grouped into partitions, which act as queues with their own limits. Users interact through a small set of commands: sbatch submits a batch script for later execution, srun launches a job or a step interactively, salloc reserves an allocation, squeue lists queued and running jobs, sinfo shows partition and node state, and scancel removes a job. A job requests nodes, tasks, GPUs and a time limit, waits until the allocation is available, then runs on bare metal with direct access to the hardware and interconnect.

How it compares

Slurm treats a multi-node job as a single unit by default, so gang scheduling is inherent, and time limits let it backfill short jobs into gaps. Kubernetes offers stronger isolation through containers, a richer API and better fit for services. Platform teams increasingly need both mental models.

Sources

Related lessons

Practice this
Distributed training across multiple GPU nodesModel registry, packaging and versioning