GPU compute on Kubernetes with the NVIDIA stack

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: GPU compute · also covers NVIDIA hardware · Step 12 of 35 in MLOps / ML platform engineer · version 1

Kubernetes does not understand GPUs natively; it learns about them through a device plugin that advertises them as an extended resource. The NVIDIA stack supplies the pieces that make this work: kernel driver, container toolkit, device plugin and monitoring, usually installed together by the GPU Operator. This lesson covers how those layers fit, how pods request GPUs, and how a GPU can be shared or partitioned.

How it is done in practice

A platform team creates dedicated GPU node pools per instance type, taints them, and installs the GPU Operator with Helm through the same GitOps flow as every other cluster add-on. On managed clouds they decide whether to use the provider's GPU-ready node image with a preinstalled driver or let the operator manage the driver. DCGM exporter metrics for utilization, memory and temperature flow into Prometheus and feed dashboards and idle-GPU alerts. Interactive and small inference workloads go on time-sliced or MIG pools, training jobs get whole GPUs, and driver upgrades are rolled out pool by pool after a canary node passes the operator's validation.

Technologies

Diagram

flowchart TB
  subgraph Node[GPU node]
    D[NVIDIA kernel driver] --> T[NVIDIA Container Toolkit]
    T --> R[containerd runtime]
    P[Device plugin DaemonSet] -->|registers nvidia.com/gpu| K[kubelet]
    K --> R
    G[GPU Feature Discovery] -->|node labels| K
    M[DCGM exporter] --> Prom[Prometheus]
  end
  O[GPU Operator and ClusterPolicy] --> D
  O --> T
  O --> P
  O --> G
  O --> M
  K -->|allocatable GPUs| A[API server]
  A --> S[kube-scheduler]
  S -->|binds pod requesting GPU| K

flowchart TB
  subgraph Node[GPU node]
    D[NVIDIA kernel driver] --> T[NVIDIA Container Toolkit]
    T --> R[containerd runtime]
    P[Device plugin DaemonSet] -->|registers nvidia.com/gpu| K[kubelet]
    K --> R
    G[GPU Feature Discovery] -->|node labels| K
    M[DCGM exporter] --> Prom[Prometheus]
  end
  O[GPU Operator and ClusterPolicy] --> D
  O --> T
  O --> P
  O --> G
  O --> M
  K -->|allocatable GPUs| A[API server]
  A --> S[kube-scheduler]
  S -->|binds pod requesting GPU| K

Code

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: cuda
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

Subtopics

The layers between a pod and a GPU

From hardware to an extended resource

Four layers must line up on every GPU node. The NVIDIA kernel driver talks to the hardware. The NVIDIA Container Toolkit configures the container runtime, such as containerd, so that a container can be started with the driver libraries and device files mounted in. The NVIDIA device plugin runs as a DaemonSet, registers with the kubelet through the device plugin gRPC API, and advertises the node's GPUs as the extended resource nvidia.com/gpu. Finally the kubelet reports that capacity to the API server, and the scheduler can place pods against it. When a pod is admitted, the kubelet asks the plugin which devices to allocate and the runtime injects them.

Where it breaks

Most GPU node problems are a mismatch between layers: a driver version that does not support the CUDA version in the image, a runtime not configured to use the NVIDIA runtime, or a device plugin pod that is not running so the node shows zero allocatable GPUs. The first debugging steps are to check node allocatable resources, check the plugin and driver pods on that node, and run nvidia-smi inside a test container.

Automating the stack with the NVIDIA GPU Operator

One operator, many components

Installing the layers by hand on every node image does not scale, so the GPU Operator manages them as containers. It is installed with Helm and uses the operator pattern to deploy and reconcile: the driver, either as a container or by using a driver preinstalled in the node image, the Container Toolkit, the device plugin, GPU Feature Discovery which labels nodes with GPU model, memory and capabilities, the DCGM exporter for Prometheus metrics, and a validator that confirms a node is ready. A custom resource called ClusterPolicy holds the configuration. Node Feature Discovery detects which nodes have NVIDIA hardware, so the components land only there. Upgrading drivers becomes a controlled rollout rather than a node image rebuild.

Requesting GPUs and keeping GPU nodes for GPU work

Resource requests

A container asks for GPUs in its resource limits, for example nvidia.com/gpu set to 1. GPUs are whole-number resources and cannot be overcommitted: if a request is given it must equal the limit, and specifying only the limit is enough. A pod that needs a particular GPU type selects it with a node selector or node affinity against labels published by GPU Feature Discovery, such as the product name or memory size.

Taints and tolerations

GPU nodes are expensive, so clusters stop ordinary pods from landing on them. The node pool carries a taint, commonly keyed nvidia.com/gpu with effect NoSchedule, and only pods with a matching toleration are scheduled there. A toleration permits placement but does not attract the pod, so it is paired with a GPU request or node affinity. Together with separate node pools per GPU type, this keeps system workloads on cheap CPU nodes and lets the autoscaler scale GPU pools to zero when idle.

Sharing GPUs: time-slicing, MPS, MIG and Dynamic Resource Allocation

Three sharing mechanisms

A whole GPU per pod wastes capacity for notebooks and small inference services. Time-slicing lets the device plugin advertise each GPU as several replicas; workloads take turns on the device with no memory or fault isolation. MPS, the Multi-Process Service, lets processes run concurrently on one GPU with better throughput than time-slicing, with limited isolation. MIG, Multi-Instance GPU, is available on supported data center GPUs and partitions one card into hardware-isolated instances with dedicated memory and compute, which is the right choice for multi-tenant use. Training jobs normally still take whole GPUs.

Dynamic Resource Allocation

The device plugin model can only count identical devices. Dynamic Resource Allocation is the newer Kubernetes API for richer requests. Drivers publish devices and their attributes in ResourceSlices, administrators define DeviceClasses, and workloads state what they need in a ResourceClaim, optionally generated from a ResourceClaimTemplate. The scheduler matches claims to devices using attribute selectors, so a pod can ask for a GPU with a minimum amount of memory, or several containers can share one claimed device.

Sources

• docNVIDIA GPU Operator (docs.nvidia.com)
• docInstalling the NVIDIA Container Toolkit (docs.nvidia.com)
• docSchedule GPUs (Kubernetes) (kubernetes.io)
• repoNVIDIA device plugin for Kubernetes (github.com)
• docKubernetes Documentation: Device Plugins (kubernetes.io)
• docKubernetes Documentation: Dynamic Resource Allocation (kubernetes.io)
• docKubernetes Documentation: Taints and Tolerations (kubernetes.io)

Related lessons

• Kubernetes for ML workloads: containers, deployments and jobs
• AWS and EKS foundations for an ML platform
• Distributed training across multiple GPU nodes
• Workload scheduling, capacity management and GPU cluster operations
• Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization
• Serving open-weight LLMs with vLLM and SGLang