Day 10: GPUs in Kubernetes
Goal:
rke2-gpu-1reportsnvidia.com/gpu, pods see the card vianvidia-smi, the GPU can be shared via time-slicing, and DCGM metrics land in Prometheus. · Duration: approx. 150 min · Prerequisite: Day 9
Briefing
- Driver in the guest (Ubuntu package via Ansible) or as a container from the GPU Operator – in the lab you use the guest driver, and the Operator provides the rest.
- GPU Operator = Container Toolkit, device plugin, GPU Feature Discovery (GFD), DCGM Exporter, validator.
- RKE2 has its own containerd under
/run/k3s/containerd/– the toolkit has to know that. - The taint
nvidia.com/gpukeeps normal pods away from the expensive node. - Time-slicing shares a GPU without memory isolation. DRA is the successor of the device plugin.
rke2-gpu-1 (VM, passthrough L4)
Kernel: nvidia.ko (role nvidia_host)
RKE2 containerd /run/k3s/containerd <-- toolkit: runtime "nvidia" + CDI
Device plugin --> node: nvidia.com/gpu: 1 (time-slicing: 4)
GFD --> labels nvidia.com/gpu.product=NVIDIA-L4 ...
DCGM Exporter --> :9400/metrics --> ServiceMonitor --> Prometheus
Warning: Since there’s no GPU on this machine, all GPU outputs have been checked against the NVIDIA and RKE2 docs (GPU Operator v26.7.1, RKE2 v1.36) but not tested live. Manifests and Helm values were validated with
helm template+kubeconformandkubectl apply --dry-run=server, the Ansible role withansible-lint.
Drill 1: Driver in the guest or from the Operator?
| Criterion | Driver in the guest (package) | Driver container (Operator) |
|---|---|---|
| Installation | Ansible, reboot | DaemonSet, compiles/loads module |
| Kernel update | DKMS/package must match | image must match the kernel |
| Debugging | nvidia-smi directly on the host |
only in the driver pod |
| Good for | few, long-lived VMs (lab) | large, homogeneous fleets |
You install the driver in the guest. The nvidia_host
role uses ubuntu-drivers with server branch 580 (DRA
requires ≥ 580). The Operator variant is included as
code/gpu/gpu-operator-driver-values.yaml
(driver.enabled: true, driver 595.91.07).
cd ~/ki-lab/code/infra-rke2
mkdir -p roles/nvidia_host/{defaults,tasks,handlers}
cat > roles/nvidia_host/defaults/main.yml <<'EOF'
---
# Driver branch in the guest (Day 10). DRA needs >= 580.
nvidia_host_driver_branch: "580-server"
EOF
cat > roles/nvidia_host/tasks/main.yml <<'EOF'
---
- name: Werkzeug für Ubuntu-Treiberauswahl installieren
ansible.builtin.apt:
name: ubuntu-drivers-common
update_cache: true
- name: Ist der Treiber schon geladen?
ansible.builtin.command: nvidia-smi -L
register: nvidia_host_smi
changed_when: false
failed_when: false
- name: Headless-Treiber (GPGPU) installieren
ansible.builtin.command: >-
ubuntu-drivers install --gpgpu nvidia:{{ nvidia_host_driver_branch }}
when: nvidia_host_smi.rc != 0
changed_when: true
notify: Reboot GPU node
- name: Userspace-Tools (nvidia-smi, libnvidia-ml)
ansible.builtin.apt:
name: "nvidia-utils-{{ nvidia_host_driver_branch }}"
notify: Reboot GPU node
EOF
cat > roles/nvidia_host/handlers/main.yml <<'EOF'
---
- name: Reboot GPU node
ansible.builtin.reboot:
reboot_timeout: 600
EOF
cat > gpu.yml <<'EOF'
---
# Day 10: NVIDIA driver in the guest of the GPU VMs
- name: NVIDIA-Treiber auf GPU-Nodes
hosts: gpu_nodes
become: true
serial: 1
roles:
- nvidia_host
EOF
ansible-lint gpu.yml roles/nvidia_host
ansible-playbook -i inventory.yaml gpu.yml
ssh [email protected] nvidia-smi --query-gpu=name,driver_version,memory.total \
--format=csvExpected (lint tested live, the rest abbreviated, not tested live):
Passed: 0 failure(s), 0 warning(s) in 5 files processed of 5 encountered.
...
RUNNING HANDLER [nvidia_host : Reboot GPU node] ********************************
changed: [rke2-gpu-1]
rke2-gpu-1 : ok=6 changed=3 unreachable=0 failed=0
name, driver_version, memory.total [MiB]
NVIDIA L4, 580.95.05, 23034 MiB
Why --gpgpu? It installs the headless driver without X11
packages and with precompiled, signed kernel modules. The VM from Day 7
boots OVMF without preloaded Secure Boot keys – with Secure Boot, you’d
otherwise need MOK enrollment.
Save your progress? Sign up for free
Drill 2: Reserve the GPU node (taint)
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl taint node rke2-gpu-1 nvidia.com/gpu=present:NoSchedule
kubectl get node rke2-gpu-1 -o jsonpath='{.spec.taints}{"\n"}' | jq -c .Expected:
node/rke2-gpu-1 tainted
[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"present"}]
Why NoSchedule instead of NoExecute?
Running DaemonSets (Cilium, Traefik, node-exporter, Longhorn) stay, and
new normal pods no longer land on the GPU VM. The Operator DaemonSets
already tolerate exactly this key (chart default
daemonsets.tolerations). RKE2 applies taints from
config.yaml (node-taint) only at first
registration – for existing nodes you use kubectl.
Save your progress? Sign up for free
Drill 3: Install the GPU Operator on RKE2
mkdir -p ~/ki-lab/code/gpu && cd ~/ki-lab/code/gpu
cat > gpu-operator-values.yaml <<'EOF'
# NVIDIA GPU Operator v26.7.1 on RKE2 (Day 10)
# The driver lives in the guest (role nvidia_host); the Operator does the rest.
driver:
enabled: false
toolkit:
env:
# RKE2 runs its own containerd under /run/k3s
- name: CONTAINERD_SOCKET
value: /run/k3s/containerd/containerd.sock
dcgmExporter:
serviceMonitor:
enabled: true # kube-prometheus-stack from Day 9 picks it up
interval: 15s
EOF
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update nvidia
helm upgrade --install gpu-operator nvidia/gpu-operator --version v26.7.1 \
-n gpu-operator --create-namespace -f gpu-operator-values.yaml --waitExpected (abbreviated, not tested live):
Release "gpu-operator" does not exist. Installing it now.
NAME: gpu-operator
NAMESPACE: gpu-operator
STATUS: deployed
REVISION: 1
Why only CONTAINERD_SOCKET? According to the RKE2 docs,
that’s the only adjustment needed for Operator v25.10/v26.x. The toolkit
registers the runtime nvidia in RKE2’s containerd via this
socket. Older guides additionally set CONTAINERD_CONFIG to
config.toml.tmpl – no longer needed with current versions;
with v25.3 you still need runtimeClassName: nvidia in every
pod.
Tip: Alternatively, put the same values as a
HelmChartresource (helm.cattle.io/v1) into/var/lib/rancher/rke2/server/manifests/– that’s how the RKE2 docs do it. In the lab you stick with Helm as on Day 9.
Save your progress? Sign up for free
Drill 4: Validate the Operator
kubectl -n gpu-operator get pods
kubectl -n gpu-operator logs -l app=nvidia-operator-validator \
-c nvidia-operator-validator --tail=3
kubectl get runtimeclass
ssh [email protected] \
'sudo grep -rl nvidia /var/lib/rancher/rke2/agent/etc/containerd/'Expected (abbreviated, not tested live, pod names will differ):
NAME READY STATUS
gpu-feature-discovery-x7k2p 1/1 Running
gpu-operator-7c9d8f6b5-l2m4n 1/1 Running
gpu-operator-node-feature-discovery-gc-5f8b-q9w 1/1 Running
gpu-operator-node-feature-discovery-master-6d-tz 1/1 Running
gpu-operator-node-feature-discovery-worker-abcde 1/1 Running
nvidia-container-toolkit-daemonset-h5j8k 1/1 Running
nvidia-cuda-validator-p4r7s 0/1 Completed
nvidia-dcgm-exporter-m3n6b 1/1 Running
nvidia-device-plugin-daemonset-w8e2c 1/1 Running
nvidia-operator-validator-f6g9h 1/1 Running
all validations are successful
NAME HANDLER AGE
nvidia nvidia 4m
nvidia-cdi nvidia-cdi 4m
nvidia-legacy nvidia-legacy 4m
/var/lib/rancher/rke2/agent/etc/containerd/config.toml
No nvidia-driver-daemonset – that’s correct, the driver
comes from the guest. If nvidia-operator-validator hangs in
Init,
kubectl -n gpu-operator logs <pod> -c driver-validation
usually shows the reason (driver not loaded, wrong socket).
Save your progress? Sign up for free
Drill 5: Labels, capacity and nvidia-smi in a pod
GFD writes the card’s properties as labels onto the node – that lets you schedule workloads in a targeted way.
kubectl get node rke2-gpu-1 -o json | jq -r '.metadata.labels
| with_entries(select(.key | test("nvidia.com/(gpu|cuda|mig)")))
| to_entries[] | "\(.key)=\(.value)"' | sort
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'Expected (excerpt, not tested live, example L4):
nvidia.com/cuda.driver-version.major=580
nvidia.com/gpu.count=1
nvidia.com/gpu.family=ada-lovelace
nvidia.com/gpu.memory=23034
nvidia.com/gpu.present=true
nvidia.com/gpu.product=NVIDIA-L4
nvidia.com/mig.capable=false
1
Now the smoke test: a pod with nvidia-smi (CUDA base
image; the toolkit injects the driver and binary).
cat > gpu-smoke.yaml <<'EOF'
# GPU smoke test (Day 10): nvidia-smi on rke2-gpu-1
apiVersion: v1
kind: Namespace
metadata:
name: gpu-test
---
apiVersion: v1
kind: Pod
metadata:
name: nvidia-smi
namespace: gpu-test
spec:
restartPolicy: Never
nodeSelector:
nvidia.com/gpu.present: "true"
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: smi
image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
command: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1
EOF
kubectl apply -f gpu-smoke.yaml
kubectl -n gpu-test wait pod/nvidia-smi --for=jsonpath='{.status.phase}'=Succeeded \
--timeout=180s
kubectl -n gpu-test logs nvidia-smi | head -12Expected (server dry run tested live, outputs abbreviated, not tested live):
pod/nvidia-smi condition met
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name | Bus-Id Disp.A | Volatile Uncorr. ECC |
| 0 NVIDIA L4 On | 00000000:01:00.0 Off | 0 |
| N/A 34C P8 12W / 72W | 0MiB / 23034MiB | 0% Default |
...
Cross-check: a GPU pod without a toleration.
kubectl -n gpu-test get pod nvidia-smi -o json | jq '.metadata = {name: "no-tol",
namespace: "gpu-test"} | del(.spec.tolerations, .spec.nodeSelector,
.spec.nodeName, .status)' | kubectl create -f -
sleep 5
kubectl -n gpu-test get events --field-selector involvedObject.name=no-tol \
-o jsonpath='{.items[-1].message}{"\n"}'
kubectl -n gpu-test delete pod no-tolExpected (not tested live, abbreviated):
0/6 nodes are available: 1 node(s) had untolerated taint {nvidia.com/gpu:
present}, 2 Insufficient nvidia.com/gpu, 3 node(s) had untolerated taint
{CriticalAddonsOnly: true}. preemption: 0/6 nodes are available: ...
pod "no-tol" deleted
Why does every GPU pod need the toleration itself? The admission
controller ExtendedResourceToleration, which would add it
automatically, isn’t active in RKE2.
Save your progress? Sign up for free
Drill 6: Time-slicing – one GPU, four slots
vLLM chat (Day 11) and the embedding model (Day 13) should share the
L4. Time-slicing reports the card multiple times. Memory and faults are
not isolated – later,
--gpu-memory-utilization per vLLM handles that.
cat > time-slicing.yaml <<'EOF'
# Time-slicing: 1 physical GPU appears as 4 x nvidia.com/gpu
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config
namespace: gpu-operator
data:
any: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
renameByDefault: false
failRequestsGreaterThanOne: true
resources:
- name: nvidia.com/gpu
replicas: 4
EOF
cat > gpu-operator-timeslicing-values.yaml <<'EOF'
# Overlay for gpu-operator-values.yaml: the device plugin reads time-slicing-config
devicePlugin:
config:
name: time-slicing-config
default: any
EOF
kubectl apply -f time-slicing.yaml
helm upgrade gpu-operator nvidia/gpu-operator --version v26.7.1 -n gpu-operator \
-f gpu-operator-values.yaml -f gpu-operator-timeslicing-values.yaml --wait
kubectl -n gpu-operator rollout status ds/nvidia-device-plugin-daemonset
kubectl get node rke2-gpu-1 -o json | jq -c '{
gpu: .status.allocatable["nvidia.com/gpu"],
replicas: .metadata.labels["nvidia.com/gpu.replicas"],
product: .metadata.labels["nvidia.com/gpu.product"]}'Expected (dry run tested live, node output not tested live):
configmap/time-slicing-config created
daemon set "nvidia-device-plugin-daemonset" successfully rolled out
{"gpu":"4","replicas":"4","product":"NVIDIA-L4-SHARED"}
Why via Helm instead of kubectl patch clusterpolicies?
The NVIDIA docs patch the ClusterPolicy directly. But the
next helm upgrade would overwrite the patch again.
failRequestsGreaterThanOne: true makes pods with
nvidia.com/gpu: 2 fail – two slots wouldn’t be twice as
fast anyway. The suffix -SHARED on the product label
reveals via nodeSelector that the card is shared.
Save your progress? Sign up for free
Drill 7: DCGM Exporter in Prometheus
The Operator brings a Service and a ServiceMonitor. Prometheus from Day 9 collects all ServiceMonitors – without a release label.
kubectl -n gpu-operator get svc,servicemonitor -l app=nvidia-dcgm-exporter
# PromQL helper for the next chapters: query inside the Prometheus pod
promq() {
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-c prometheus -- wget -qO- "http://localhost:9090/api/v1/query?query=$(
jq -rn --arg q "$1" '$q|@uri')" | jq -r '.data.result[]
| "\(.metric.Hostname // .metric.pod // "-") \(.value[1])"'
}
promq 'DCGM_FI_DEV_GPU_UTIL'
promq 'DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)'
promq 'DCGM_FI_DEV_GPU_TEMP'
promq 'DCGM_FI_DEV_POWER_USAGE'Expected (abbreviated, not tested live):
NAME TYPE CLUSTER-IP PORT(S)
service/nvidia-dcgm-exporter ClusterIP 10.43.88.12 9400/TCP
NAME AGE
servicemonitor.monitoring.coreos.com/nvidia-dcgm-exporter 12m
rke2-gpu-1 0
rke2-gpu-1 0.0004
rke2-gpu-1 34
rke2-gpu-1 12.1
| Metric | Meaning | Alert from (rule of thumb) |
|---|---|---|
DCGM_FI_DEV_GPU_UTIL |
compute utilization in % | permanently 100% + queue |
DCGM_FI_DEV_FB_USED |
used VRAM in MiB | vLLM deliberately occupies ~90% |
DCGM_FI_DEV_GPU_TEMP |
temperature in °C | > 85 °C |
DCGM_FI_DEV_XID_ERRORS |
last Xid error | any value > 0 |
Why Hostname? DCGM sets the label to the node name. Even
with time-slicing you only see one physical GPU – the mapping to pods
comes through the pod/namespace labels as soon
as a pod holds the GPU.
Save your progress? Sign up for free
Drill 8: MIG check and DRA outlook
MIG (Multi-Instance GPU) splits A100/H100/H200/B200 into isolated instances with their own memory. An L4 or RTX can’t do that:
ssh [email protected] nvidia-smi --query-gpu=name,mig.mode.current --format=csvExpected (not tested live): NVIDIA L4, [N/A] after the
CSV header line.
With a MIG card, you set mig.strategy: single in the
chart and label the node with
nvidia.com/mig.config=all-1g.10gb. The MIG Manager then
partitions it and the device plugin reports the instances.
DRA (Dynamic Resource Allocation, resource.k8s.io/v1, GA
since Kubernetes 1.34) replaces nvidia.com/gpu: 1 with
claims that filter on attributes (model, memory). In GPU Operator v26.7,
that’s the experimental GPUCluster mode. It excludes the
ClusterPolicy, and there’s no in-place switch. So you
rebuild, and switch back again at the end.
cat > gpu-operator-dra-values.yaml <<'EOF'
# DRA mode (experimental in v26.7): GPUCluster instead of ClusterPolicy
# No in-place switch: uninstall the Operator first!
# Without device plugin and toolkit: the DRA driver provides CDI specs itself.
clusterPolicy:
deployCR: false
gpuCluster:
deployCR: true
driver:
enabled: false # driver from the role nvidia_host (>= 580)
draDriver:
computeDomains:
enabled: false # only for multi-node NVLink (GB200 & co.)
EOF
kubectl delete ns gpu-test --wait
helm uninstall gpu-operator -n gpu-operator --wait
helm install gpu-operator nvidia/gpu-operator --version v26.7.1 \
-n gpu-operator -f gpu-operator-dra-values.yaml --wait
kubectl get gpucluster gpu-cluster
kubectl get deviceclass
kubectl get resourceslice -o json | jq -r '.items[].spec.devices[]
| "\(.name) \(.attributes.productName.string) \(.capacity.memory.value)"'Expected (rendering and GPUCluster dry run tested live,
the rest not tested live):
NAME STATUS AGE
gpu-cluster ready 3m
NAME AGE
gpu.nvidia.com 3m
mig.nvidia.com 3m
vfio.gpu.nvidia.com 3m
gpu-0 NVIDIA L4 23034Mi
The claim requests “a GPU with more than 16 GiB” – via CEL instead of a node label:
kubectl create ns gpu-test
cat > dra-test.yaml <<'EOF'
# DRA (Day 10, Drill 8): GPU via ResourceClaim instead of nvidia.com/gpu
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: single-gpu
namespace: gpu-test
spec:
spec:
devices:
requests:
- name: gpu
exactly:
deviceClassName: gpu.nvidia.com
selectors:
- cel:
expression: >-
device.capacity['gpu.nvidia.com'].memory
.isGreaterThan(quantity('16Gi'))
---
apiVersion: v1
kind: Pod
metadata:
name: dra-smi
namespace: gpu-test
spec:
restartPolicy: Never
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: smi
image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
command: ["nvidia-smi", "-L"]
resources:
claims:
- name: gpu
resourceClaims:
- name: gpu
resourceClaimTemplateName: single-gpu
EOF
kubectl apply -f dra-test.yaml
kubectl -n gpu-test wait pod/dra-smi --for=jsonpath='{.status.phase}'=Succeeded \
--timeout=180s
kubectl -n gpu-test logs dra-smi
kubectl -n gpu-test get resourceclaimsExpected (dry run tested live, output not tested live):
resourceclaimtemplate.resource.k8s.io/single-gpu created
pod/dra-smi created
pod/dra-smi condition met
GPU 0: NVIDIA L4 (UUID: GPU-5c1f...)
NAME STATE AGE
dra-smi-gpu-x8k2q allocated,reserved 40s
Back to device plugin mode with time-slicing – chapters 11–14 build on it:
kubectl delete ns gpu-test
helm uninstall gpu-operator -n gpu-operator --wait
helm install gpu-operator nvidia/gpu-operator --version v26.7.1 -n gpu-operator \
-f gpu-operator-values.yaml -f gpu-operator-timeslicing-values.yaml --wait
kubectl get node rke2-gpu-1 \
-o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'Expected (not tested live):
4
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Taint nvidia.com/gpu auf rke2-gpu-1" 'kubectl get node rke2-gpu-1 \
-o jsonpath="{.spec.taints[*].key}" | grep -q nvidia.com/gpu'
check "Operator-Validator Running" '[ "$(kubectl -n gpu-operator get pods \
-l app=nvidia-operator-validator -o jsonpath="{.items[0].status.phase}")" \
= Running ]'
check "GFD-Label gpu.present" '[ "$(kubectl get node rke2-gpu-1 -o \
jsonpath="{.metadata.labels.nvidia\.com/gpu\.present}")" = true ]'
check "Time-Slicing: 4 GPUs" '[ "$(kubectl get node rke2-gpu-1 -o \
jsonpath="{.status.allocatable.nvidia\.com/gpu}")" = 4 ]'
check "ServiceMonitor DCGM" 'kubectl -n gpu-operator get servicemonitor \
nvidia-dcgm-exporter'
check "DCGM-Metrik in Prometheus" '[ -n "$(promq DCGM_FI_DEV_GPU_TEMP)" ]'Expected:
✔ Taint nvidia.com/gpu auf rke2-gpu-1
✔ Operator-Validator Running
✔ GFD-Label gpu.present
✔ Time-Slicing: 4 GPUs
✔ ServiceMonitor DCGM
✔ DCGM-Metrik in Prometheus
The last check needs the promq function from Drill 7
(source it in the same shell).
Challenge
A second GPU node should later get only 2 slots. Create a second
profile l4-2 in the same ConfigMap and assign it via node
label only to rke2-gpu-1 – for testing.
Hint: The device plugin reads the profile from the
node label nvidia.com/device-plugin.config;
default: any applies only to nodes without the label.
Solution
kubectl -n gpu-operator patch configmap time-slicing-config --type merge -p '
data:
l4-2: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
failRequestsGreaterThanOne: true
resources:
- name: nvidia.com/gpu
replicas: 2
'
kubectl label node rke2-gpu-1 nvidia.com/device-plugin.config=l4-2 --overwrite
sleep 60
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
kubectl label node rke2-gpu-1 nvidia.com/device-plugin.config-Expected (not tested live):
2
Save your progress? Sign up for free
Enterprise Note
- Version driver, toolkit and kernel together and test them on a
canary GPU node. With the Operator driver, use precompiled images
(
driver.usePrecompiled) for reproducible rollouts. - Time-slicing only for dev/test or clearly budgeted inference. For
tenant isolation, MIG (A100/H100) or dedicated GPUs, plus ResourceQuotas
on
requests.nvidia.com/gpuper namespace. - DRA is the future (filters on memory/model, shared claims, dynamic
MIG). In the GPU Operator, the
GPUClustermode is still experimental – plan it for production only once NVIDIA releases it. - DCGM alerts: Xid errors, ECC errors, temperature, throttling
(
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS). Manage GPU nodes via node pools and the cluster autoscaler (cloud) or capacity planning (on-prem). - Hardening: the CIS profile and PSA
restrictedneed exceptions for thegpu-operatornamespace (privileged DaemonSets) – document and audit them.