Skip to content
Hands-on IT IT Drills

Day 10: GPUs in Kubernetes

Goal: rke2-gpu-1 reports nvidia.com/gpu, pods see the card via nvidia-smi, the GPU can be shared via time-slicing, and DCGM metrics land in Prometheus. · Duration: approx. 150 min · Prerequisite: Day 9

Briefing

  • Driver in the guest (Ubuntu package via Ansible) or as a container from the GPU Operator – in the lab you use the guest driver, and the Operator provides the rest.
  • GPU Operator = Container Toolkit, device plugin, GPU Feature Discovery (GFD), DCGM Exporter, validator.
  • RKE2 has its own containerd under /run/k3s/containerd/ – the toolkit has to know that.
  • The taint nvidia.com/gpu keeps normal pods away from the expensive node.
  • Time-slicing shares a GPU without memory isolation. DRA is the successor of the device plugin.
rke2-gpu-1 (VM, passthrough L4)
  Kernel: nvidia.ko (role nvidia_host)
  RKE2 containerd /run/k3s/containerd  <-- toolkit: runtime "nvidia" + CDI
  Device plugin  --> node: nvidia.com/gpu: 1 (time-slicing: 4)
  GFD            --> labels nvidia.com/gpu.product=NVIDIA-L4 ...
  DCGM Exporter  --> :9400/metrics --> ServiceMonitor --> Prometheus

Warning: Since there’s no GPU on this machine, all GPU outputs have been checked against the NVIDIA and RKE2 docs (GPU Operator v26.7.1, RKE2 v1.36) but not tested live. Manifests and Helm values were validated with helm template + kubeconform and kubectl apply --dry-run=server, the Ansible role with ansible-lint.

Drill 1: Driver in the guest or from the Operator?

Criterion Driver in the guest (package) Driver container (Operator)
Installation Ansible, reboot DaemonSet, compiles/loads module
Kernel update DKMS/package must match image must match the kernel
Debugging nvidia-smi directly on the host only in the driver pod
Good for few, long-lived VMs (lab) large, homogeneous fleets

You install the driver in the guest. The nvidia_host role uses ubuntu-drivers with server branch 580 (DRA requires ≥ 580). The Operator variant is included as code/gpu/gpu-operator-driver-values.yaml (driver.enabled: true, driver 595.91.07).

cd ~/ki-lab/code/infra-rke2
mkdir -p roles/nvidia_host/{defaults,tasks,handlers}
cat > roles/nvidia_host/defaults/main.yml <<'EOF'
---
# Driver branch in the guest (Day 10). DRA needs >= 580.
nvidia_host_driver_branch: "580-server"
EOF
cat > roles/nvidia_host/tasks/main.yml <<'EOF'
---
- name: Werkzeug für Ubuntu-Treiberauswahl installieren
  ansible.builtin.apt:
    name: ubuntu-drivers-common
    update_cache: true

- name: Ist der Treiber schon geladen?
  ansible.builtin.command: nvidia-smi -L
  register: nvidia_host_smi
  changed_when: false
  failed_when: false

- name: Headless-Treiber (GPGPU) installieren
  ansible.builtin.command: >-
    ubuntu-drivers install --gpgpu nvidia:{{ nvidia_host_driver_branch }}
  when: nvidia_host_smi.rc != 0
  changed_when: true
  notify: Reboot GPU node

- name: Userspace-Tools (nvidia-smi, libnvidia-ml)
  ansible.builtin.apt:
    name: "nvidia-utils-{{ nvidia_host_driver_branch }}"
  notify: Reboot GPU node
EOF
cat > roles/nvidia_host/handlers/main.yml <<'EOF'
---
- name: Reboot GPU node
  ansible.builtin.reboot:
    reboot_timeout: 600
EOF
cat > gpu.yml <<'EOF'
---
# Day 10: NVIDIA driver in the guest of the GPU VMs
- name: NVIDIA-Treiber auf GPU-Nodes
  hosts: gpu_nodes
  become: true
  serial: 1
  roles:
    - nvidia_host
EOF
ansible-lint gpu.yml roles/nvidia_host
ansible-playbook -i inventory.yaml gpu.yml
ssh [email protected] nvidia-smi --query-gpu=name,driver_version,memory.total \
  --format=csv

Expected (lint tested live, the rest abbreviated, not tested live):

Passed: 0 failure(s), 0 warning(s) in 5 files processed of 5 encountered.
...
RUNNING HANDLER [nvidia_host : Reboot GPU node] ********************************
changed: [rke2-gpu-1]
rke2-gpu-1   : ok=6    changed=3    unreachable=0    failed=0
name, driver_version, memory.total [MiB]
NVIDIA L4, 580.95.05, 23034 MiB

Why --gpgpu? It installs the headless driver without X11 packages and with precompiled, signed kernel modules. The VM from Day 7 boots OVMF without preloaded Secure Boot keys – with Secure Boot, you’d otherwise need MOK enrollment.

Save your progress? Sign up for free

Drill 2: Reserve the GPU node (taint)

export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl taint node rke2-gpu-1 nvidia.com/gpu=present:NoSchedule
kubectl get node rke2-gpu-1 -o jsonpath='{.spec.taints}{"\n"}' | jq -c .

Expected:

node/rke2-gpu-1 tainted
[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"present"}]

Why NoSchedule instead of NoExecute? Running DaemonSets (Cilium, Traefik, node-exporter, Longhorn) stay, and new normal pods no longer land on the GPU VM. The Operator DaemonSets already tolerate exactly this key (chart default daemonsets.tolerations). RKE2 applies taints from config.yaml (node-taint) only at first registration – for existing nodes you use kubectl.

Save your progress? Sign up for free

Drill 3: Install the GPU Operator on RKE2

mkdir -p ~/ki-lab/code/gpu && cd ~/ki-lab/code/gpu
cat > gpu-operator-values.yaml <<'EOF'
# NVIDIA GPU Operator v26.7.1 on RKE2 (Day 10)
# The driver lives in the guest (role nvidia_host); the Operator does the rest.
driver:
  enabled: false
toolkit:
  env:
    # RKE2 runs its own containerd under /run/k3s
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock
dcgmExporter:
  serviceMonitor:
    enabled: true           # kube-prometheus-stack from Day 9 picks it up
    interval: 15s
EOF
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update nvidia
helm upgrade --install gpu-operator nvidia/gpu-operator --version v26.7.1 \
  -n gpu-operator --create-namespace -f gpu-operator-values.yaml --wait

Expected (abbreviated, not tested live):

Release "gpu-operator" does not exist. Installing it now.
NAME: gpu-operator
NAMESPACE: gpu-operator
STATUS: deployed
REVISION: 1

Why only CONTAINERD_SOCKET? According to the RKE2 docs, that’s the only adjustment needed for Operator v25.10/v26.x. The toolkit registers the runtime nvidia in RKE2’s containerd via this socket. Older guides additionally set CONTAINERD_CONFIG to config.toml.tmpl – no longer needed with current versions; with v25.3 you still need runtimeClassName: nvidia in every pod.

Tip: Alternatively, put the same values as a HelmChart resource (helm.cattle.io/v1) into /var/lib/rancher/rke2/server/manifests/ – that’s how the RKE2 docs do it. In the lab you stick with Helm as on Day 9.

Save your progress? Sign up for free

Drill 4: Validate the Operator

kubectl -n gpu-operator get pods
kubectl -n gpu-operator logs -l app=nvidia-operator-validator \
  -c nvidia-operator-validator --tail=3
kubectl get runtimeclass
ssh [email protected] \
  'sudo grep -rl nvidia /var/lib/rancher/rke2/agent/etc/containerd/'

Expected (abbreviated, not tested live, pod names will differ):

NAME                                                READY   STATUS
gpu-feature-discovery-x7k2p                         1/1     Running
gpu-operator-7c9d8f6b5-l2m4n                        1/1     Running
gpu-operator-node-feature-discovery-gc-5f8b-q9w     1/1     Running
gpu-operator-node-feature-discovery-master-6d-tz    1/1     Running
gpu-operator-node-feature-discovery-worker-abcde    1/1     Running
nvidia-container-toolkit-daemonset-h5j8k            1/1     Running
nvidia-cuda-validator-p4r7s                         0/1     Completed
nvidia-dcgm-exporter-m3n6b                          1/1     Running
nvidia-device-plugin-daemonset-w8e2c                1/1     Running
nvidia-operator-validator-f6g9h                     1/1     Running
all validations are successful
NAME             HANDLER          AGE
nvidia           nvidia           4m
nvidia-cdi       nvidia-cdi       4m
nvidia-legacy    nvidia-legacy    4m
/var/lib/rancher/rke2/agent/etc/containerd/config.toml

No nvidia-driver-daemonset – that’s correct, the driver comes from the guest. If nvidia-operator-validator hangs in Init, kubectl -n gpu-operator logs <pod> -c driver-validation usually shows the reason (driver not loaded, wrong socket).

Save your progress? Sign up for free

Drill 5: Labels, capacity and nvidia-smi in a pod

GFD writes the card’s properties as labels onto the node – that lets you schedule workloads in a targeted way.

kubectl get node rke2-gpu-1 -o json | jq -r '.metadata.labels
  | with_entries(select(.key | test("nvidia.com/(gpu|cuda|mig)")))
  | to_entries[] | "\(.key)=\(.value)"' | sort
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'

Expected (excerpt, not tested live, example L4):

nvidia.com/cuda.driver-version.major=580
nvidia.com/gpu.count=1
nvidia.com/gpu.family=ada-lovelace
nvidia.com/gpu.memory=23034
nvidia.com/gpu.present=true
nvidia.com/gpu.product=NVIDIA-L4
nvidia.com/mig.capable=false
1

Now the smoke test: a pod with nvidia-smi (CUDA base image; the toolkit injects the driver and binary).

cat > gpu-smoke.yaml <<'EOF'
# GPU smoke test (Day 10): nvidia-smi on rke2-gpu-1
apiVersion: v1
kind: Namespace
metadata:
  name: gpu-test
---
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
  namespace: gpu-test
spec:
  restartPolicy: Never
  nodeSelector:
    nvidia.com/gpu.present: "true"
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: smi
      image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
EOF
kubectl apply -f gpu-smoke.yaml
kubectl -n gpu-test wait pod/nvidia-smi --for=jsonpath='{.status.phase}'=Succeeded \
  --timeout=180s
kubectl -n gpu-test logs nvidia-smi | head -12

Expected (server dry run tested live, outputs abbreviated, not tested live):

pod/nvidia-smi condition met
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05     Driver Version: 580.95.05     CUDA Version: 13.0   |
|-------------------------------+----------------------+----------------------+
| GPU  Name                     | Bus-Id        Disp.A | Volatile Uncorr. ECC |
|   0  NVIDIA L4             On | 00000000:01:00.0 Off |                    0 |
| N/A   34C    P8    12W /  72W |      0MiB / 23034MiB |      0%      Default |
...

Cross-check: a GPU pod without a toleration.

kubectl -n gpu-test get pod nvidia-smi -o json | jq '.metadata = {name: "no-tol",
  namespace: "gpu-test"} | del(.spec.tolerations, .spec.nodeSelector,
  .spec.nodeName, .status)' | kubectl create -f -
sleep 5
kubectl -n gpu-test get events --field-selector involvedObject.name=no-tol \
  -o jsonpath='{.items[-1].message}{"\n"}'
kubectl -n gpu-test delete pod no-tol

Expected (not tested live, abbreviated):

0/6 nodes are available: 1 node(s) had untolerated taint {nvidia.com/gpu:
present}, 2 Insufficient nvidia.com/gpu, 3 node(s) had untolerated taint
{CriticalAddonsOnly: true}. preemption: 0/6 nodes are available: ...
pod "no-tol" deleted

Why does every GPU pod need the toleration itself? The admission controller ExtendedResourceToleration, which would add it automatically, isn’t active in RKE2.

Save your progress? Sign up for free

Drill 6: Time-slicing – one GPU, four slots

vLLM chat (Day 11) and the embedding model (Day 13) should share the L4. Time-slicing reports the card multiple times. Memory and faults are not isolated – later, --gpu-memory-utilization per vLLM handles that.

cat > time-slicing.yaml <<'EOF'
# Time-slicing: 1 physical GPU appears as 4 x nvidia.com/gpu
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4
EOF
cat > gpu-operator-timeslicing-values.yaml <<'EOF'
# Overlay for gpu-operator-values.yaml: the device plugin reads time-slicing-config
devicePlugin:
  config:
    name: time-slicing-config
    default: any
EOF
kubectl apply -f time-slicing.yaml
helm upgrade gpu-operator nvidia/gpu-operator --version v26.7.1 -n gpu-operator \
  -f gpu-operator-values.yaml -f gpu-operator-timeslicing-values.yaml --wait
kubectl -n gpu-operator rollout status ds/nvidia-device-plugin-daemonset
kubectl get node rke2-gpu-1 -o json | jq -c '{
  gpu: .status.allocatable["nvidia.com/gpu"],
  replicas: .metadata.labels["nvidia.com/gpu.replicas"],
  product: .metadata.labels["nvidia.com/gpu.product"]}'

Expected (dry run tested live, node output not tested live):

configmap/time-slicing-config created
daemon set "nvidia-device-plugin-daemonset" successfully rolled out
{"gpu":"4","replicas":"4","product":"NVIDIA-L4-SHARED"}

Why via Helm instead of kubectl patch clusterpolicies? The NVIDIA docs patch the ClusterPolicy directly. But the next helm upgrade would overwrite the patch again. failRequestsGreaterThanOne: true makes pods with nvidia.com/gpu: 2 fail – two slots wouldn’t be twice as fast anyway. The suffix -SHARED on the product label reveals via nodeSelector that the card is shared.

Save your progress? Sign up for free

Drill 7: DCGM Exporter in Prometheus

The Operator brings a Service and a ServiceMonitor. Prometheus from Day 9 collects all ServiceMonitors – without a release label.

kubectl -n gpu-operator get svc,servicemonitor -l app=nvidia-dcgm-exporter
# PromQL helper for the next chapters: query inside the Prometheus pod
promq() {
  kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
    -c prometheus -- wget -qO- "http://localhost:9090/api/v1/query?query=$(
    jq -rn --arg q "$1" '$q|@uri')" | jq -r '.data.result[]
    | "\(.metric.Hostname // .metric.pod // "-") \(.value[1])"'
}
promq 'DCGM_FI_DEV_GPU_UTIL'
promq 'DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)'
promq 'DCGM_FI_DEV_GPU_TEMP'
promq 'DCGM_FI_DEV_POWER_USAGE'

Expected (abbreviated, not tested live):

NAME                           TYPE        CLUSTER-IP     PORT(S)
service/nvidia-dcgm-exporter   ClusterIP   10.43.88.12    9400/TCP
NAME                                                        AGE
servicemonitor.monitoring.coreos.com/nvidia-dcgm-exporter   12m
rke2-gpu-1 0
rke2-gpu-1 0.0004
rke2-gpu-1 34
rke2-gpu-1 12.1
Metric Meaning Alert from (rule of thumb)
DCGM_FI_DEV_GPU_UTIL compute utilization in % permanently 100% + queue
DCGM_FI_DEV_FB_USED used VRAM in MiB vLLM deliberately occupies ~90%
DCGM_FI_DEV_GPU_TEMP temperature in °C > 85 °C
DCGM_FI_DEV_XID_ERRORS last Xid error any value > 0

Why Hostname? DCGM sets the label to the node name. Even with time-slicing you only see one physical GPU – the mapping to pods comes through the pod/namespace labels as soon as a pod holds the GPU.

Save your progress? Sign up for free

Drill 8: MIG check and DRA outlook

MIG (Multi-Instance GPU) splits A100/H100/H200/B200 into isolated instances with their own memory. An L4 or RTX can’t do that:

ssh [email protected] nvidia-smi --query-gpu=name,mig.mode.current --format=csv

Expected (not tested live): NVIDIA L4, [N/A] after the CSV header line.

With a MIG card, you set mig.strategy: single in the chart and label the node with nvidia.com/mig.config=all-1g.10gb. The MIG Manager then partitions it and the device plugin reports the instances.

DRA (Dynamic Resource Allocation, resource.k8s.io/v1, GA since Kubernetes 1.34) replaces nvidia.com/gpu: 1 with claims that filter on attributes (model, memory). In GPU Operator v26.7, that’s the experimental GPUCluster mode. It excludes the ClusterPolicy, and there’s no in-place switch. So you rebuild, and switch back again at the end.

cat > gpu-operator-dra-values.yaml <<'EOF'
# DRA mode (experimental in v26.7): GPUCluster instead of ClusterPolicy
# No in-place switch: uninstall the Operator first!
# Without device plugin and toolkit: the DRA driver provides CDI specs itself.
clusterPolicy:
  deployCR: false
gpuCluster:
  deployCR: true
driver:
  enabled: false            # driver from the role nvidia_host (>= 580)
draDriver:
  computeDomains:
    enabled: false          # only for multi-node NVLink (GB200 & co.)
EOF
kubectl delete ns gpu-test --wait
helm uninstall gpu-operator -n gpu-operator --wait
helm install gpu-operator nvidia/gpu-operator --version v26.7.1 \
  -n gpu-operator -f gpu-operator-dra-values.yaml --wait
kubectl get gpucluster gpu-cluster
kubectl get deviceclass
kubectl get resourceslice -o json | jq -r '.items[].spec.devices[]
  | "\(.name) \(.attributes.productName.string) \(.capacity.memory.value)"'

Expected (rendering and GPUCluster dry run tested live, the rest not tested live):

NAME          STATUS   AGE
gpu-cluster   ready    3m
NAME                  AGE
gpu.nvidia.com        3m
mig.nvidia.com        3m
vfio.gpu.nvidia.com   3m
gpu-0 NVIDIA L4 23034Mi

The claim requests “a GPU with more than 16 GiB” – via CEL instead of a node label:

kubectl create ns gpu-test
cat > dra-test.yaml <<'EOF'
# DRA (Day 10, Drill 8): GPU via ResourceClaim instead of nvidia.com/gpu
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-gpu
  namespace: gpu-test
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            selectors:
              - cel:
                  expression: >-
                    device.capacity['gpu.nvidia.com'].memory
                    .isGreaterThan(quantity('16Gi'))
---
apiVersion: v1
kind: Pod
metadata:
  name: dra-smi
  namespace: gpu-test
spec:
  restartPolicy: Never
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: smi
      image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
      command: ["nvidia-smi", "-L"]
      resources:
        claims:
          - name: gpu
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: single-gpu
EOF
kubectl apply -f dra-test.yaml
kubectl -n gpu-test wait pod/dra-smi --for=jsonpath='{.status.phase}'=Succeeded \
  --timeout=180s
kubectl -n gpu-test logs dra-smi
kubectl -n gpu-test get resourceclaims

Expected (dry run tested live, output not tested live):

resourceclaimtemplate.resource.k8s.io/single-gpu created
pod/dra-smi created
pod/dra-smi condition met
GPU 0: NVIDIA L4 (UUID: GPU-5c1f...)
NAME                STATE                AGE
dra-smi-gpu-x8k2q   allocated,reserved   40s

Back to device plugin mode with time-slicing – chapters 11–14 build on it:

kubectl delete ns gpu-test
helm uninstall gpu-operator -n gpu-operator --wait
helm install gpu-operator nvidia/gpu-operator --version v26.7.1 -n gpu-operator \
  -f gpu-operator-values.yaml -f gpu-operator-timeslicing-values.yaml --wait
kubectl get node rke2-gpu-1 \
  -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'

Expected (not tested live):

4

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Taint nvidia.com/gpu auf rke2-gpu-1" 'kubectl get node rke2-gpu-1 \
  -o jsonpath="{.spec.taints[*].key}" | grep -q nvidia.com/gpu'
check "Operator-Validator Running" '[ "$(kubectl -n gpu-operator get pods \
  -l app=nvidia-operator-validator -o jsonpath="{.items[0].status.phase}")" \
  = Running ]'
check "GFD-Label gpu.present" '[ "$(kubectl get node rke2-gpu-1 -o \
  jsonpath="{.metadata.labels.nvidia\.com/gpu\.present}")" = true ]'
check "Time-Slicing: 4 GPUs" '[ "$(kubectl get node rke2-gpu-1 -o \
  jsonpath="{.status.allocatable.nvidia\.com/gpu}")" = 4 ]'
check "ServiceMonitor DCGM" 'kubectl -n gpu-operator get servicemonitor \
  nvidia-dcgm-exporter'
check "DCGM-Metrik in Prometheus" '[ -n "$(promq DCGM_FI_DEV_GPU_TEMP)" ]'

Expected:

✔ Taint nvidia.com/gpu auf rke2-gpu-1
✔ Operator-Validator Running
✔ GFD-Label gpu.present
✔ Time-Slicing: 4 GPUs
✔ ServiceMonitor DCGM
✔ DCGM-Metrik in Prometheus

The last check needs the promq function from Drill 7 (source it in the same shell).

Challenge

A second GPU node should later get only 2 slots. Create a second profile l4-2 in the same ConfigMap and assign it via node label only to rke2-gpu-1 – for testing.

Hint: The device plugin reads the profile from the node label nvidia.com/device-plugin.config; default: any applies only to nodes without the label.

Solution

kubectl -n gpu-operator patch configmap time-slicing-config --type merge -p '
data:
  l4-2: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 2
'
kubectl label node rke2-gpu-1 nvidia.com/device-plugin.config=l4-2 --overwrite
sleep 60
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
kubectl label node rke2-gpu-1 nvidia.com/device-plugin.config-

Expected (not tested live):

2

Save your progress? Sign up for free

Enterprise Note

  • Version driver, toolkit and kernel together and test them on a canary GPU node. With the Operator driver, use precompiled images (driver.usePrecompiled) for reproducible rollouts.
  • Time-slicing only for dev/test or clearly budgeted inference. For tenant isolation, MIG (A100/H100) or dedicated GPUs, plus ResourceQuotas on requests.nvidia.com/gpu per namespace.
  • DRA is the future (filters on memory/model, shared claims, dynamic MIG). In the GPU Operator, the GPUCluster mode is still experimental – plan it for production only once NVIDIA releases it.
  • DCGM alerts: Xid errors, ECC errors, temperature, throttling (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS). Manage GPU nodes via node pools and the cluster autoscaler (cloud) or capacity planning (on-prem).
  • Hardening: the CIS profile and PSA restricted need exceptions for the gpu-operator namespace (privileged DaemonSets) – document and audit them.