Skip to content
Hands-on IT IT Drills

Day 14: Nodes

Goal: You recognize when a fault lies in the node rather than in the Pod, investigate the kubelet and the runtime on the node, and know the deadlines after which Kubernetes replaces the Pods of a failed node. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • The kubelet checks in via a Lease in kube-node-lease. If it stays silent for 50 s, the controller manager sets the conditions to Unknown and adds a taint.
  • Unknown means: the kubelet is silent. False means: it is running and reports a problem itself.
  • Pods tolerate an unreachable node for 300 s, then they are marked for deletion – and hang in Terminating until the kubelet is back.
Fault Symptom Where the answer is
Kubelet not running Node NotReady Lease, journal, kubelet.log
Node failure Replacement only after ~6 min Taints, tolerations
Local storage full Evicted, ever new Pods .status.message
Drain hangs Cannot evict pod ALLOWED DISRUPTIONS

Warning: Today you stop rke2-agent on worker2. The containers keep running, but for Kubernetes the node is gone. Every drill that stops it starts it again at the end.

Drill 1: The toolbox

On an RKE2 node, the kubelet and containerd are child processes of rke2-agent (on cp1: rke2-server). The four questions to ask a node:

# on cp1
mkdir -p ~/debug/tag14 && cd ~/debug/tag14
ssh -n worker2 systemctl status rke2-agent --no-pager | sed -n '1,3p;8,9p' \
  | cut -c1-80
ssh -n worker2 sudo journalctl -u rke2-agent -b --no-pager | tail -2 | cut -c1-88
ssh -n worker2 sudo grep '^E' /var/lib/rancher/rke2/agent/logs/kubelet.log \
  | tail -1 | cut -c1-88
ssh -n worker2 sudo -i crictl info | jq -c '.status.conditions[] | {type, status}'

Expected (abridged):

● rke2-agent.service - Rancher Kubernetes Engine v2 (agent)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-agent.service; enabled;
     Active: active (running) since Fri 2026-11-20 11:07:38 UTC; 3min 1s ago
   Main PID: 156459 (rke2)
      Tasks: 156
Nov 20 11:07:38 worker2 rke2[156459]: time="2026-11-20T11:07:38Z" level=info msg="rke2 a
Nov 20 11:07:38 worker2 systemd[1]: Started rke2-agent.service - Rancher Kubernetes Engi
E1120 11:07:25.187534  156608 remote_runtime.go:657] "ContainerStatus from runtime servi
{"type":"RuntimeReady","status":true}
{"type":"NetworkReady","status":true}
{"type":"ContainerdHasNoDeprecationWarnings","status":true}

Service, journal, kubelet log as a file, and the runtime. Individual E lines in the kubelet log are normal; what counts is what matches the time of the problem.

The API server delivers the running kubelet configuration, no SSH needed:

# on cp1
kubectl get --raw /api/v1/nodes/worker2/proxy/configz \
  | jq -c '.kubeletconfig | {evictionHard, nodeStatusUpdateFrequency, maxPods}'

Expected:

{"evictionHard":{"imagefs.available":"5%","nodefs.available":"5%"},
  "nodeStatusUpdateFrequency":"10s","maxPods":110}

RKE2 sets both thresholds to 5 % – Kubernetes itself uses 10 % (nodefs) and 15 % (imagefs); if you look it up online, you will find different numbers. memory.available (Kubernetes default 100Mi) is missing entirely: according to the Kubernetes documentation, if you set individual hard thresholds, you lose the other defaults. That is why on RKE2 the kubelet does not evict because of low memory – only the OOM killer takes care of that.

Save your progress? Sign up for free

Drill 2: The node goes NotReady

Break it:

Warning: systemctl stop rke2-agent takes the node out of the cluster. Lab only.

# on cp1
ssh -n worker2 sudo systemctl stop rke2-agent

Symptom: Time and Lease every 10 seconds:

# on cp1
for i in $(seq 7); do
  echo "$(date +%T) $(kubectl get node worker2 --no-headers | awk '{print $2}')" \
    "$(kubectl -n kube-node-lease get lease worker2 -o jsonpath='{.spec.renewTime}')"
  sleep 10
done

Expected:

11:10:54 Ready 2026-11-20T11:10:46.265098Z
11:11:04 Ready 2026-11-20T11:10:46.265098Z
11:11:15 Ready 2026-11-20T11:10:46.265098Z
11:11:26 Ready 2026-11-20T11:10:46.265098Z
11:11:36 Ready 2026-11-20T11:10:46.265098Z
11:11:47 NotReady 2026-11-20T11:10:46.265098Z
11:11:57 NotReady 2026-11-20T11:10:46.265098Z

The Lease is no longer renewed; about 50 seconds later (node-monitor-grace-period) the status flips.

Investigate:

# on cp1
kubectl describe node worker2 | grep -A1 '^Taints'
J='{range .status.conditions[*]}{.type} {.status} {.reason}{"\n"}{end}'
kubectl get node worker2 -o jsonpath="$J"
ssh -n worker2 systemctl is-active rke2-agent
ssh -n worker2 sudo journalctl -u rke2-agent -b --no-pager | grep Stopped | tail -1 \
  | cut -c1-88

Expected:

Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule
NetworkUnavailable False FlannelIsUp
MemoryPressure Unknown NodeStatusUnknown
DiskPressure Unknown NodeStatusUnknown
PIDPressure Unknown NodeStatusUnknown
Ready Unknown NodeStatusUnknown
inactive
Nov 20 11:10:51 worker2 systemd[1]: Stopped rke2-agent.service - Rancher Kubernetes Engi

Cause: The kubelet is not running because someone stopped rke2-agent (Stopped in the journal; a crash would have left an exit code and restarts). Without a kubelet, nobody renews the Lease; the node lifecycle controller sets Unknown and the taint unreachable. NetworkUnavailable False comes from Canal, not from the kubelet – it stays as it is. kubectl debug node/worker2 does not help here: the debug Pod would have to be started by exactly this kubelet.

Fix:

# on cp1
ssh -n worker2 sudo systemctl start rke2-agent

Verify:

# on cp1
kubectl wait --for=condition=Ready node/worker2 --timeout=120s
kubectl describe node worker2 | grep '^Taints'

Expected:

node/worker2 condition met
Taints:             <none>

Save your progress? Sign up for free

Drill 3: Pods hang after the failure

Break it: Two web Pods that preferably run on worker2:

# on cp1
kubectl create namespace tag14
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag14}
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - {key: kubernetes.io/hostname, operator: In, values: [worker2]}
      containers:
      - name: web
        image: traefik/whoami:v1.11
EOF
kubectl apply -f web.yaml
kubectl -n tag14 rollout status deploy/web
kubectl -n tag14 get pod -l app=web -o jsonpath='{.items[0].spec.tolerations}' \
  | jq -c '.[]'

Expected (lines wrapped):

namespace/tag14 created
deployment.apps/web created
deployment "web" successfully rolled out
{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists",
  "tolerationSeconds":300}
{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists",
  "tolerationSeconds":300}

The manifest contains no toleration – both are added by the admission plugin DefaultTolerationSeconds.

Symptom: The node fails, and you check back after two minutes.

Warning: The next block stops worker2 again; the drill takes six minutes in total.

# on cp1
ssh -n worker2 sudo systemctl stop rke2-agent
sleep 120
kubectl -n tag14 get pods -o wide | cut -c1-80

Expected: worker2 has long been NotReady, yet the Pods are supposedly healthy:

NAME                   READY   STATUS    RESTARTS   AGE    IP            NODE
web-57764fdd87-2l7mv   1/1     Running   0          2m7s   10.42.2.120   worker2
web-57764fdd87-9fms4   1/1     Running   0          2m7s   10.42.2.119   worker2

Four minutes later:

# on cp1
sleep 240
kubectl -n tag14 get pods -o wide | cut -c1-80

Expected:

NAME                   READY   STATUS        RESTARTS   AGE     IP            NODE
web-57764fdd87-2l7mv   1/1     Terminating   0          6m10s   10.42.2.120   worker2
web-57764fdd87-9fms4   1/1     Terminating   0          6m10s   10.42.2.119   worker2
web-57764fdd87-f8v7n   1/1     Running       0          9s      10.42.0.157   cp1
web-57764fdd87-z8whd   1/1     Running       0          9s      10.42.1.120   worker1

Investigate: Take the Pod name from your output:

# on cp1
kubectl -n tag14 events --for pod/web-57764fdd87-2l7mv | tail -2 | tr -s ' ' \
  | cut -c1-88

Expected:

5m13s Warning NodeNotReady Pod/web-57764fdd87-2l7mv Node is not ready
13s Normal TaintManagerEviction Pod/web-57764fdd87-2l7mv Marking for deletion Pod tag14/

Cause: 50 s until NotReady, then 300 s of toleration: only after almost six minutes does the taint eviction controller mark the Pods for deletion. Only the kubelet of worker2 can confirm that – so they hang in Terminating. The ReplicaSet controller starts the replacements anyway, because it no longer counts Pods with a deletionTimestamp. The waiting time is intentional: Kubernetes cannot tell a dead node from a network disruption; for a StatefulSet, no replacement is created at all until the old Pod is really gone.

Fix: Bring the node back; afterwards the stateless web gets a shorter deadline:

# on cp1
ssh -n worker2 sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/worker2 --timeout=120s
sleep 15
kubectl -n tag14 get pods -l app=web | grep -c Terminating
cat >> web.yaml <<'EOF'
      tolerations:
      - {key: node.kubernetes.io/unreachable, operator: Exists, effect: NoExecute,
         tolerationSeconds: 30}
      - {key: node.kubernetes.io/not-ready, operator: Exists, effect: NoExecute,
         tolerationSeconds: 30}
EOF
kubectl apply -f web.yaml
kubectl -n tag14 rollout status deploy/web

Expected (abridged):

node/worker2 condition met
0
deployment.apps/web configured
deployment "web" successfully rolled out

Verify: The same failure, now with a stopwatch:

# on cp1
sleep 10
kubectl -n tag14 get pods -l app=web -o wide | cut -c1-80
ssh -n worker2 sudo systemctl stop rke2-agent
T0=$(date +%s)
until kubectl -n tag14 get pods -l app=web | grep -q Terminating; do sleep 2; done
echo "Ersatz nach $(( $(date +%s) - T0 )) s"
ssh -n worker2 sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/worker2 --timeout=120s

Expected (abridged):

NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE
web-846654ffcc-h6fhz   1/1     Running   0          17s   10.42.2.121   worker2
web-846654ffcc-nm4r9   1/1     Running   0          15s   10.42.2.122   worker2
Ersatz nach 94 s
node/worker2 condition met

Ersatz nach 94 s means ‘replacement after 94 s’: 50 s from the last Lease renewal (up to 10 s before the stop) plus 30 s of toleration plus the controllers’ sync interval – 75 to 95 seconds depending on the run, instead of six minutes. If the toleration is already in the manifest, the API server does not add its own.

Save your progress? Sign up for free

Drill 4: Pods get evicted

Break it: A Pod writes 200 MiB into an emptyDir, while only 100 MiB of local storage (ephemeral-storage) are allowed – without the node itself being full:

# on cp1
cat > fueller.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: fueller, namespace: tag14}
spec:
  replicas: 1
  selector: {matchLabels: {app: fueller}}
  template:
    metadata: {labels: {app: fueller}}
    spec:
      nodeSelector: {kubernetes.io/hostname: worker1}
      containers:
      - name: fueller
        image: busybox:1.37
        command:
        - sh
        - -c
        - dd if=/dev/zero of=/daten/block bs=1M count=200; sleep infinity
        resources: {limits: {ephemeral-storage: 100Mi}}
        volumeMounts: [{name: daten, mountPath: /daten}]
      volumes:
      - {name: daten, emptyDir: {}}
EOF
kubectl apply -f fueller.yaml

Symptom (after about two minutes):

# on cp1
sleep 120
kubectl -n tag14 get pods -l app=fueller

Expected:

NAME                       READY   STATUS    RESTARTS   AGE
fueller-6cd6db4c95-475sz   1/1     Running   0          25s
fueller-6cd6db4c95-pfvdp   0/1     Error     0          90s
fueller-6cd6db4c95-vqjmq   0/1     Error     0          2m3s

Ever new Pods, while the old ones are left lying around. Depending on the timing, you may also see ContainerStatusUnknown.

Investigate: STATUS shows the container. The reason is in the Pod status:

# on cp1
kubectl -n tag14 get pods -l app=fueller \
  -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.reason
kubectl -n tag14 get pods -l app=fueller --field-selector=status.phase=Failed \
  -o jsonpath='{.items[0].status.message}{"\n"}'

Expected:

NAME                       PHASE     REASON
fueller-6cd6db4c95-475sz   Running   <none>
fueller-6cd6db4c95-pfvdp   Failed    Evicted
fueller-6cd6db4c95-vqjmq   Failed    Evicted
Pod ephemeral local storage usage exceeds the total limit of containers 100Mi.

Cause: The emptyDir lives on the node’s disk and counts toward the Pod’s local storage. The kubelet’s eviction manager terminates the Pod without a grace period; an evicted Pod does not restart, it stays Failed, and the Deployment creates a new one that does the same. Under real node pressure, the message would read The node was low on resource: ephemeral-storage., and the node would show DiskPressure=True. You know the case with sizeLimit on the volume from Day 10.

Fix: Set the limit to what is needed, additionally cap the volume, clean up the leftovers:

# on cp1
sed -i 's/ephemeral-storage: 100Mi/ephemeral-storage: 300Mi/' fueller.yaml
sed -i 's/emptyDir: {}/emptyDir: {sizeLimit: 250Mi}/' fueller.yaml
kubectl apply -f fueller.yaml
kubectl -n tag14 rollout status deploy/fueller
kubectl -n tag14 delete pods --field-selector=status.phase=Failed

Verify (after one minute):

# on cp1
sleep 60
kubectl -n tag14 get pods -l app=fueller

Expected:

NAME                       READY   STATUS    RESTARTS   AGE
fueller-795b88576b-6gqr8   1/1     Running   0          65s

Otherwise, evicted Pods are only cleaned up by the Pod garbage collector – starting at 1000 terminated Pods (RKE2’s value for terminated-pod-gc-threshold in the controller manager; the Kubernetes default is 12,500).

Save your progress? Sign up for free

Drill 5: The drain hangs

Break it: Two api Pods preferably on worker1, and a PodDisruptionBudget that requires both “just to be safe”:

# on cp1
kubectl -n tag14 delete deploy fueller
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag14}
spec:
  replicas: 2
  selector: {matchLabels: {app: api}}
  template:
    metadata: {labels: {app: api}}
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - {key: kubernetes.io/hostname, operator: In, values: [worker1]}
      containers:
      - name: api
        image: traefik/whoami:v1.11
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: api, namespace: tag14}
spec:
  minAvailable: 2
  selector: {matchLabels: {app: api}}
EOF
kubectl apply -f api.yaml
kubectl -n tag14 rollout status deploy/api

Symptom:

# on cp1
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 \
  | grep -E 'cordoned|api|timeout' | cut -c1-88 | sort | uniq -c

Expected (abridged):

      6 error when evicting pods/"api-599cf68b74-dznbc" -n "tag14" (will retry after 5s
      1 error when evicting pods/"api-599cf68b74-dznbc" -n "tag14": global timeout reach
      6 error when evicting pods/"api-599cf68b74-qgknb" -n "tag14" (will retry after 5s
      1 error when evicting pods/"api-599cf68b74-qgknb" -n "tag14": global timeout reach
      7 evicting pod tag14/api-599cf68b74-dznbc
      7 evicting pod tag14/api-599cf68b74-qgknb
      1 node/worker1 cordoned

The same attempt every five seconds until the timeout kicks in; the full message ends with Cannot evict pod as it would violate the pod's disruption budget.

Investigate:

# on cp1
kubectl -n tag14 get pdb api
kubectl get node worker1

Expected:

NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    2               N/A               0                     39s
NAME      STATUS                     ROLES    AGE     VERSION
worker1   Ready,SchedulingDisabled   <none>   4h52m   v1.37.2+rke2r1

Cause: minAvailable: 2 with two replicas allows zero voluntary disruptions; the Eviction API answers with 429, and kubectl drain keeps retrying endlessly. And the aborted drain left the node cordoned – new Pods never land there, without any error showing up.

Fix:

# on cp1
sed -i 's/minAvailable: 2/maxUnavailable: 1/' api.yaml
kubectl apply -f api.yaml
kubectl -n tag14 get pdb api
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data --timeout=120s 2>&1 \
  | grep -E 'drained|evicted'
kubectl uncordon worker1

Expected (abridged):

deployment.apps/api unchanged
poddisruptionbudget.policy/api configured
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    N/A             1                 1                     44s
pod/api-599cf68b74-qgknb evicted
pod/api-599cf68b74-dznbc evicted
node/worker1 drained
node/worker1 uncordoned

Verify:

# on cp1
kubectl get nodes
kubectl -n tag14 get pods -l app=api -o wide | cut -c1-80

Expected:

NAME      STATUS   ROLES                AGE     VERSION
cp1       Ready    control-plane,etcd   4h55m   v1.37.2+rke2r1
worker1   Ready    <none>               4h52m   v1.37.2+rke2r1
worker2   Ready    <none>               4h51m   v1.37.2+rke2r1
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE
api-599cf68b74-mbxm5   1/1     Running   0          11s   10.42.2.126   worker2
api-599cf68b74-q5hxq   1/1     Running   0          5s    10.42.0.159   cp1

The drain also evicts other Pods from worker1 – in the lab, for example, CoreDNS or Kyverno controllers. They move elsewhere; after uncordon they do not come back on their own.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show for all nodes when their Lease was last renewed. Target time: 30 s
  2. Show the taints of all nodes in a table. Target time: 30 s
  3. Find all evicted Pods in the entire cluster. Target time: 30 s
  4. Show the kubelet’s maxPods on worker1 without logging in. Target time: 30 s

Solution

# on cp1
kubectl -n kube-node-lease get lease \
  -o custom-columns=NAME:.metadata.name,RENEW:.spec.renewTime
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl get pods -A --field-selector=status.phase=Failed \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.reason
kubectl get --raw /api/v1/nodes/worker1/proxy/configz | jq .kubeletconfig.maxPods

Expected:

NAME      RENEW
cp1       2026-11-20T11:26:35.670313Z
worker1   2026-11-20T11:26:31.395700Z
worker2   2026-11-20T11:26:32.608976Z
NAME      TAINTS
cp1       <none>
worker1   <none>
worker2   <none>
NS    NAME   REASON
110

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl get nodes
kubectl -n tag14 get pdb
kubectl -n tag14 get deploy

Expected:

NAME      STATUS   ROLES                AGE     VERSION
cp1       Ready    control-plane,etcd   4h55m   v1.37.2+rke2r1
worker1   Ready    <none>               4h52m   v1.37.2+rke2r1
worker2   Ready    <none>               4h51m   v1.37.2+rke2r1
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    N/A             1                 1                     63s
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    2/2     2            2           63s
web    2/2     2            2           13m

Challenge

A colleague raised the kubelet’s Pod limit on worker2 and restarted the agent. Since then the node has been NotReady, even though systemctl reports the service as active (running). Find the cause and fix it.

# on cp1
ssh -n worker2 sudo mkdir -p /etc/rancher/rke2/config.yaml.d
echo 'kubelet-arg: [max-pod=150]' \
  | ssh worker2 sudo tee /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
ssh -n worker2 sudo systemctl restart rke2-agent

Hint: The service is rke2-agent – the kubelet is just a child process.

Solution

After about a minute:

# on cp1
sleep 60
kubectl get node worker2
ssh -n worker2 systemctl is-active rke2-agent
ssh -n worker2 sudo journalctl -u rke2-agent -b --since -2min --no-pager \
  | grep 'Kubelet exited' | tail -1 | cut -c1-88
L=/var/lib/rancher/rke2/agent/logs/kubelet.log
ssh -n worker2 "sudo grep 'command failed' $L" | tail -1 | cut -c1-120

Expected (line wrapped):

NAME      STATUS     ROLES    AGE     VERSION
worker2   NotReady   <none>   4h56m   v1.37.2+rke2r1
active
Nov 20 11:32:08 worker2 rke2[170810]: time="2026-11-20T11:32:08Z" level=error msg="Kubel
E1120 11:32:13.611217  171561 run.go:72] "command failed" err="failed to parse kubelet
  flag: unknown flag: --max-pod"

rke2-agent is running and restarts the kubelet every five seconds (Kubelet exited: exit status 1); the kubelet aborts while parsing its flags: max-pods was meant. RKE2 passes kubelet-arg through unchecked – the same happens with flags that an upgrade has removed.

# on cp1
ssh -n worker2 sudo rm /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
ssh -n worker2 sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/worker2 --timeout=120s

Expected:

node/worker2 condition met

Talos: The same typo via KubeletConfig, investigated with talosctl service kubelet and talosctl logs kubelet (Day 19).

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag14
ssh -n worker2 sudo rmdir /etc/rancher/rke2/config.yaml.d
kubectl get nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Expected:

namespace "tag14" deleted
NAME      STATUS   ROLES                AGE     VERSION
cp1       Ready    control-plane,etcd   4h56m   v1.37.2+rke2r1
worker1   Ready    <none>               4h54m   v1.37.2+rke2r1
worker2   Ready    <none>               4h52m   v1.37.2+rke2r1
NAME      TAINTS
cp1       <none>
worker1   <none>
worker2   <none>

Interview Note

  • Node NotReady – what first? Read the conditions: Unknown → the kubelet is silent (service, network, node gone); False with a message → the kubelet is running and names the problem.
  • Why does the replacement take so long? 50 s node-monitor-grace-period plus 300 s tolerationSeconds from DefaultTolerationSeconds; the old Pod hangs in Terminating until the kubelet is back or the Node object is deleted.
  • Drain hangs? kubectl get pdb, column ALLOWED DISRUPTIONS; after an abort, do not forget to uncordon.

Enterprise Note

  • A permanently lost node is removed with kubectl delete node; only then does the garbage collector clean up its Pods. For StatefulSets, only when it is certain that the node is really off (fencing). Since Kubernetes 1.28 it also works without deleting: after fencing, set the taint node.kubernetes.io/out-of-service=nodeshutdown:NoExecute – Pods and volumes are released immediately.
  • Shorter tolerationSeconds only for stateless services; monitoring alerts on NotReady long before the 300 s are up.
  • PDBs with maxUnavailable instead of minAvailable = replica count; before any maintenance, look for kubectl get pdb -A entries with ALLOWED DISRUPTIONS 0.
  • ephemeral-storage requests and limits for all Pods, otherwise the kubelet evicts in its own order under disk pressure.
  • Manage kubelet arguments in RKE2 centrally via config.yaml (Ansible, Rancher), and test on one node before rolling out.