Day 14: Nodes
Goal: You recognize when a fault lies in the node rather than in the Pod, investigate the kubelet and the runtime on the node, and know the deadlines after which Kubernetes replaces the Pods of a failed node. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- The kubelet checks in via a Lease in
kube-node-lease. If it stays silent for 50 s, the controller manager sets the conditions toUnknownand adds a taint. Unknownmeans: the kubelet is silent.Falsemeans: it is running and reports a problem itself.- Pods tolerate an unreachable node for 300 s, then they are marked
for deletion – and hang in
Terminatinguntil the kubelet is back.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Kubelet not running | Node NotReady |
Lease, journal, kubelet.log |
| Node failure | Replacement only after ~6 min | Taints, tolerations |
| Local storage full | Evicted, ever new Pods |
.status.message |
| Drain hangs | Cannot evict pod |
ALLOWED DISRUPTIONS |
Warning: Today you stop
rke2-agentonworker2. The containers keep running, but for Kubernetes the node is gone. Every drill that stops it starts it again at the end.
Drill 1: The toolbox
On an RKE2 node, the kubelet and containerd are child processes of
rke2-agent (on cp1: rke2-server).
The four questions to ask a node:
# on cp1
mkdir -p ~/debug/tag14 && cd ~/debug/tag14
ssh -n worker2 systemctl status rke2-agent --no-pager | sed -n '1,3p;8,9p' \
| cut -c1-80
ssh -n worker2 sudo journalctl -u rke2-agent -b --no-pager | tail -2 | cut -c1-88
ssh -n worker2 sudo grep '^E' /var/lib/rancher/rke2/agent/logs/kubelet.log \
| tail -1 | cut -c1-88
ssh -n worker2 sudo -i crictl info | jq -c '.status.conditions[] | {type, status}'Expected (abridged):
● rke2-agent.service - Rancher Kubernetes Engine v2 (agent)
Loaded: loaded (/usr/local/lib/systemd/system/rke2-agent.service; enabled;
Active: active (running) since Fri 2026-11-20 11:07:38 UTC; 3min 1s ago
Main PID: 156459 (rke2)
Tasks: 156
Nov 20 11:07:38 worker2 rke2[156459]: time="2026-11-20T11:07:38Z" level=info msg="rke2 a
Nov 20 11:07:38 worker2 systemd[1]: Started rke2-agent.service - Rancher Kubernetes Engi
E1120 11:07:25.187534 156608 remote_runtime.go:657] "ContainerStatus from runtime servi
{"type":"RuntimeReady","status":true}
{"type":"NetworkReady","status":true}
{"type":"ContainerdHasNoDeprecationWarnings","status":true}
Service, journal, kubelet log as a file, and the runtime. Individual
E lines in the kubelet log are normal; what counts is what
matches the time of the problem.
The API server delivers the running kubelet configuration, no SSH needed:
# on cp1
kubectl get --raw /api/v1/nodes/worker2/proxy/configz \
| jq -c '.kubeletconfig | {evictionHard, nodeStatusUpdateFrequency, maxPods}'Expected:
{"evictionHard":{"imagefs.available":"5%","nodefs.available":"5%"},
"nodeStatusUpdateFrequency":"10s","maxPods":110}
RKE2 sets both thresholds to 5 % – Kubernetes itself uses 10 %
(nodefs) and 15 % (imagefs); if you look it up
online, you will find different numbers. memory.available
(Kubernetes default 100Mi) is missing entirely: according to the
Kubernetes documentation, if you set individual hard thresholds, you
lose the other defaults. That is why on RKE2 the kubelet does not evict
because of low memory – only the OOM killer takes care of that.
Save your progress? Sign up for free
Drill 2: The node goes NotReady
Break it:
Warning:
systemctl stop rke2-agenttakes the node out of the cluster. Lab only.
# on cp1
ssh -n worker2 sudo systemctl stop rke2-agentSymptom: Time and Lease every 10 seconds:
# on cp1
for i in $(seq 7); do
echo "$(date +%T) $(kubectl get node worker2 --no-headers | awk '{print $2}')" \
"$(kubectl -n kube-node-lease get lease worker2 -o jsonpath='{.spec.renewTime}')"
sleep 10
doneExpected:
11:10:54 Ready 2026-11-20T11:10:46.265098Z
11:11:04 Ready 2026-11-20T11:10:46.265098Z
11:11:15 Ready 2026-11-20T11:10:46.265098Z
11:11:26 Ready 2026-11-20T11:10:46.265098Z
11:11:36 Ready 2026-11-20T11:10:46.265098Z
11:11:47 NotReady 2026-11-20T11:10:46.265098Z
11:11:57 NotReady 2026-11-20T11:10:46.265098Z
The Lease is no longer renewed; about 50 seconds later
(node-monitor-grace-period) the status flips.
Investigate:
# on cp1
kubectl describe node worker2 | grep -A1 '^Taints'
J='{range .status.conditions[*]}{.type} {.status} {.reason}{"\n"}{end}'
kubectl get node worker2 -o jsonpath="$J"
ssh -n worker2 systemctl is-active rke2-agent
ssh -n worker2 sudo journalctl -u rke2-agent -b --no-pager | grep Stopped | tail -1 \
| cut -c1-88Expected:
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
NetworkUnavailable False FlannelIsUp
MemoryPressure Unknown NodeStatusUnknown
DiskPressure Unknown NodeStatusUnknown
PIDPressure Unknown NodeStatusUnknown
Ready Unknown NodeStatusUnknown
inactive
Nov 20 11:10:51 worker2 systemd[1]: Stopped rke2-agent.service - Rancher Kubernetes Engi
Cause: The kubelet is not running because someone
stopped rke2-agent (Stopped in the journal; a
crash would have left an exit code and restarts). Without a kubelet,
nobody renews the Lease; the node lifecycle controller sets
Unknown and the taint unreachable.
NetworkUnavailable False comes from Canal, not from the
kubelet – it stays as it is. kubectl debug node/worker2
does not help here: the debug Pod would have to be started by exactly
this kubelet.
Fix:
# on cp1
ssh -n worker2 sudo systemctl start rke2-agentVerify:
# on cp1
kubectl wait --for=condition=Ready node/worker2 --timeout=120s
kubectl describe node worker2 | grep '^Taints'Expected:
node/worker2 condition met
Taints: <none>
Save your progress? Sign up for free
Drill 3: Pods hang after the failure
Break it: Two web Pods that preferably
run on worker2:
# on cp1
kubectl create namespace tag14
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag14}
spec:
replicas: 2
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- {key: kubernetes.io/hostname, operator: In, values: [worker2]}
containers:
- name: web
image: traefik/whoami:v1.11
EOF
kubectl apply -f web.yaml
kubectl -n tag14 rollout status deploy/web
kubectl -n tag14 get pod -l app=web -o jsonpath='{.items[0].spec.tolerations}' \
| jq -c '.[]'Expected (lines wrapped):
namespace/tag14 created
deployment.apps/web created
deployment "web" successfully rolled out
{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists",
"tolerationSeconds":300}
{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists",
"tolerationSeconds":300}
The manifest contains no toleration – both are added by the admission
plugin DefaultTolerationSeconds.
Symptom: The node fails, and you check back after two minutes.
Warning: The next block stops
worker2again; the drill takes six minutes in total.
# on cp1
ssh -n worker2 sudo systemctl stop rke2-agent
sleep 120
kubectl -n tag14 get pods -o wide | cut -c1-80Expected: worker2 has long been
NotReady, yet the Pods are supposedly healthy:
NAME READY STATUS RESTARTS AGE IP NODE
web-57764fdd87-2l7mv 1/1 Running 0 2m7s 10.42.2.120 worker2
web-57764fdd87-9fms4 1/1 Running 0 2m7s 10.42.2.119 worker2
Four minutes later:
# on cp1
sleep 240
kubectl -n tag14 get pods -o wide | cut -c1-80Expected:
NAME READY STATUS RESTARTS AGE IP NODE
web-57764fdd87-2l7mv 1/1 Terminating 0 6m10s 10.42.2.120 worker2
web-57764fdd87-9fms4 1/1 Terminating 0 6m10s 10.42.2.119 worker2
web-57764fdd87-f8v7n 1/1 Running 0 9s 10.42.0.157 cp1
web-57764fdd87-z8whd 1/1 Running 0 9s 10.42.1.120 worker1
Investigate: Take the Pod name from your output:
# on cp1
kubectl -n tag14 events --for pod/web-57764fdd87-2l7mv | tail -2 | tr -s ' ' \
| cut -c1-88Expected:
5m13s Warning NodeNotReady Pod/web-57764fdd87-2l7mv Node is not ready
13s Normal TaintManagerEviction Pod/web-57764fdd87-2l7mv Marking for deletion Pod tag14/
Cause: 50 s until NotReady, then 300 s
of toleration: only after almost six minutes does the taint eviction
controller mark the Pods for deletion. Only the kubelet of
worker2 can confirm that – so they hang in
Terminating. The ReplicaSet controller starts the
replacements anyway, because it no longer counts Pods with a
deletionTimestamp. The waiting time is intentional:
Kubernetes cannot tell a dead node from a network disruption; for a
StatefulSet, no replacement is created at all until the old Pod is
really gone.
Fix: Bring the node back; afterwards the stateless
web gets a shorter deadline:
# on cp1
ssh -n worker2 sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/worker2 --timeout=120s
sleep 15
kubectl -n tag14 get pods -l app=web | grep -c Terminating
cat >> web.yaml <<'EOF'
tolerations:
- {key: node.kubernetes.io/unreachable, operator: Exists, effect: NoExecute,
tolerationSeconds: 30}
- {key: node.kubernetes.io/not-ready, operator: Exists, effect: NoExecute,
tolerationSeconds: 30}
EOF
kubectl apply -f web.yaml
kubectl -n tag14 rollout status deploy/webExpected (abridged):
node/worker2 condition met
0
deployment.apps/web configured
deployment "web" successfully rolled out
Verify: The same failure, now with a stopwatch:
# on cp1
sleep 10
kubectl -n tag14 get pods -l app=web -o wide | cut -c1-80
ssh -n worker2 sudo systemctl stop rke2-agent
T0=$(date +%s)
until kubectl -n tag14 get pods -l app=web | grep -q Terminating; do sleep 2; done
echo "Ersatz nach $(( $(date +%s) - T0 )) s"
ssh -n worker2 sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/worker2 --timeout=120sExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
web-846654ffcc-h6fhz 1/1 Running 0 17s 10.42.2.121 worker2
web-846654ffcc-nm4r9 1/1 Running 0 15s 10.42.2.122 worker2
Ersatz nach 94 s
node/worker2 condition met
Ersatz nach 94 s means ‘replacement after 94 s’: 50 s
from the last Lease renewal (up to 10 s before the stop) plus 30 s of
toleration plus the controllers’ sync interval – 75 to 95 seconds
depending on the run, instead of six minutes. If the toleration is
already in the manifest, the API server does not add its own.
Save your progress? Sign up for free
Drill 4: Pods get evicted
Break it: A Pod writes 200 MiB into an
emptyDir, while only 100 MiB of local storage
(ephemeral-storage) are allowed – without the node itself
being full:
# on cp1
cat > fueller.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: fueller, namespace: tag14}
spec:
replicas: 1
selector: {matchLabels: {app: fueller}}
template:
metadata: {labels: {app: fueller}}
spec:
nodeSelector: {kubernetes.io/hostname: worker1}
containers:
- name: fueller
image: busybox:1.37
command:
- sh
- -c
- dd if=/dev/zero of=/daten/block bs=1M count=200; sleep infinity
resources: {limits: {ephemeral-storage: 100Mi}}
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- {name: daten, emptyDir: {}}
EOF
kubectl apply -f fueller.yamlSymptom (after about two minutes):
# on cp1
sleep 120
kubectl -n tag14 get pods -l app=fuellerExpected:
NAME READY STATUS RESTARTS AGE
fueller-6cd6db4c95-475sz 1/1 Running 0 25s
fueller-6cd6db4c95-pfvdp 0/1 Error 0 90s
fueller-6cd6db4c95-vqjmq 0/1 Error 0 2m3s
Ever new Pods, while the old ones are left lying around. Depending on
the timing, you may also see ContainerStatusUnknown.
Investigate: STATUS shows the
container. The reason is in the Pod status:
# on cp1
kubectl -n tag14 get pods -l app=fueller \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.reason
kubectl -n tag14 get pods -l app=fueller --field-selector=status.phase=Failed \
-o jsonpath='{.items[0].status.message}{"\n"}'Expected:
NAME PHASE REASON
fueller-6cd6db4c95-475sz Running <none>
fueller-6cd6db4c95-pfvdp Failed Evicted
fueller-6cd6db4c95-vqjmq Failed Evicted
Pod ephemeral local storage usage exceeds the total limit of containers 100Mi.
Cause: The emptyDir lives on the node’s
disk and counts toward the Pod’s local storage. The kubelet’s eviction
manager terminates the Pod without a grace period; an evicted Pod does
not restart, it stays Failed, and the Deployment creates a
new one that does the same. Under real node pressure, the message would
read The node was low on resource: ephemeral-storage., and
the node would show DiskPressure=True. You know the case
with sizeLimit on the volume from Day 10.
Fix: Set the limit to what is needed, additionally cap the volume, clean up the leftovers:
# on cp1
sed -i 's/ephemeral-storage: 100Mi/ephemeral-storage: 300Mi/' fueller.yaml
sed -i 's/emptyDir: {}/emptyDir: {sizeLimit: 250Mi}/' fueller.yaml
kubectl apply -f fueller.yaml
kubectl -n tag14 rollout status deploy/fueller
kubectl -n tag14 delete pods --field-selector=status.phase=FailedVerify (after one minute):
# on cp1
sleep 60
kubectl -n tag14 get pods -l app=fuellerExpected:
NAME READY STATUS RESTARTS AGE
fueller-795b88576b-6gqr8 1/1 Running 0 65s
Otherwise, evicted Pods are only cleaned up by the Pod garbage
collector – starting at 1000 terminated Pods (RKE2’s value for
terminated-pod-gc-threshold in the controller manager; the
Kubernetes default is 12,500).
Save your progress? Sign up for free
Drill 5: The drain hangs
Break it: Two api Pods preferably on
worker1, and a PodDisruptionBudget that requires both “just
to be safe”:
# on cp1
kubectl -n tag14 delete deploy fueller
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag14}
spec:
replicas: 2
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- {key: kubernetes.io/hostname, operator: In, values: [worker1]}
containers:
- name: api
image: traefik/whoami:v1.11
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: api, namespace: tag14}
spec:
minAvailable: 2
selector: {matchLabels: {app: api}}
EOF
kubectl apply -f api.yaml
kubectl -n tag14 rollout status deploy/apiSymptom:
# on cp1
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 \
| grep -E 'cordoned|api|timeout' | cut -c1-88 | sort | uniq -cExpected (abridged):
6 error when evicting pods/"api-599cf68b74-dznbc" -n "tag14" (will retry after 5s
1 error when evicting pods/"api-599cf68b74-dznbc" -n "tag14": global timeout reach
6 error when evicting pods/"api-599cf68b74-qgknb" -n "tag14" (will retry after 5s
1 error when evicting pods/"api-599cf68b74-qgknb" -n "tag14": global timeout reach
7 evicting pod tag14/api-599cf68b74-dznbc
7 evicting pod tag14/api-599cf68b74-qgknb
1 node/worker1 cordoned
The same attempt every five seconds until the timeout kicks in; the
full message ends with
Cannot evict pod as it would violate the pod's disruption budget.
Investigate:
# on cp1
kubectl -n tag14 get pdb api
kubectl get node worker1Expected:
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
api 2 N/A 0 39s
NAME STATUS ROLES AGE VERSION
worker1 Ready,SchedulingDisabled <none> 4h52m v1.37.2+rke2r1
Cause: minAvailable: 2 with two
replicas allows zero voluntary disruptions; the Eviction API answers
with 429, and kubectl drain keeps retrying endlessly. And
the aborted drain left the node cordoned – new Pods
never land there, without any error showing up.
Fix:
# on cp1
sed -i 's/minAvailable: 2/maxUnavailable: 1/' api.yaml
kubectl apply -f api.yaml
kubectl -n tag14 get pdb api
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data --timeout=120s 2>&1 \
| grep -E 'drained|evicted'
kubectl uncordon worker1Expected (abridged):
deployment.apps/api unchanged
poddisruptionbudget.policy/api configured
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
api N/A 1 1 44s
pod/api-599cf68b74-qgknb evicted
pod/api-599cf68b74-dznbc evicted
node/worker1 drained
node/worker1 uncordoned
Verify:
# on cp1
kubectl get nodes
kubectl -n tag14 get pods -l app=api -o wide | cut -c1-80Expected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane,etcd 4h55m v1.37.2+rke2r1
worker1 Ready <none> 4h52m v1.37.2+rke2r1
worker2 Ready <none> 4h51m v1.37.2+rke2r1
NAME READY STATUS RESTARTS AGE IP NODE
api-599cf68b74-mbxm5 1/1 Running 0 11s 10.42.2.126 worker2
api-599cf68b74-q5hxq 1/1 Running 0 5s 10.42.0.159 cp1
The drain also evicts other Pods from worker1 – in the
lab, for example, CoreDNS or Kyverno controllers. They move elsewhere;
after uncordon they do not come back on their own.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show for all nodes when their Lease was last renewed. Target time: 30 s
- Show the taints of all nodes in a table. Target time: 30 s
- Find all evicted Pods in the entire cluster. Target time: 30 s
- Show the kubelet’s
maxPodsonworker1without logging in. Target time: 30 s
Solution
# on cp1
kubectl -n kube-node-lease get lease \
-o custom-columns=NAME:.metadata.name,RENEW:.spec.renewTime
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl get pods -A --field-selector=status.phase=Failed \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.reason
kubectl get --raw /api/v1/nodes/worker1/proxy/configz | jq .kubeletconfig.maxPodsExpected:
NAME RENEW
cp1 2026-11-20T11:26:35.670313Z
worker1 2026-11-20T11:26:31.395700Z
worker2 2026-11-20T11:26:32.608976Z
NAME TAINTS
cp1 <none>
worker1 <none>
worker2 <none>
NS NAME REASON
110
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl get nodes
kubectl -n tag14 get pdb
kubectl -n tag14 get deployExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane,etcd 4h55m v1.37.2+rke2r1
worker1 Ready <none> 4h52m v1.37.2+rke2r1
worker2 Ready <none> 4h51m v1.37.2+rke2r1
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
api N/A 1 1 63s
NAME READY UP-TO-DATE AVAILABLE AGE
api 2/2 2 2 63s
web 2/2 2 2 13m
Challenge
A colleague raised the kubelet’s Pod limit on worker2
and restarted the agent. Since then the node has been
NotReady, even though systemctl reports the
service as active (running). Find the cause and fix it.
# on cp1
ssh -n worker2 sudo mkdir -p /etc/rancher/rke2/config.yaml.d
echo 'kubelet-arg: [max-pod=150]' \
| ssh worker2 sudo tee /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
ssh -n worker2 sudo systemctl restart rke2-agentHint: The service is rke2-agent – the
kubelet is just a child process.
Solution
After about a minute:
# on cp1
sleep 60
kubectl get node worker2
ssh -n worker2 systemctl is-active rke2-agent
ssh -n worker2 sudo journalctl -u rke2-agent -b --since -2min --no-pager \
| grep 'Kubelet exited' | tail -1 | cut -c1-88
L=/var/lib/rancher/rke2/agent/logs/kubelet.log
ssh -n worker2 "sudo grep 'command failed' $L" | tail -1 | cut -c1-120Expected (line wrapped):
NAME STATUS ROLES AGE VERSION
worker2 NotReady <none> 4h56m v1.37.2+rke2r1
active
Nov 20 11:32:08 worker2 rke2[170810]: time="2026-11-20T11:32:08Z" level=error msg="Kubel
E1120 11:32:13.611217 171561 run.go:72] "command failed" err="failed to parse kubelet
flag: unknown flag: --max-pod"
rke2-agent is running and restarts the kubelet every
five seconds (Kubelet exited: exit status 1); the kubelet
aborts while parsing its flags: max-pods was meant. RKE2
passes kubelet-arg through unchecked – the same happens
with flags that an upgrade has removed.
# on cp1
ssh -n worker2 sudo rm /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
ssh -n worker2 sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/worker2 --timeout=120sExpected:
node/worker2 condition met
Talos: The same typo via
KubeletConfig, investigated withtalosctl service kubeletandtalosctl logs kubelet(Day 19).
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag14
ssh -n worker2 sudo rmdir /etc/rancher/rke2/config.yaml.d
kubectl get nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taintsExpected:
namespace "tag14" deleted
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane,etcd 4h56m v1.37.2+rke2r1
worker1 Ready <none> 4h54m v1.37.2+rke2r1
worker2 Ready <none> 4h52m v1.37.2+rke2r1
NAME TAINTS
cp1 <none>
worker1 <none>
worker2 <none>
Interview Note
- Node
NotReady– what first? Read the conditions:Unknown→ the kubelet is silent (service, network, node gone);Falsewith a message → the kubelet is running and names the problem. - Why does the replacement take so long? 50 s
node-monitor-grace-periodplus 300 stolerationSecondsfromDefaultTolerationSeconds; the old Pod hangs inTerminatinguntil the kubelet is back or the Node object is deleted. - Drain hangs?
kubectl get pdb, columnALLOWED DISRUPTIONS; after an abort, do not forget touncordon.
Enterprise Note
- A permanently lost node is removed with
kubectl delete node; only then does the garbage collector clean up its Pods. For StatefulSets, only when it is certain that the node is really off (fencing). Since Kubernetes 1.28 it also works without deleting: after fencing, set the taintnode.kubernetes.io/out-of-service=nodeshutdown:NoExecute– Pods and volumes are released immediately. - Shorter
tolerationSecondsonly for stateless services; monitoring alerts onNotReadylong before the 300 s are up. - PDBs with
maxUnavailableinstead ofminAvailable= replica count; before any maintenance, look forkubectl get pdb -Aentries withALLOWED DISRUPTIONS 0. ephemeral-storagerequests and limits for all Pods, otherwise the kubelet evicts in its own order under disk pressure.- Manage kubelet arguments in RKE2 centrally via
config.yaml(Ansible, Rancher), and test on one node before rolling out.