Day 14: Operations – Drain, Failure, Upgrade, etcd Snapshot
Goal: Drain nodes for maintenance without disturbing the app, measure and defuse a hard node failure, back up and restore etcd, and lift the cluster to Kubernetes 1.37. · Duration: approx. 120 min · Prerequisite: Day 13
Briefing
- Drain closes a node to new Pods
(
cordon) and evicts the running ones. A PodDisruptionBudget (PDB) defines how many Pods must keep running at least. - A failure is not a drain: Kubernetes first waits
until the node counts as
NotReady, and evicts its Pods another five minutes later (Day 12). - etcd holds the entire state of the cluster. RKE2
backs it up with
rke2 etcd-snapshotand restores it with--cluster-reset. - An upgrade goes node by node: first the server,
then each agent with drain, installer, restart and
uncordon.
Drain Failure
cordon ─► evict (PDB) ─► node goes silent ─► ~40 s NotReady ─► +5 min evict
Pods start elsewhere Pods on the node count as Running until then
For this day you need a second terminal. In it runs a measurement loop that queries notes every second and shows the HTTP status. That way you see every disruption immediately.
Drill 1: Start of the Day and Measurement Loop
source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
mkdir -p "$LAB/tag14" && cd "$LAB/tag14"
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
cp -R "$LAB/code/tag13/notes/." . && cp -R "$LAB/code/tag14/notes/." .
sed -i '' "s/SERVER_IP/$SERVER/" values-rke2.yaml
kubectl get pods -o wideExpected (abridged, names, IPs and times will differ on your machine):
NAME READY STATUS RESTARTS AGE IP NODE
api-7dd6f4bd7f-4lbf8 1/1 Running 3 (1h ago) 1h … rke2-server
…
db-0 1/1 Running 0 1h … rke2-agent2
…
Remember the node of db-0; in the book it is
rke2-agent2. In the second terminal you start the
measurement loop and leave it running all day:
source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
while true; do
printf '%s %s\n' "$(date +%T)" "$(curl -s -o /dev/null -m 2 -w '%{http_code}' \
--resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes)"
sleep 1
doneExpected:
10:24:55 200
10:24:56 200
10:24:57 200
000 means: no answer within 2 seconds. 500
and 503 come from web or Envoy.
Save your progress? Sign up for free
Drill 2: PodDisruptionBudgets
Chart 0.3.0 brings a new template pdb.yaml. For api and
web it defines how many Pods a drain must leave standing at least:
{{/* chart/notes/templates/pdb.yaml – state at end of Day 14 */}}
{{- if .Values.pdb.enabled }}
{{- range $name := list "api" "web" }}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ $name }}
labels:
{{- include "notes.labels" (dict "ctx" $ "name" $name) | nindent 4 }}
spec:
minAvailable: {{ (index $.Values.pdb $name).minAvailable }}
selector:
matchLabels:
{{- include "notes.selector" $name | nindent 6 }}
{{- end }}
{{- end }}Plus this in values.yaml (complete in the companion code
code/tag14/notes/):
# From Day 14: this many Pods must keep running at least during a drain.
# db deliberately has no budget: one replica, one volume on one node (Day 14, Drill 4).
pdb:
enabled: true
api: {minAvailable: 2}
web: {minAvailable: 1}helm lint chart/notes -f values-rke2.yaml | tail -1
helm upgrade notes chart/notes -f values-rke2.yaml --wait --timeout 5m \
| grep -E '^(STATUS|REVISION):'
kubectl get pdbExpected (abridged):
1 chart(s) linted, 0 chart(s) failed
STATUS: deployed
REVISION: …
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
api 2 N/A 1 58s
web 1 N/A 1 58s
The upgrade takes about a minute, because the new chart version rolls
all Pods (Day 11). ALLOWED DISRUPTIONS 1: of three api
Pods, one may be gone at a time. Without --reuse-values,
the api version from the chart (0.2.0) applies again.
Save your progress? Sign up for free
Drill 3: Drain Without the Database
Drain the agent on which db-0 is not
running (in the book rke2-agent1):
kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data
kubectl get nodes
kubectl get pods -o wideExpected (abridged):
node/rke2-agent1 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/rke2-canal-kmpdq
evicting pod notes/worker-d54b9d4df-jwv62
evicting pod notes/api-69c4fc7c84-8749n
evicting pod envoy-gateway-system/envoy-gateway-666cb7f75f-2tx5j
…
pod/api-69c4fc7c84-8749n evicted
…
node/rke2-agent1 drained
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready,SchedulingDisabled <none> 31m v1.36.5+rke2r1
rke2-agent2 Ready <none> 31m v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 34m v1.36.5+rke2r1
NAME READY STATUS RESTARTS AGE IP NODE
api-69c4fc7c84-7kcqd 1/1 Running 0 12s … rke2-server
api-69c4fc7c84-bgwtk 1/1 Running 0 1m … rke2-agent2
api-69c4fc7c84-n9jgk 1/1 Running 0 1m … rke2-server
db-0 1/1 Running 0 1m … rke2-agent2
web-5bcd78bdd9-mj5x8 1/1 Running 0 12s … rke2-agent2
web-5bcd78bdd9-n94xv 1/1 Running 0 1m … rke2-server
worker-d54b9d4df-drzhr 1/1 Running 0 12s … rke2-server
The drain takes about 15 seconds, and the measurement loop stays at
200. The PDB makes sure that two api Pods are always
running. DaemonSets like rke2-canal stay, because they
belong to every node. Release the node again:
kubectl uncordon rke2-agent1Expected:
node/rke2-agent1 uncordoned
The Pods don’t move back on their own. Only new Pods land on
rke2-agent1 again.
Save your progress? Sign up for free
Drill 4: Drain with the Database
Now the agent with db-0. Watch the measurement loop
while you do it:
kubectl drain rke2-agent2 --ignore-daemonsets --delete-emptydir-data
kubectl get pods
kubectl get events --field-selector involvedObject.name=db-0,reason=FailedScheduling \
-o custom-columns=MSG:.message | tail -1Expected (abridged):
node/rke2-agent2 cordoned
…
pod/db-0 evicted
…
node/rke2-agent2 drained
NAME READY STATUS RESTARTS AGE
api-69c4fc7c84-7kcqd 0/1 Running 0 3m
api-69c4fc7c84-hjtwt 0/1 CrashLoopBackOff 2 24s
api-69c4fc7c84-n9jgk 0/1 Running 0 3m
db-0 0/1 Pending 0 10s
web-5bcd78bdd9-6z5hh 1/1 Running 0 24s
web-5bcd78bdd9-n94xv 1/1 Running 0 3m
worker-d54b9d4df-drzhr 1/1 Running 0 2m
0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match
PersistentVolume's node affinity. …
In the measurement loop:
10:24:59 500
10:25:00 500
…
The local-path volume lives on rke2-agent2 (Day 12). The
scheduler may only start db-0 there, and that is exactly
where nothing may go right now. Without the database, the api reports
/healthz with 503, drops out of the Service and restarts
(Day 7). web answers with 500. Bring the node back:
kubectl uncordon rke2-agent2
kubectl wait --for=condition=Ready pod/db-0 --timeout=180s
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=api --timeout=180s
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echoExpected (abridged):
node/rke2-agent2 uncordoned
pod/db-0 condition met
pod/api-69c4fc7c84-7kcqd condition met
…
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
About 40 seconds after uncordon, the loop delivers
200 again, and the data is still there. A PDB wouldn’t have
helped here: with maxUnavailable: 0 for db,
the drain would simply hang. A database with one replica on local
storage needs a maintenance window.
Warning: In the test, the only Envoy Pod was also running on
rke2-agent2. The drain evicted it as well, and the loop showed000for a few seconds. Drill 6 fixes that.
Save your progress? Sign up for free
Drill 5: Hard Failure
A drain is polite. A crashed node simply stops reporting. Find the
node on which Envoy runs and switch everything off there:
rke2-killall.sh stops the service and all containers, as if
the power were gone.
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoy
multipass exec rke2-agent1 -- sudo systemctl stop rke2-agent
multipass exec rke2-agent1 -- sudo rke2-killall.sh > /dev/null
kubectl wait --for=condition=Ready=Unknown node/rke2-agent1 --timeout=120s
kubectl get pods -o wideExpected (abridged, on your machine possibly a different agent):
NAME READY STATUS … NODE
envoy-notes-notes-3aa9e90e-d4bdf7b8b-6hxtn 2/2 Running … rke2-agent1
node/rke2-agent1 condition met
NAME READY STATUS RESTARTS AGE IP NODE
api-69c4fc7c84-hjtwt 1/1 Running 0 3m … rke2-agent1
…
web-5bcd78bdd9-6z5hh 1/1 Running 0 3m … rke2-agent1
…
After about 45 seconds the node is NotReady, but its
Pods are still shown as Running: the kubelet, which would
know better, is dead. The loop only shows 000. Wait a good
five minutes:
kubectl get pods -o wide
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoyExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
api-69c4fc7c84-9569q 0/1 ContainerCreating 0 2s … rke2-agent2
api-69c4fc7c84-hjtwt 1/1 Terminating 0 8m … rke2-agent1
…
web-5bcd78bdd9-6z5hh 1/1 Terminating 0 8m … rke2-agent1
web-5bcd78bdd9-sf9n6 0/1 ContainerCreating 0 2s … rke2-agent2
…
NAME READY STATUS … NODE
envoy-notes-notes-3aa9e90e-d4bdf7b8b-6hxtn 2/2 Terminating … rke2-agent1
envoy-notes-notes-3aa9e90e-d4bdf7b8b-rsq5l 0/2 ContainerCreating … rke2-agent2
Only now does the NoExecute taint from Day 12 take
effect, and the Pods move. In the test, notes was gone for almost six
minutes (10:27:01 to 10:32:59), because the whole Gateway depended on
one Pod. The old Pods stay Terminating until the node is
back:
multipass exec rke2-agent1 -- sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent1 --timeout=180s
kubectl get podsExpected (abridged):
node/rke2-agent1 condition met
NAME READY STATUS RESTARTS AGE
api-69c4fc7c84-7kcqd 1/1 Running 0 14m
api-69c4fc7c84-9569q 1/1 Running 0 6m
…
Save your progress? Sign up for free
Drill 6: Envoy Twice
The platform file k8s/gateway.yaml gets two Envoy Pods,
spread across different nodes. Only the block
envoyDeployment is new:
# k8s/gateway.yaml (excerpt) – state at end of Day 14
spec:
provider:
type: Kubernetes
kubernetes:
# From Day 14: two Envoy Pods on different nodes; one failure no longer
# takes the Gateway down.
envoyDeployment:
replicas: 2
pod:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: envoy
matchLabelKeys: [pod-template-hash]topologySpreadConstraints with
kubernetes.io/hostname forbids both Pods from landing on
the same node. matchLabelKeys only counts the Pods of the
same revision, so the rollout doesn’t block itself.
kubectl apply --dry-run=server -f k8s/gateway.yaml
kubectl apply -f k8s/gateway.yaml
kubectl -n envoy-gateway-system rollout status deploy \
-l gateway.envoyproxy.io/owning-gateway-name=notes
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoyExpected (abridged):
envoyproxy.gateway.envoyproxy.io/nodeport configured (server dry run)
gatewayclass.gateway.networking.k8s.io/eg unchanged (server dry run)
envoyproxy.gateway.envoyproxy.io/nodeport configured
gatewayclass.gateway.networking.k8s.io/eg unchanged
…
deployment "envoy-notes-notes-3aa9e90e" successfully rolled out
NAME READY STATUS … NODE
envoy-notes-notes-3aa9e90e-789cbc89d8-twnsx 2/2 Running … rke2-agent1
envoy-notes-notes-3aa9e90e-789cbc89d8-zd94d 2/2 Running … rke2-agent2
Repeat the failure from Drill 5 on one of the two Envoy nodes and watch the loop:
multipass exec rke2-agent1 -- sudo systemctl stop rke2-agent
multipass exec rke2-agent1 -- sudo rke2-killall.sh > /dev/nullExpected in the measurement loop (abridged):
10:52:16 200
10:52:18 000
…
10:53:01 503
10:53:03 200
10:53:04 200
About 45 seconds of disruption instead of six minutes. As soon as the
node is NotReady, Kubernetes takes its Pods out of the
Services, and the second Envoy takes over. Node back:
multipass exec rke2-agent1 -- sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node --all --timeout=180sExpected:
node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met
Save your progress? Sign up for free
Drill 7: Back Up and Restore etcd
A snapshot backs up all Kubernetes objects, not the data in the volumes. On the server as root:
multipass shell rke2-serverIn the VM:
sudo -i
rke2 etcd-snapshot save --name vor-dem-unfall 2>&1 | grep saved
rke2 etcd-snapshot ls 2>/dev/nullExpected (abridged):
time="…" level=info msg="Snapshot vor-dem-unfall-rke2-server-1793442998 saved."
Name Location …
vor-dem-unfall-rke2-server-1793442998 file:///var/lib/rancher/rke2/server/db/snap…
The command also warns
Unknown flag --ingress-controller found in config.yaml.
That is harmless: the snapshot command reads the same configuration as
the server, but doesn’t need that option. Now the accident, in a further
terminal on the Mac; the measurement loop keeps running alongside:
kubectl delete deployment web workerExpected:
deployment.apps "web" deleted from notes namespace
deployment.apps "worker" deleted from notes namespace
The loop shows 503. Back in the root shell on the
server: stop the service, reset etcd to the snapshot, start the
service.
systemctl stop rke2-server
S=$(ls -t /var/lib/rancher/rke2/server/db/snapshots | head -1)
echo "$S"
rke2 server --cluster-reset \
--cluster-reset-restore-path="/var/lib/rancher/rke2/server/db/snapshots/$S" \
2>&1 | grep -o 'Managed etcd cluster membership has been reset[^.]*'
systemctl start rke2-server
exit
exitExpected (abridged):
vor-dem-unfall-rke2-server-1793442998
Managed etcd cluster membership has been reset, restart without --cluster-reset flag now
The reset takes about a minute, and the start just as long again. On the Mac:
kubectl wait --for=condition=Ready node --all --timeout=300s
kubectl get deployExpected:
node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met
NAME READY UP-TO-DATE AVAILABLE AGE
api 3/3 3 3 26m
web 2/2 2 2 26m
worker 1/1 1 1 26m
web and worker are back. Still, look at the measurement loop: in the
test, the Gateway kept returning 503 for five more minutes.
The Envoy Gateway controller was working with the state from before the
restore and didn’t know the new Pod addresses. That is why the
Kubernetes docs recommend restarting controllers after a restore:
kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway
kubectl -n envoy-gateway-system rollout status deploy/envoy-gateway
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echoExpected (abridged):
deployment.apps/envoy-gateway restarted
…
deployment "envoy-gateway" successfully rolled out
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]
The note never came from etcd, but from Postgres. If you want to back up data, you also need a database backup (Challenge Day 15).
Save your progress? Sign up for free
Drill 8: Upgrade to 1.37
First a fresh snapshot, then the server. The installer only replaces the files; restarting the service brings the new version:
multipass exec rke2-server -- sudo rke2 etcd-snapshot save --name vor-upgrade \
2>&1 | grep saved
multipass exec rke2-server -- bash -c \
'curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.37.1+rke2r1 sh -'
multipass exec rke2-server -- sudo systemctl restart rke2-server
kubectl get nodes
kubectl version | tail -1Expected (abridged):
time="…" level=info msg="Snapshot vor-upgrade-rke2-server-1793443177 saved."
[INFO] unpacking tarball file to /usr/local
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 50m v1.36.5+rke2r1
rke2-agent2 Ready <none> 50m v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 53m v1.37.1+rke2r1
Server Version: v1.37.1+rke2r1
In the test, the restart took just under four minutes. Don’t be
alarmed if kubectl get pods -n kube-system briefly shows
CrashLoopBackOff for kube-controller-manager
and cloud-controller-manager afterwards. The static Pods
are replaced one after the other; after another minute or two everything
runs. A kubelet may be older than the API server (up to three minor
versions), but never newer. That is why the server comes first. Now the
agents, one after the other:
for n in rke2-agent1 rke2-agent2; do
kubectl drain "$n" --ignore-daemonsets --delete-emptydir-data | tail -1
multipass exec "$n" -- bash -c 'curl -sfL https://get.rke2.io \
| sudo INSTALL_RKE2_VERSION=v1.37.1+rke2r1 INSTALL_RKE2_TYPE=agent sh -'
multipass exec "$n" -- sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready "node/$n" --timeout=300s
kubectl uncordon "$n"
done
kubectl get nodesExpected (abridged):
node/rke2-agent1 drained
[INFO] unpacking tarball file to /usr/local
node/rke2-agent1 condition met
node/rke2-agent1 uncordoned
node/rke2-agent2 drained
…
node/rke2-agent2 uncordoned
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 54m v1.37.1+rke2r1
rke2-agent2 Ready <none> 54m v1.37.1+rke2r1
rke2-server Ready control-plane,etcd 57m v1.37.1+rke2r1
Each agent takes about half a minute. For the agent with
db-0 you again see 500 in the loop as in Drill
4 – this time as planned.
A new release sometimes brings new components. After the upgrade, look at what RKE2 now installs by itself:
kubectl get helmcharts -n kube-system rke2-gateway-api-crd
c='{.metadata.annotations.meta\.helm\.sh/release-name} '
c+='{.metadata.annotations.gateway\.networking\.k8s\.io/channel}'
kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath="$c"; echo
kubectl get gatewayclass egExpected (abridged):
NAME REPO CHART VERSION TARGETNAMESPACE BOOTSTRAP FAILED
rke2-gateway-api-crd false False
rke2-gateway-api-crd standard
NAME CONTROLLER ACCEPTED AGE
eg gateway.envoyproxy.io/gatewayclass-controller True 2h
Warning: RKE2 1.37 ships the Gateway API CRDs itself and takes over the existing ones from Envoy Gateway (Day 11), switching from the
experimentalchannel tostandardin the process. notes keeps running because it only uses Gateway and HTTPRoute. If you need experimental routes, you have to clarify that before the upgrade. Read the RKE2 release notes before every upgrade.
Save your progress? Sign up for free
Checkpoint
kubectl get nodesExpected: all three Ready with
v1.37.1+rke2r1, none with
SchedulingDisabled.
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 54m v1.37.1+rke2r1
rke2-agent2 Ready <none> 54m v1.37.1+rke2r1
rke2-server Ready control-plane,etcd 57m v1.37.1+rke2r1
kubectl get pdbExpected: api with MIN AVAILABLE 2, web with 1.
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
api 2 N/A 1 30m
web 1 N/A 1 30m
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoyExpected: two Envoy Pods 2/2, on two different
nodes.
NAME READY STATUS … NODE
envoy-notes-notes-3aa9e90e-789cbc89d8-twnsx 2/2 Running … rke2-agent1
envoy-notes-notes-3aa9e90e-789cbc89d8-zd94d 2/2 Running … rke2-agent2
multipass exec rke2-server -- sudo rke2 etcd-snapshot ls 2>/dev/nullExpected: the snapshots vor-dem-unfall and
vor-upgrade.
Name Location …
vor-dem-unfall-rke2-server-1793442998 file:///… …
vor-upgrade-rke2-server-1793443177 file:///… …
Challenge
RKE2 backs up etcd by itself, by default at midnight and noon, and keeps five snapshots. Change that for the lab to every five minutes and three snapshots, and prove that an automatic snapshot is created.
Hint: The options are named like the flags of
rke2 server without the two dashes and belong in
/etc/rancher/rke2/config.yaml.
Solution
multipass exec rke2-server -- sudo tee -a /etc/rancher/rke2/config.yaml <<'EOF'
etcd-snapshot-schedule-cron: "*/5 * * * *"
etcd-snapshot-retention: 3
EOF
multipass exec rke2-server -- sudo systemctl restart rke2-serverAfter five minutes at most:
multipass exec rke2-server -- sudo rke2 etcd-snapshot ls 2>/dev/nullExpected (abridged):
Name Location …
vor-dem-unfall-rke2-server-1793442998 file:///… …
vor-upgrade-rke2-server-1793443177 file:///… …
etcd-snapshot-rke2-server-1793444405 file:///… …
Automatic snapshots are named
etcd-snapshot-<node>-<time>. The retention only
applies to them; your named snapshots stay until you remove them with
rke2 etcd-snapshot delete.
Save your progress? Sign up for free
Enterprise Note
- Database: A Postgres replica on local storage needs maintenance windows. An operator like CloudNativePG runs the primary and standbys on different nodes and switches over during a drain (“Kubernetes operators hands-on”).
- Gateway and ingress: Every component in the data path runs at least twice, spread across nodes or zones, with a PDB.
- Downtime: The five minutes until the move can be
shortened per Pod with
tolerationsfornode.kubernetes.io/unreachableand a shortertolerationSeconds, but not down to zero. - Snapshots: In production they leave the cluster via
the S3 options (
etcd-s3). A snapshot only on the server that just failed doesn’t help. - Upgrades: Rancher’s system-upgrade-controller
automates the steps from Drill 8 with
Planobjects. Rancher Manager controls upgrades for many clusters.