Skip to content
Hands-on IT IT Drills

Day 14: Operations – Drain, Failure, Upgrade, etcd Snapshot

Goal: Drain nodes for maintenance without disturbing the app, measure and defuse a hard node failure, back up and restore etcd, and lift the cluster to Kubernetes 1.37. · Duration: approx. 120 min · Prerequisite: Day 13

Briefing

  • Drain closes a node to new Pods (cordon) and evicts the running ones. A PodDisruptionBudget (PDB) defines how many Pods must keep running at least.
  • A failure is not a drain: Kubernetes first waits until the node counts as NotReady, and evicts its Pods another five minutes later (Day 12).
  • etcd holds the entire state of the cluster. RKE2 backs it up with rke2 etcd-snapshot and restores it with --cluster-reset.
  • An upgrade goes node by node: first the server, then each agent with drain, installer, restart and uncordon.
 Drain                       Failure
 cordon ─► evict (PDB) ─►    node goes silent ─► ~40 s NotReady ─► +5 min evict
 Pods start elsewhere        Pods on the node count as Running until then

For this day you need a second terminal. In it runs a measurement loop that queries notes every second and shows the HTTP status. That way you see every disruption immediately.

Drill 1: Start of the Day and Measurement Loop

source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
mkdir -p "$LAB/tag14" && cd "$LAB/tag14"
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag11/notes/." .
cp -R "$LAB/code/tag13/notes/." . && cp -R "$LAB/code/tag14/notes/." .
sed -i '' "s/SERVER_IP/$SERVER/" values-rke2.yaml
kubectl get pods -o wide

Expected (abridged, names, IPs and times will differ on your machine):

NAME                      READY   STATUS      RESTARTS      AGE   IP    NODE
api-7dd6f4bd7f-4lbf8      1/1     Running     3 (1h ago)    1h    …     rke2-server
…
db-0                      1/1     Running     0             1h    …     rke2-agent2
…

Remember the node of db-0; in the book it is rke2-agent2. In the second terminal you start the measurement loop and leave it running all day:

source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
while true; do
  printf '%s %s\n' "$(date +%T)" "$(curl -s -o /dev/null -m 2 -w '%{http_code}' \
    --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes)"
  sleep 1
done

Expected:

10:24:55 200
10:24:56 200
10:24:57 200

000 means: no answer within 2 seconds. 500 and 503 come from web or Envoy.

Save your progress? Sign up for free

Drill 2: PodDisruptionBudgets

Chart 0.3.0 brings a new template pdb.yaml. For api and web it defines how many Pods a drain must leave standing at least:

{{/* chart/notes/templates/pdb.yaml – state at end of Day 14 */}}
{{- if .Values.pdb.enabled }}
{{- range $name := list "api" "web" }}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ $name }}
  labels:
    {{- include "notes.labels" (dict "ctx" $ "name" $name) | nindent 4 }}
spec:
  minAvailable: {{ (index $.Values.pdb $name).minAvailable }}
  selector:
    matchLabels:
      {{- include "notes.selector" $name | nindent 6 }}
{{- end }}
{{- end }}

Plus this in values.yaml (complete in the companion code code/tag14/notes/):

# From Day 14: this many Pods must keep running at least during a drain.
# db deliberately has no budget: one replica, one volume on one node (Day 14, Drill 4).
pdb:
  enabled: true
  api: {minAvailable: 2}
  web: {minAvailable: 1}
helm lint chart/notes -f values-rke2.yaml | tail -1
helm upgrade notes chart/notes -f values-rke2.yaml --wait --timeout 5m \
  | grep -E '^(STATUS|REVISION):'
kubectl get pdb

Expected (abridged):

1 chart(s) linted, 0 chart(s) failed
STATUS: deployed
REVISION: …
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    2               N/A               1                     58s
web    1               N/A               1                     58s

The upgrade takes about a minute, because the new chart version rolls all Pods (Day 11). ALLOWED DISRUPTIONS 1: of three api Pods, one may be gone at a time. Without --reuse-values, the api version from the chart (0.2.0) applies again.

Save your progress? Sign up for free

Drill 3: Drain Without the Database

Drain the agent on which db-0 is not running (in the book rke2-agent1):

kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data
kubectl get nodes
kubectl get pods -o wide

Expected (abridged):

node/rke2-agent1 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/rke2-canal-kmpdq
evicting pod notes/worker-d54b9d4df-jwv62
evicting pod notes/api-69c4fc7c84-8749n
evicting pod envoy-gateway-system/envoy-gateway-666cb7f75f-2tx5j
…
pod/api-69c4fc7c84-8749n evicted
…
node/rke2-agent1 drained
NAME          STATUS                     ROLES                AGE   VERSION
rke2-agent1   Ready,SchedulingDisabled   <none>               31m   v1.36.5+rke2r1
rke2-agent2   Ready                      <none>               31m   v1.36.5+rke2r1
rke2-server   Ready                      control-plane,etcd   34m   v1.36.5+rke2r1
NAME                     READY   STATUS    RESTARTS   AGE   IP   NODE
api-69c4fc7c84-7kcqd     1/1     Running   0          12s   …    rke2-server
api-69c4fc7c84-bgwtk     1/1     Running   0          1m    …    rke2-agent2
api-69c4fc7c84-n9jgk     1/1     Running   0          1m    …    rke2-server
db-0                     1/1     Running   0          1m    …    rke2-agent2
web-5bcd78bdd9-mj5x8     1/1     Running   0          12s   …    rke2-agent2
web-5bcd78bdd9-n94xv     1/1     Running   0          1m    …    rke2-server
worker-d54b9d4df-drzhr   1/1     Running   0          12s   …    rke2-server

The drain takes about 15 seconds, and the measurement loop stays at 200. The PDB makes sure that two api Pods are always running. DaemonSets like rke2-canal stay, because they belong to every node. Release the node again:

kubectl uncordon rke2-agent1

Expected:

node/rke2-agent1 uncordoned

The Pods don’t move back on their own. Only new Pods land on rke2-agent1 again.

Save your progress? Sign up for free

Drill 4: Drain with the Database

Now the agent with db-0. Watch the measurement loop while you do it:

kubectl drain rke2-agent2 --ignore-daemonsets --delete-emptydir-data
kubectl get pods
kubectl get events --field-selector involvedObject.name=db-0,reason=FailedScheduling \
  -o custom-columns=MSG:.message | tail -1

Expected (abridged):

node/rke2-agent2 cordoned
…
pod/db-0 evicted
…
node/rke2-agent2 drained
NAME                     READY   STATUS             RESTARTS   AGE
api-69c4fc7c84-7kcqd     0/1     Running            0          3m
api-69c4fc7c84-hjtwt     0/1     CrashLoopBackOff   2          24s
api-69c4fc7c84-n9jgk     0/1     Running            0          3m
db-0                     0/1     Pending            0          10s
web-5bcd78bdd9-6z5hh     1/1     Running            0          24s
web-5bcd78bdd9-n94xv     1/1     Running            0          3m
worker-d54b9d4df-drzhr   1/1     Running            0          2m
0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match
PersistentVolume's node affinity. …

In the measurement loop:

10:24:59 500
10:25:00 500
…

The local-path volume lives on rke2-agent2 (Day 12). The scheduler may only start db-0 there, and that is exactly where nothing may go right now. Without the database, the api reports /healthz with 503, drops out of the Service and restarts (Day 7). web answers with 500. Bring the node back:

kubectl uncordon rke2-agent2
kubectl wait --for=condition=Ready pod/db-0 --timeout=180s
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=api --timeout=180s
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected (abridged):

node/rke2-agent2 uncordoned
pod/db-0 condition met
pod/api-69c4fc7c84-7kcqd condition met
…
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

About 40 seconds after uncordon, the loop delivers 200 again, and the data is still there. A PDB wouldn’t have helped here: with maxUnavailable: 0 for db, the drain would simply hang. A database with one replica on local storage needs a maintenance window.

Warning: In the test, the only Envoy Pod was also running on rke2-agent2. The drain evicted it as well, and the loop showed 000 for a few seconds. Drill 6 fixes that.

Save your progress? Sign up for free

Drill 5: Hard Failure

A drain is polite. A crashed node simply stops reporting. Find the node on which Envoy runs and switch everything off there: rke2-killall.sh stops the service and all containers, as if the power were gone.

kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoy
multipass exec rke2-agent1 -- sudo systemctl stop rke2-agent
multipass exec rke2-agent1 -- sudo rke2-killall.sh > /dev/null
kubectl wait --for=condition=Ready=Unknown node/rke2-agent1 --timeout=120s
kubectl get pods -o wide

Expected (abridged, on your machine possibly a different agent):

NAME                                         READY   STATUS    …   NODE
envoy-notes-notes-3aa9e90e-d4bdf7b8b-6hxtn   2/2     Running   …   rke2-agent1
node/rke2-agent1 condition met
NAME                     READY   STATUS    RESTARTS   AGE   IP   NODE
api-69c4fc7c84-hjtwt     1/1     Running   0          3m    …    rke2-agent1
…
web-5bcd78bdd9-6z5hh     1/1     Running   0          3m    …    rke2-agent1
…

After about 45 seconds the node is NotReady, but its Pods are still shown as Running: the kubelet, which would know better, is dead. The loop only shows 000. Wait a good five minutes:

kubectl get pods -o wide
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoy

Expected (abridged):

NAME                     READY   STATUS              RESTARTS   AGE   IP   NODE
api-69c4fc7c84-9569q     0/1     ContainerCreating   0          2s    …    rke2-agent2
api-69c4fc7c84-hjtwt     1/1     Terminating         0          8m    …    rke2-agent1
…
web-5bcd78bdd9-6z5hh     1/1     Terminating         0          8m    …    rke2-agent1
web-5bcd78bdd9-sf9n6     0/1     ContainerCreating   0          2s    …    rke2-agent2
…
NAME                                         READY   STATUS              …   NODE
envoy-notes-notes-3aa9e90e-d4bdf7b8b-6hxtn   2/2     Terminating         …   rke2-agent1
envoy-notes-notes-3aa9e90e-d4bdf7b8b-rsq5l   0/2     ContainerCreating   …   rke2-agent2

Only now does the NoExecute taint from Day 12 take effect, and the Pods move. In the test, notes was gone for almost six minutes (10:27:01 to 10:32:59), because the whole Gateway depended on one Pod. The old Pods stay Terminating until the node is back:

multipass exec rke2-agent1 -- sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent1 --timeout=180s
kubectl get pods

Expected (abridged):

node/rke2-agent1 condition met
NAME                     READY   STATUS    RESTARTS   AGE
api-69c4fc7c84-7kcqd     1/1     Running   0          14m
api-69c4fc7c84-9569q     1/1     Running   0          6m
…

Save your progress? Sign up for free

Drill 6: Envoy Twice

The platform file k8s/gateway.yaml gets two Envoy Pods, spread across different nodes. Only the block envoyDeployment is new:

# k8s/gateway.yaml (excerpt) – state at end of Day 14
spec:
  provider:
    type: Kubernetes
    kubernetes:
      # From Day 14: two Envoy Pods on different nodes; one failure no longer
      # takes the Gateway down.
      envoyDeployment:
        replicas: 2
        pod:
          topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: envoy
              matchLabelKeys: [pod-template-hash]

topologySpreadConstraints with kubernetes.io/hostname forbids both Pods from landing on the same node. matchLabelKeys only counts the Pods of the same revision, so the rollout doesn’t block itself.

kubectl apply --dry-run=server -f k8s/gateway.yaml
kubectl apply -f k8s/gateway.yaml
kubectl -n envoy-gateway-system rollout status deploy \
  -l gateway.envoyproxy.io/owning-gateway-name=notes
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoy

Expected (abridged):

envoyproxy.gateway.envoyproxy.io/nodeport configured (server dry run)
gatewayclass.gateway.networking.k8s.io/eg unchanged (server dry run)
envoyproxy.gateway.envoyproxy.io/nodeport configured
gatewayclass.gateway.networking.k8s.io/eg unchanged
…
deployment "envoy-notes-notes-3aa9e90e" successfully rolled out
NAME                                          READY   STATUS    …   NODE
envoy-notes-notes-3aa9e90e-789cbc89d8-twnsx   2/2     Running   …   rke2-agent1
envoy-notes-notes-3aa9e90e-789cbc89d8-zd94d   2/2     Running   …   rke2-agent2

Repeat the failure from Drill 5 on one of the two Envoy nodes and watch the loop:

multipass exec rke2-agent1 -- sudo systemctl stop rke2-agent
multipass exec rke2-agent1 -- sudo rke2-killall.sh > /dev/null

Expected in the measurement loop (abridged):

10:52:16 200
10:52:18 000
…
10:53:01 503
10:53:03 200
10:53:04 200

About 45 seconds of disruption instead of six minutes. As soon as the node is NotReady, Kubernetes takes its Pods out of the Services, and the second Envoy takes over. Node back:

multipass exec rke2-agent1 -- sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node --all --timeout=180s

Expected:

node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met

Save your progress? Sign up for free

Drill 7: Back Up and Restore etcd

A snapshot backs up all Kubernetes objects, not the data in the volumes. On the server as root:

multipass shell rke2-server

In the VM:

sudo -i
rke2 etcd-snapshot save --name vor-dem-unfall 2>&1 | grep saved
rke2 etcd-snapshot ls 2>/dev/null

Expected (abridged):

time="…" level=info msg="Snapshot vor-dem-unfall-rke2-server-1793442998 saved."
Name                                  Location                                    …
vor-dem-unfall-rke2-server-1793442998 file:///var/lib/rancher/rke2/server/db/snap…

The command also warns Unknown flag --ingress-controller found in config.yaml. That is harmless: the snapshot command reads the same configuration as the server, but doesn’t need that option. Now the accident, in a further terminal on the Mac; the measurement loop keeps running alongside:

kubectl delete deployment web worker

Expected:

deployment.apps "web" deleted from notes namespace
deployment.apps "worker" deleted from notes namespace

The loop shows 503. Back in the root shell on the server: stop the service, reset etcd to the snapshot, start the service.

systemctl stop rke2-server
S=$(ls -t /var/lib/rancher/rke2/server/db/snapshots | head -1)
echo "$S"
rke2 server --cluster-reset \
  --cluster-reset-restore-path="/var/lib/rancher/rke2/server/db/snapshots/$S" \
  2>&1 | grep -o 'Managed etcd cluster membership has been reset[^.]*'
systemctl start rke2-server
exit
exit

Expected (abridged):

vor-dem-unfall-rke2-server-1793442998
Managed etcd cluster membership has been reset, restart without --cluster-reset flag now

The reset takes about a minute, and the start just as long again. On the Mac:

kubectl wait --for=condition=Ready node --all --timeout=300s
kubectl get deploy

Expected:

node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
api      3/3     3            3           26m
web      2/2     2            2           26m
worker   1/1     1            1           26m

web and worker are back. Still, look at the measurement loop: in the test, the Gateway kept returning 503 for five more minutes. The Envoy Gateway controller was working with the state from before the restore and didn’t know the new Pod addresses. That is why the Kubernetes docs recommend restarting controllers after a restore:

kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway
kubectl -n envoy-gateway-system rollout status deploy/envoy-gateway
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected (abridged):

deployment.apps/envoy-gateway restarted
…
deployment "envoy-gateway" successfully rolled out
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

The note never came from etcd, but from Postgres. If you want to back up data, you also need a database backup (Challenge Day 15).

Save your progress? Sign up for free

Drill 8: Upgrade to 1.37

First a fresh snapshot, then the server. The installer only replaces the files; restarting the service brings the new version:

multipass exec rke2-server -- sudo rke2 etcd-snapshot save --name vor-upgrade \
  2>&1 | grep saved
multipass exec rke2-server -- bash -c \
  'curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.37.1+rke2r1 sh -'
multipass exec rke2-server -- sudo systemctl restart rke2-server
kubectl get nodes
kubectl version | tail -1

Expected (abridged):

time="…" level=info msg="Snapshot vor-upgrade-rke2-server-1793443177 saved."
[INFO]  unpacking tarball file to /usr/local
NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               50m   v1.36.5+rke2r1
rke2-agent2   Ready    <none>               50m   v1.36.5+rke2r1
rke2-server   Ready    control-plane,etcd   53m   v1.37.1+rke2r1
Server Version: v1.37.1+rke2r1

In the test, the restart took just under four minutes. Don’t be alarmed if kubectl get pods -n kube-system briefly shows CrashLoopBackOff for kube-controller-manager and cloud-controller-manager afterwards. The static Pods are replaced one after the other; after another minute or two everything runs. A kubelet may be older than the API server (up to three minor versions), but never newer. That is why the server comes first. Now the agents, one after the other:

for n in rke2-agent1 rke2-agent2; do
  kubectl drain "$n" --ignore-daemonsets --delete-emptydir-data | tail -1
  multipass exec "$n" -- bash -c 'curl -sfL https://get.rke2.io \
    | sudo INSTALL_RKE2_VERSION=v1.37.1+rke2r1 INSTALL_RKE2_TYPE=agent sh -'
  multipass exec "$n" -- sudo systemctl restart rke2-agent
  kubectl wait --for=condition=Ready "node/$n" --timeout=300s
  kubectl uncordon "$n"
done
kubectl get nodes

Expected (abridged):

node/rke2-agent1 drained
[INFO]  unpacking tarball file to /usr/local
node/rke2-agent1 condition met
node/rke2-agent1 uncordoned
node/rke2-agent2 drained
…
node/rke2-agent2 uncordoned
NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               54m   v1.37.1+rke2r1
rke2-agent2   Ready    <none>               54m   v1.37.1+rke2r1
rke2-server   Ready    control-plane,etcd   57m   v1.37.1+rke2r1

Each agent takes about half a minute. For the agent with db-0 you again see 500 in the loop as in Drill 4 – this time as planned.

A new release sometimes brings new components. After the upgrade, look at what RKE2 now installs by itself:

kubectl get helmcharts -n kube-system rke2-gateway-api-crd
c='{.metadata.annotations.meta\.helm\.sh/release-name} '
c+='{.metadata.annotations.gateway\.networking\.k8s\.io/channel}'
kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath="$c"; echo
kubectl get gatewayclass eg

Expected (abridged):

NAME                   REPO   CHART   VERSION   TARGETNAMESPACE   BOOTSTRAP   FAILED
rke2-gateway-api-crd                                              false       False
rke2-gateway-api-crd standard
NAME   CONTROLLER                                      ACCEPTED   AGE
eg     gateway.envoyproxy.io/gatewayclass-controller   True       2h

Warning: RKE2 1.37 ships the Gateway API CRDs itself and takes over the existing ones from Envoy Gateway (Day 11), switching from the experimental channel to standard in the process. notes keeps running because it only uses Gateway and HTTPRoute. If you need experimental routes, you have to clarify that before the upgrade. Read the RKE2 release notes before every upgrade.

Save your progress? Sign up for free

Checkpoint

kubectl get nodes

Expected: all three Ready with v1.37.1+rke2r1, none with SchedulingDisabled.

NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               54m   v1.37.1+rke2r1
rke2-agent2   Ready    <none>               54m   v1.37.1+rke2r1
rke2-server   Ready    control-plane,etcd   57m   v1.37.1+rke2r1
kubectl get pdb

Expected: api with MIN AVAILABLE 2, web with 1.

NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    2               N/A               1                     30m
web    1               N/A               1                     30m
kubectl -n envoy-gateway-system get pods -o wide -l app.kubernetes.io/name=envoy

Expected: two Envoy Pods 2/2, on two different nodes.

NAME                                          READY   STATUS    …   NODE
envoy-notes-notes-3aa9e90e-789cbc89d8-twnsx   2/2     Running   …   rke2-agent1
envoy-notes-notes-3aa9e90e-789cbc89d8-zd94d   2/2     Running   …   rke2-agent2
multipass exec rke2-server -- sudo rke2 etcd-snapshot ls 2>/dev/null

Expected: the snapshots vor-dem-unfall and vor-upgrade.

Name                                  Location   …
vor-dem-unfall-rke2-server-1793442998 file:///…  …
vor-upgrade-rke2-server-1793443177    file:///…  …

Challenge

RKE2 backs up etcd by itself, by default at midnight and noon, and keeps five snapshots. Change that for the lab to every five minutes and three snapshots, and prove that an automatic snapshot is created.

Hint: The options are named like the flags of rke2 server without the two dashes and belong in /etc/rancher/rke2/config.yaml.

Solution

multipass exec rke2-server -- sudo tee -a /etc/rancher/rke2/config.yaml <<'EOF'
etcd-snapshot-schedule-cron: "*/5 * * * *"
etcd-snapshot-retention: 3
EOF
multipass exec rke2-server -- sudo systemctl restart rke2-server

After five minutes at most:

multipass exec rke2-server -- sudo rke2 etcd-snapshot ls 2>/dev/null

Expected (abridged):

Name                                  Location   …
vor-dem-unfall-rke2-server-1793442998 file:///…  …
vor-upgrade-rke2-server-1793443177    file:///…  …
etcd-snapshot-rke2-server-1793444405  file:///…  …

Automatic snapshots are named etcd-snapshot-<node>-<time>. The retention only applies to them; your named snapshots stay until you remove them with rke2 etcd-snapshot delete.

Save your progress? Sign up for free

Enterprise Note

  • Database: A Postgres replica on local storage needs maintenance windows. An operator like CloudNativePG runs the primary and standbys on different nodes and switches over during a drain (“Kubernetes operators hands-on”).
  • Gateway and ingress: Every component in the data path runs at least twice, spread across nodes or zones, with a PDB.
  • Downtime: The five minutes until the move can be shortened per Pod with tolerations for node.kubernetes.io/unreachable and a shorter tolerationSeconds, but not down to zero.
  • Snapshots: In production they leave the cluster via the S3 options (etcd-s3). A snapshot only on the server that just failed doesn’t help.
  • Upgrades: Rancher’s system-upgrade-controller automates the steps from Drill 8 with Plan objects. Rancher Manager controls upgrades for many clusters.