Skip to content
Hands-on IT IT Drills

Day 13: Cluster Lifecycle

Goal: You back up etcd and restore it, upgrade the cluster from 1.36 to 1.37, and renew certificates – without losing the workloads. · Duration: approx. 90 min · Prerequisite: Day 1

Briefing

CKA: “Manage the lifecycle of Kubernetes clusters” (part of the 25 %). etcd backup and upgrade are classics in every exam.

  Upgrade order (always just one minor version):
  1. kubeadm on cp1       2. kubeadm upgrade apply    3. kubelet+kubectl on cp1
  4. per worker: drain -> kubeadm upgrade node -> kubelet -> uncordon
  • etcd holds the entire cluster state. No backup, no way back.
  • kubeadm upgrades the control plane; you upgrade the kubelet on each node yourself.

Drill 1: Get etcdctl

On cp1. Which etcd version is running?

k get pod etcd-cp1 -n kube-system -o jsonpath='{.spec.containers[0].image}{"\n"}'

Expected: registry.k8s.io/etcd:3.6.8-0

The matching tools from the etcd project:

cd /tmp
E=3.6.8
U=https://github.com/etcd-io/etcd/releases/download/v$E
curl -sSLO $U/etcd-v$E-linux-amd64.tar.gz
curl -sSL $U/SHA256SUMS | grep "etcd-v$E-linux-amd64.tar.gz$" | sha256sum -c
tar xzf etcd-v$E-linux-amd64.tar.gz
sudo install -m 755 etcd-v$E-linux-amd64/etcdctl etcd-v$E-linux-amd64/etcdutl \
  /usr/local/bin/
etcdctl version
cd ~/cka

Expected:

etcd-v3.6.8-linux-amd64.tar.gz: OK
etcdctl version: 3.6.8
API version: 3.6

etcdctl talks to the running etcd, etcdutl works offline with files.

Save your progress? Sign up for free

Drill 2: Create a Snapshot

First create a state we will recognize:

k create ns vorher
k create configmap marker -n vorher --from-literal=zeit="$(date +%H:%M)"

The certificates for etcdctl are listed in the etcd manifest (--cert-file, --key-file, --trusted-ca-file):

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd-backup.db
sudo etcdutl snapshot status /var/lib/etcd-backup.db -w table

Expected:

Snapshot saved at /var/lib/etcd-backup.db
+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 7c9f621d |    16769 |        902 |      19 MB |   3.6.0 |
+----------+----------+------------+------------+---------+

Now “something happens”:

k create ns nachher
k delete ns vorher

Save your progress? Sign up for free

Drill 3: Restore the Snapshot

Restore into a new directory and point etcd at it:

sudo etcdutl snapshot restore /var/lib/etcd-backup.db --data-dir /var/lib/etcd-restore
sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.bak
sudo sed -i 's#path: /var/lib/etcd$#path: /var/lib/etcd-restore#' \
  /etc/kubernetes/manifests/etcd.yaml
sudo grep -n '/var/lib/etcd' /etc/kubernetes/manifests/etcd.yaml

Expected:

... restored snapshot {"path": "/var/lib/etcd-backup.db", ...
  "data-dir": "/var/lib/etcd-restore", ...}
18:    - --data-dir=/var/lib/etcd
75:    - mountPath: /var/lib/etcd
91:      path: /var/lib/etcd-restore

Only the hostPath (line 91) changes: inside the container the path stays the same, on the node it points to the restored data. The manifest backup is deliberately kept outside manifests – otherwise the kubelet would start it as a second pod.

The kubelet restarts etcd, and the API server is gone for one to two minutes. Then:

k get ns vorher nachher
k get cm marker -n vorher -o jsonpath='{.data.zeit}{"\n"}'

Expected:

NAME     STATUS   AGE
vorher   Active   106s
Error from server (NotFound): namespaces "nachher" not found
22:25

The cluster is back at the state of the snapshot.

Save your progress? Sign up for free

Drill 4: The Trap Before the Upgrade

kubeadm upgrade regenerates the etcd manifest – from the configuration in kubeadm-config:

k get cm kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' \
  | grep -A2 'etcd:'

Expected:

etcd:
  local:
    dataDir: /var/lib/etcd

After the upgrade, etcd would point at /var/lib/etcd again – at the old data from before the restore. That is why you move the restored data to the default location. To do so, stop etcd by briefly removing the manifest:

sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd-restore.yaml
sudo crictl ps --name '^etcd$'

Expected: no container line anymore (you may need to wait a few seconds).

sudo mv /var/lib/etcd /var/lib/etcd-alt
sudo mv /var/lib/etcd-restore /var/lib/etcd
sudo cp /root/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yaml

After one to two minutes:

k get ns vorher

Expected: vorher is Active – now with the data in the default location.

Save your progress? Sign up for free

Drill 5: Upgrade the Control Plane

Switch the package repository to 1.37 and upgrade kubeadm:

sudo sed -i 's/v1.36/v1.37/g' /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
  | sudo gpg --yes --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo apt-get update
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan

Expected (abbreviated):

[upgrade/versions] Cluster version: 1.36.6
[upgrade/versions] kubeadm version: v1.37.2
COMPONENT                 NODE      CURRENT   TARGET
kube-apiserver            cp1       v1.36.6   v1.37.2
kube-controller-manager   cp1       v1.36.6   v1.37.2
kube-scheduler            cp1       v1.36.6   v1.37.2
kube-proxy                          1.36.6    v1.37.2
CoreDNS                             v1.14.2   v1.14.6
etcd                      cp1       3.6.8-0   3.7.0-0

You can now apply the upgrade by executing the following command:

    kubeadm upgrade apply v1.37.2
sudo kubeadm upgrade apply v1.37.2 -y

Expected (last lines):

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.37.2".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right
order.

Now the kubelet on cp1:

k drain cp1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Restarting the kubelet restarts the static pods – the API server is briefly gone (connection refused). After about a minute:

k uncordon cp1
k get nodes

Expected:

NAME      STATUS   ROLES           AGE   VERSION
cp1       Ready    control-plane   82m   v1.37.2
worker1   Ready    <none>          81m   v1.36.6
worker2   Ready    <none>          81m   v1.36.6

Save your progress? Sign up for free

Drill 6: Upgrade the Workers

On cp1:

k drain worker1 --ignore-daemonsets --delete-emptydir-data

Expected (last line): node/worker1 drained

Without --delete-emptydir-data, drain aborts because metrics-server and Traefik use an emptyDir. On worker1:

sudo sed -i 's/v1.36/v1.37/g' /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
  | sudo gpg --yes --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo apt-get update
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubeadm kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Expected (from kubeadm upgrade node, abbreviated):

[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.

Back on cp1:

k uncordon worker1

The same for worker2. Then:

k get nodes

Expected:

NAME      STATUS   ROLES           AGE   VERSION
cp1       Ready    control-plane   86m   v1.37.2
worker1   Ready    <none>          85m   v1.37.2
worker2   Ready    <none>          85m   v1.37.2

Save your progress? Sign up for free

Drill 7: Renew Certificates

sudo kubeadm certs check-expiration

Expected (abbreviated):

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTH...
admin.conf                 Nov 17, 2027 22:29 UTC   364d            ca
apiserver                  Nov 17, 2027 22:29 UTC   364d            ca
...
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Nov 14, 2036 21:11 UTC   9y              no

All certificates are valid for a full year again: kubeadm upgrade apply renews them automatically. If you upgrade every year, you never have to renew them by hand. The CAs last ten years. By hand, it works like this:

sudo kubeadm certs renew all

Expected (last lines):

certificate embedded in the kubeconfig file for the super-admin renewed

Done renewing certificates. You must restart the kube-apiserver,
kube-controller-manager, kube-scheduler and etcd, so that they can use the new
certificates.

The components read their certificates only at startup. You restart static pods by briefly removing their manifest – here the scheduler and controller manager:

sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
sudo crictl ps --name 'kube-(scheduler|controller-manager)' -q | wc -l

Expected (after about 20 seconds): 0

sudo mv /tmp/kube-scheduler.yaml /tmp/kube-controller-manager.yaml \
  /etc/kubernetes/manifests/

Do the same with kube-apiserver.yaml and etcd.yaml. Finally, take over the renewed admin kubeconfig:

sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
k get pods -n kube-system -l tier=control-plane

Expected: four pods, all Running.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show whether etcd is healthy. Target time: 45 s
  2. Show the kubelet version of all nodes on one line. Target time: 20 s
  3. Check whether drain worker1 would succeed without further flags – without actually draining the node. Target time: 30 s

Solution

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint health -w table
k get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}{"\n"}'
k drain worker1 --ignore-daemonsets --dry-run=client

Expected (abbreviated):

+------------------------+--------+-------------+-------+
|        ENDPOINT        | HEALTH |    TOOK     | ERROR |
+------------------------+--------+-------------+-------+
| https://127.0.0.1:2379 |   true | 33.755384ms |       |
+------------------------+--------+-------------+-------+
v1.37.2 v1.37.2 v1.37.2
node/worker1 cordoned (dry run)
error: unable to drain node "worker1" due to error: cannot delete Pods with local
storage (use --delete-emptydir-data to override): ...

Save your progress? Sign up for free

Checkpoint

k version | tail -1

Expected: Server Version: v1.37.2

k get pod etcd-cp1 -n kube-system -o jsonpath='{.spec.containers[0].image}{"\n"}'

Expected: registry.k8s.io/etcd:3.7.0-0

sudo ls -lh /var/lib/etcd-backup.db

Expected: a file of about 18 MB (18M).

Challenge

After the upgrade, etcd 3.7 is running. Is your snapshot from earlier still usable? Check it with the new etcd version without touching the cluster.

Hint: etcdutl works offline and does not need a running etcd.

Solution

cd /tmp
E=3.7.0
U=https://github.com/etcd-io/etcd/releases/download/v$E
curl -sSLO $U/etcd-v$E-linux-amd64.tar.gz
tar xzf etcd-v$E-linux-amd64.tar.gz
sudo ./etcd-v$E-linux-amd64/etcdutl snapshot status /var/lib/etcd-backup.db -w table

Expected:

┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 7c9f621d │    16769 │        902 │      19 MB │   3.6.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

The same values as in Drill 2, just with a new table border. From now on, use the etcdctl version of the running etcd for new snapshots.

Clean up:

k delete ns vorher
sudo rm -rf /var/lib/etcd-alt

Save your progress? Sign up for free

Exam Tip

  • etcd tasks: copy the endpoint and certificate paths from the etcd manifest, don’t guess. sudo grep -E 'cert|key|trusted|listen-client' /etc/kubernetes/manifests/etcd.yaml.
  • Restore in the exam: new --data-dir, then switch the hostPath in the manifest. Wait until the API server is back.
  • Upgrade: the docs page “Upgrading kubeadm clusters” has all the commands to copy. Just adjust the version numbers.

Enterprise Note

  • Automate etcd snapshots (CronJob or systemd timer), encrypt them, and store them outside the cluster; practice the restore regularly.
  • Run upgrades in a staging environment first, and check the deprecation list beforehand (kubectl deprecations, Pluto).
  • With three control planes, the upgrade runs without API downtime (Day 16).