Day 13: Cluster Lifecycle
Goal: You back up etcd and restore it, upgrade the cluster from 1.36 to 1.37, and renew certificates – without losing the workloads. · Duration: approx. 90 min · Prerequisite: Day 1
Briefing
CKA: “Manage the lifecycle of Kubernetes clusters” (part of the 25 %). etcd backup and upgrade are classics in every exam.
Upgrade order (always just one minor version):
1. kubeadm on cp1 2. kubeadm upgrade apply 3. kubelet+kubectl on cp1
4. per worker: drain -> kubeadm upgrade node -> kubelet -> uncordon
- etcd holds the entire cluster state. No backup, no way back.
kubeadmupgrades the control plane; you upgrade the kubelet on each node yourself.
Drill 1: Get etcdctl
On cp1. Which etcd version is running?
k get pod etcd-cp1 -n kube-system -o jsonpath='{.spec.containers[0].image}{"\n"}'Expected:
registry.k8s.io/etcd:3.6.8-0
The matching tools from the etcd project:
cd /tmp
E=3.6.8
U=https://github.com/etcd-io/etcd/releases/download/v$E
curl -sSLO $U/etcd-v$E-linux-amd64.tar.gz
curl -sSL $U/SHA256SUMS | grep "etcd-v$E-linux-amd64.tar.gz$" | sha256sum -c
tar xzf etcd-v$E-linux-amd64.tar.gz
sudo install -m 755 etcd-v$E-linux-amd64/etcdctl etcd-v$E-linux-amd64/etcdutl \
/usr/local/bin/
etcdctl version
cd ~/ckaExpected:
etcd-v3.6.8-linux-amd64.tar.gz: OK
etcdctl version: 3.6.8
API version: 3.6
etcdctl talks to the running etcd, etcdutl
works offline with files.
Save your progress? Sign up for free
Drill 2: Create a Snapshot
First create a state we will recognize:
k create ns vorher
k create configmap marker -n vorher --from-literal=zeit="$(date +%H:%M)"The certificates for etcdctl are listed in the etcd manifest
(--cert-file, --key-file,
--trusted-ca-file):
sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /var/lib/etcd-backup.db
sudo etcdutl snapshot status /var/lib/etcd-backup.db -w tableExpected:
Snapshot saved at /var/lib/etcd-backup.db
+----------+----------+------------+------------+---------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 7c9f621d | 16769 | 902 | 19 MB | 3.6.0 |
+----------+----------+------------+------------+---------+
Now “something happens”:
k create ns nachher
k delete ns vorherSave your progress? Sign up for free
Drill 3: Restore the Snapshot
Restore into a new directory and point etcd at it:
sudo etcdutl snapshot restore /var/lib/etcd-backup.db --data-dir /var/lib/etcd-restore
sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.bak
sudo sed -i 's#path: /var/lib/etcd$#path: /var/lib/etcd-restore#' \
/etc/kubernetes/manifests/etcd.yaml
sudo grep -n '/var/lib/etcd' /etc/kubernetes/manifests/etcd.yamlExpected:
... restored snapshot {"path": "/var/lib/etcd-backup.db", ...
"data-dir": "/var/lib/etcd-restore", ...}
18: - --data-dir=/var/lib/etcd
75: - mountPath: /var/lib/etcd
91: path: /var/lib/etcd-restore
Only the hostPath (line 91) changes: inside the
container the path stays the same, on the node it points to the restored
data. The manifest backup is deliberately kept outside
manifests – otherwise the kubelet would start it as a
second pod.
The kubelet restarts etcd, and the API server is gone for one to two minutes. Then:
k get ns vorher nachher
k get cm marker -n vorher -o jsonpath='{.data.zeit}{"\n"}'Expected:
NAME STATUS AGE
vorher Active 106s
Error from server (NotFound): namespaces "nachher" not found
22:25
The cluster is back at the state of the snapshot.
Save your progress? Sign up for free
Drill 4: The Trap Before the Upgrade
kubeadm upgrade regenerates the etcd manifest – from the
configuration in kubeadm-config:
k get cm kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' \
| grep -A2 'etcd:'Expected:
etcd:
local:
dataDir: /var/lib/etcd
After the upgrade, etcd would point at /var/lib/etcd
again – at the old data from before the restore. That
is why you move the restored data to the default location. To do so,
stop etcd by briefly removing the manifest:
sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd-restore.yaml
sudo crictl ps --name '^etcd$'Expected: no container line anymore (you may need to wait a few seconds).
sudo mv /var/lib/etcd /var/lib/etcd-alt
sudo mv /var/lib/etcd-restore /var/lib/etcd
sudo cp /root/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yamlAfter one to two minutes:
k get ns vorherExpected: vorher is Active
– now with the data in the default location.
Save your progress? Sign up for free
Drill 5: Upgrade the Control Plane
Switch the package repository to 1.37 and upgrade kubeadm:
sudo sed -i 's/v1.36/v1.37/g' /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
| sudo gpg --yes --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo apt-get update
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo apt-mark hold kubeadm
sudo kubeadm upgrade planExpected (abbreviated):
[upgrade/versions] Cluster version: 1.36.6
[upgrade/versions] kubeadm version: v1.37.2
COMPONENT NODE CURRENT TARGET
kube-apiserver cp1 v1.36.6 v1.37.2
kube-controller-manager cp1 v1.36.6 v1.37.2
kube-scheduler cp1 v1.36.6 v1.37.2
kube-proxy 1.36.6 v1.37.2
CoreDNS v1.14.2 v1.14.6
etcd cp1 3.6.8-0 3.7.0-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.37.2
sudo kubeadm upgrade apply v1.37.2 -yExpected (last lines):
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.37.2".
[upgrade] Now please proceed with upgrading the rest of the nodes by following the right
order.
Now the kubelet on cp1:
k drain cp1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubeletRestarting the kubelet restarts the static pods – the API server is
briefly gone (connection refused). After about a
minute:
k uncordon cp1
k get nodesExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 82m v1.37.2
worker1 Ready <none> 81m v1.36.6
worker2 Ready <none> 81m v1.36.6
Save your progress? Sign up for free
Drill 6: Upgrade the Workers
On cp1:
k drain worker1 --ignore-daemonsets --delete-emptydir-dataExpected (last line):
node/worker1 drained
Without --delete-emptydir-data, drain
aborts because metrics-server and Traefik use an emptyDir.
On worker1:
sudo sed -i 's/v1.36/v1.37/g' /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key \
| sudo gpg --yes --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo apt-get update
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get install -y kubeadm=1.37.2-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.2-1.1 kubectl=1.37.2-1.1
sudo apt-mark hold kubeadm kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubeletExpected (from kubeadm upgrade node,
abbreviated):
[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.
Back on cp1:
k uncordon worker1The same for worker2. Then:
k get nodesExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 86m v1.37.2
worker1 Ready <none> 85m v1.37.2
worker2 Ready <none> 85m v1.37.2
Save your progress? Sign up for free
Drill 7: Renew Certificates
sudo kubeadm certs check-expirationExpected (abbreviated):
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTH...
admin.conf Nov 17, 2027 22:29 UTC 364d ca
apiserver Nov 17, 2027 22:29 UTC 364d ca
...
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 14, 2036 21:11 UTC 9y no
All certificates are valid for a full year again:
kubeadm upgrade apply renews them automatically. If you
upgrade every year, you never have to renew them by hand. The CAs last
ten years. By hand, it works like this:
sudo kubeadm certs renew allExpected (last lines):
certificate embedded in the kubeconfig file for the super-admin renewed
Done renewing certificates. You must restart the kube-apiserver,
kube-controller-manager, kube-scheduler and etcd, so that they can use the new
certificates.
The components read their certificates only at startup. You restart static pods by briefly removing their manifest – here the scheduler and controller manager:
sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
sudo crictl ps --name 'kube-(scheduler|controller-manager)' -q | wc -lExpected (after about 20 seconds):
0
sudo mv /tmp/kube-scheduler.yaml /tmp/kube-controller-manager.yaml \
/etc/kubernetes/manifests/Do the same with kube-apiserver.yaml and
etcd.yaml. Finally, take over the renewed admin
kubeconfig:
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
k get pods -n kube-system -l tier=control-planeExpected: four pods, all Running.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show whether etcd is healthy. Target time: 45 s
- Show the kubelet version of all nodes on one line. Target time: 20 s
- Check whether
drain worker1would succeed without further flags – without actually draining the node. Target time: 30 s
Solution
sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key endpoint health -w table
k get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}{"\n"}'
k drain worker1 --ignore-daemonsets --dry-run=clientExpected (abbreviated):
+------------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+------------------------+--------+-------------+-------+
| https://127.0.0.1:2379 | true | 33.755384ms | |
+------------------------+--------+-------------+-------+
v1.37.2 v1.37.2 v1.37.2
node/worker1 cordoned (dry run)
error: unable to drain node "worker1" due to error: cannot delete Pods with local
storage (use --delete-emptydir-data to override): ...
Save your progress? Sign up for free
Checkpoint
k version | tail -1Expected: Server Version: v1.37.2
k get pod etcd-cp1 -n kube-system -o jsonpath='{.spec.containers[0].image}{"\n"}'Expected:
registry.k8s.io/etcd:3.7.0-0
sudo ls -lh /var/lib/etcd-backup.dbExpected: a file of about 18 MB
(18M).
Challenge
After the upgrade, etcd 3.7 is running. Is your snapshot from earlier still usable? Check it with the new etcd version without touching the cluster.
Hint: etcdutl works offline and does
not need a running etcd.
Solution
cd /tmp
E=3.7.0
U=https://github.com/etcd-io/etcd/releases/download/v$E
curl -sSLO $U/etcd-v$E-linux-amd64.tar.gz
tar xzf etcd-v$E-linux-amd64.tar.gz
sudo ./etcd-v$E-linux-amd64/etcdutl snapshot status /var/lib/etcd-backup.db -w tableExpected:
┌──────────┬──────────┬────────────┬────────────┬─────────┐
│ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 7c9f621d │ 16769 │ 902 │ 19 MB │ 3.6.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘
The same values as in Drill 2, just with a new table border. From now on, use the etcdctl version of the running etcd for new snapshots.
Clean up:
k delete ns vorher
sudo rm -rf /var/lib/etcd-altSave your progress? Sign up for free
Exam Tip
- etcd tasks: copy the endpoint and certificate paths from the
etcd manifest, don’t guess.
sudo grep -E 'cert|key|trusted|listen-client' /etc/kubernetes/manifests/etcd.yaml. - Restore in the exam: new
--data-dir, then switch thehostPathin the manifest. Wait until the API server is back. - Upgrade: the docs page “Upgrading kubeadm clusters” has all the commands to copy. Just adjust the version numbers.
Enterprise Note
- Automate etcd snapshots (CronJob or systemd timer), encrypt them, and store them outside the cluster; practice the restore regularly.
- Run upgrades in a staging environment first, and check the
deprecation list beforehand (
kubectl deprecations, Pluto). - With three control planes, the upgrade runs without API downtime (Day 16).