Day 16: Highly Available Control Plane
Goal: Three control planes with stacked etcd behind a virtual IP – and a cluster that survives the failure of a node. · Duration: approx. 90 min · Prerequisite: Day 13
Briefing
CKA: “Implement and configure a highly-available control plane” (part of the 25 %).
VIP 172.16.0.245:6443 (kube-vip, ARP)
/ | \
cp1 .126 worker1 .228 worker2 .83
API + etcd API + etcd API + etcd <- "stacked etcd"
- All clients talk to the VIP, never to a single node IP.
- etcd needs a majority (quorum): with three members, one may fail.
- kube-vip runs as a static Pod on every control plane and keeps the VIP on exactly one node.
Today you rebuild the cluster. All workloads and add-ons are lost in the process – Drill 7 reinstalls the most important ones.
Drill 1: Tear Down the Old Cluster
On worker1, worker2 and finally cp1:
sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d /var/lib/calico ~/.kube
sudo iptables -F
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -X
sudo systemctl rebootThe reboot removes the Calico network interfaces. Afterwards, on every VM:
sudo ls /etc/kubernetes/manifests | wc -l
ip -o link | grep -c -E 'cali|vxlan'Expected: 0 twice. containerd, kubeadm
and kubelet stay installed.
Save your progress? Sign up for free
Drill 2: Prepare the VIP
First check that the address is free:
ping -c2 -W1 172.16.0.245Expected: 100% packet loss.
On cp1, generate the kube-vip manifest – the image ships with a generator for this:
KV=v1.3.1
sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KV
sudo mkdir -p /etc/kubernetes/manifests
sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KV vip /kube-vip manifest pod \
--interface eth0 --address 172.16.0.245 --controlplane --arp --leaderElection \
| sudo tee /etc/kubernetes/manifests/kube-vip.yaml >/dev/null
sudo grep -n -E 'image:|value: 172|admin.conf' /etc/kubernetes/manifests/kube-vip.yamlExpected:
42: value: 172.16.0.245
45: image: ghcr.io/kube-vip/kube-vip:v1.3.1
57: - mountPath: /etc/kubernetes/admin.conf
66: path: /etc/kubernetes/admin.conf
A trap on the first node: admin.conf only gets
permissions after the API server has started – but kube-vip is supposed
to provide the VIP before that. So for kubeadm init,
temporarily use super-admin.conf:
sudo sed -i 's#\(path: /etc/kubernetes/\)admin.conf#\1super-admin.conf#' \
/etc/kubernetes/manifests/kube-vip.yamlSave your progress? Sign up for free
Drill 3: The First Control Plane
sudo kubeadm init --kubernetes-version=v1.37.2 \
--control-plane-endpoint=172.16.0.245:6443 --upload-certs \
--pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=172.16.0.126Expected (abridged):
You can now join any number of control-plane nodes running the following command on each
as root:
kubeadm join 172.16.0.245:6443 --token e8f79e.2x4p86m4py4k91ba \
--discovery-token-ca-cert-hash sha256:a0f7dc6016fe9fffd2de088f9eb2162cb1f936bec... \
--control-plane --certificate-key 64441cac60641c6bf7c5e4e3676a04cbb2417210bde38e2...
Please note that the certificate-key gives access to cluster sensitive data, keep it
secret!
--control-plane-endpoint writes the VIP into all
certificates and kubeconfigs. --upload-certs stores the CA
keys encrypted in the cluster so that additional control planes can
fetch them with the certificate-key – for two hours.
Switch kube-vip back to admin.conf, set up kubectl,
install Calico:
sudo sed -i 's#\(path: /etc/kubernetes/\)super-admin.conf#\1admin.conf#' \
/etc/kubernetes/manifests/kube-vip.yaml
mkdir -p ~/.kube
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
k config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'Expected: https://172.16.0.245:6443
The switch restarts kube-vip; for a few seconds, kubectl
then reports no route to host. After that:
C=https://raw.githubusercontent.com/projectcalico/calico/v3.33.1/manifests
k create -f $C/operator-crds.yaml
k create -f $C/tigera-operator.yaml
k create -f $C/custom-resources.yaml
ip -o -4 addr show eth0Expected (abridged):
2: eth0 inet 172.16.0.126/24 metric 100 brd 172.16.0.255 scope global dynamic eth0
...
2: eth0 inet 172.16.0.245/32 scope global deprecated eth0 ...
The VIP sits on cp1 as a second address.
Save your progress? Sign up for free
Drill 4: Two More Control Planes
On cp1, build a fresh join command for control
planes:
sudo kubeadm init phase upload-certs --upload-certs | tail -1
kubeadm token create --print-join-commandThe first output is the certificate-key, the second the
normal join command. On worker1, first prepare kube-vip
(as in Drill 2, but without super-admin), then join:
KV=v1.3.1
sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KV
sudo mkdir -p /etc/kubernetes/manifests
sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KV vip /kube-vip manifest pod \
--interface eth0 --address 172.16.0.245 --controlplane --arp --leaderElection \
| sudo tee /etc/kubernetes/manifests/kube-vip.yaml >/dev/null
sudo kubeadm join 172.16.0.245:6443 --token <TOKEN> \
--discovery-token-ca-cert-hash sha256:<HASH> \
--control-plane --certificate-key <KEY> --apiserver-advertise-address=172.16.0.228Expected (abridged):
This node has joined the cluster and a new control plane instance was created:
Do the same on worker2 with
--apiserver-advertise-address=172.16.0.83. Back on
cp1:
k wait --for=condition=Ready nodes --all --timeout=300s
k get nodesExpected:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 4m33s v1.37.2
worker1 Ready control-plane 110s v1.37.2
worker2 Ready control-plane 35s v1.37.2
Save your progress? Sign up for free
Drill 5: etcd with Three Members
sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key member list -w tableExpected:
+------------------+---------+---------+---------------------------+--------------------
-------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS
|
+------------------+---------+---------+---------------------------+--------------------
-------+
| 1ff2278c0fd14cc3 | started | cp1 | https://172.16.0.126:2380 |
https://172.16.0.126:2379 |
| 465cb3aab965445d | started | worker2 | https://172.16.0.83:2380 |
https://172.16.0.83:2379 |
| eda1c7a40b588aae | started | worker1 | https://172.16.0.228:2380 |
https://172.16.0.228:2379 |
+------------------+---------+---------+---------------------------+--------------------
-------+
(Column IS LEARNER omitted.) So that workloads also run
on all three nodes, remove the control-plane taints, and check who holds
the VIP:
k taint nodes --all node-role.kubernetes.io/control-plane-
k get lease -n kube-system plndr-cp-lock -o jsonpath='{.spec.holderIdentity}{"\n"}'Expected:
node/cp1 untainted
node/worker1 untainted
node/worker2 untainted
cp1
Save your progress? Sign up for free
Drill 6: Failure of a Node
Set up kubectl on worker1 as well – the kubeconfig there also points to the VIP:
mkdir -p ~/.kube
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/configNow trigger the failure on cp1:
sudo systemctl rebootImmediately afterwards, on worker1:
k get nodes
k get lease -n kube-system plndr-cp-lock -o jsonpath='{.spec.holderIdentity}{"\n"}'
k create deployment during-outage --image=nginx:1.29Expected (abridged):
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 5m4s v1.37.2
...
worker1
deployment.apps/during-outage created
The VIP has moved to worker1, the API responds, writes
work – with two of three members, etcd still has the majority.
cp1 still shows Ready because the node
controller waits about 40 seconds. After the reboot, cp1
comes back on its own:
k wait --for=condition=Ready node/cp1 --timeout=180s
k delete deployment during-outageAnd what if two of three fail? Stop etcd on worker1 and worker2:
sudo mv /etc/kubernetes/manifests/etcd.yaml /root/On cp1, after about 30 seconds:
k get nodes --request-timeout=10sExpected:
Unable to connect to the server: dial tcp 172.16.0.245:6443: connect: no route to host
Without quorum, kube-vip cannot renew its lease and gives up the VIP;
etcd itself reports Error: unhealthy cluster. Back on
worker1 and worker2:
sudo mv /root/etcd.yaml /etc/kubernetes/manifests/After about a minute, the cluster responds again.
Save your progress? Sign up for free
Drill 7: Bring Back the Add-ons
The rebuild removed everything from Days 6–12. You know the commands – here they are bundled. metrics-server (Day 6):
M=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1
k apply -f $M/components.yaml
k patch deployment metrics-server -n kube-system --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
"value":"--kubelet-insecure-tls"}]'MetalLB with a pool (Day 7):
ML=https://raw.githubusercontent.com/metallb/metallb/v0.16.2/config/manifests
k apply -f $ML/metallb-native.yaml
k wait -n metallb-system --for=condition=Ready pod --all --timeout=180s
cat <<'EOF' | k apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: lab
namespace: metallb-system
spec:
addresses:
- 172.16.0.242-172.16.0.244
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: lab
namespace: metallb-system
spec:
ipAddressPools:
- lab
EOFNFS CSI and default StorageClass (Day 10; the NFS server on
cp1 is still running):
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system \
--version 4.13.6
cat <<'EOF' | k apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: nfs.csi.k8s.io
parameters:
server: 172.16.0.126
share: /srv/nfs/k8s
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
- nfsvers=4.1
EOFGateway API CRDs and Traefik (Day 8):
G=https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0
k apply --server-side -f $G/standard-install.yaml
helm install traefik traefik/traefik --version 42.4.1 -n traefik --create-namespace \
--set providers.kubernetesGateway.enabled=true --set gateway.enabled=false
k rollout status deployment traefik -n traefik
k get svc traefik -n traefik
k get scExpected (abridged):
deployment "traefik" successfully rolled out
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
traefik LoadBalancer 10.106.40.170 172.16.0.242 80:32671/TCP,443:30441/TCP
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
ALLOWVOLUMEEXPANSION
nfs (default) nfs.csi.k8s.io Delete Immediate true
You no longer need cert-manager from Day 12.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which node is the etcd leader? Target time: 45 s
- How many etcd members may fail with five control planes? Target time: 10 s
- Show the address kubectl uses to reach the cluster. Target time: 15 s
Solution
E=https://172.16.0.126:2379,https://172.16.0.228:2379,https://172.16.0.83:2379
sudo etcdctl --endpoints=$E \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key endpoint status \
| awk -F', ' '{print $1, $9}'
k config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'Expected:
https://172.16.0.126:2379 false
https://172.16.0.228:2379 false
https://172.16.0.83:2379 true
https://172.16.0.245:6443
The ninth column of endpoint status is
IS LEADER. The leader can change – after the quorum test
from Drill 6, it was worker2 here.
On 2: two. Quorum is (n / 2) + 1, rounded down – so with
five, it is three.
Save your progress? Sign up for free
Checkpoint
k get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -lExpected: 3
k get pods -n kube-system -l component=etcd --no-headers | grep -c RunningExpected: 3
k get pods -n kube-system | grep -c kube-vipExpected: 3 – kube-vip Pods carry no
labels, so here we match by name.
Challenge
A new colleague is supposed to add a fourth control plane. The
certificate-key from Drill 4 has expired. Generate a
complete, fresh join command for a control plane – without running
it.
Hint: Two commands; the second one knows the
--certificate-key option.
Solution
KEY=$(sudo kubeadm init phase upload-certs --upload-certs | tail -1)
kubeadm token create --print-join-command --certificate-key $KEYExpected (abridged):
kubeadm join 172.16.0.245:6443 --token ... --discovery-token-ca-cert-hash sha256:...
--control-plane --certificate-key ...
Save your progress? Sign up for free
Exam Tip
- In the exam, the HA task is often phrased as “join this node as an
additional control plane”. What matters:
--control-planeand--certificate-key. kubeadm init phase upload-certs --upload-certsgenerates a new key when the old one (two hours) has expired.- Quorum: with 3 members, 1 failure; with 5 members, 2. Even numbers gain you nothing.
Enterprise Note
- Instead of kube-vip, often an external load balancer (HAProxy + keepalived, F5, cloud LB) in front of the API servers.
- Spread control planes across separate hosts, racks or zones – three VMs on the same hypervisor are not high availability.
- Large clusters separate etcd from the API servers (“external etcd”).