Skip to content
Hands-on IT IT Drills

Day 16: Highly Available Control Plane

Goal: Three control planes with stacked etcd behind a virtual IP – and a cluster that survives the failure of a node. · Duration: approx. 90 min · Prerequisite: Day 13

Briefing

CKA: “Implement and configure a highly-available control plane” (part of the 25 %).

                  VIP 172.16.0.245:6443  (kube-vip, ARP)
                 /            |              \
        cp1 .126         worker1 .228       worker2 .83
     API + etcd          API + etcd         API + etcd     <- "stacked etcd"
  • All clients talk to the VIP, never to a single node IP.
  • etcd needs a majority (quorum): with three members, one may fail.
  • kube-vip runs as a static Pod on every control plane and keeps the VIP on exactly one node.

Today you rebuild the cluster. All workloads and add-ons are lost in the process – Drill 7 reinstalls the most important ones.

Drill 1: Tear Down the Old Cluster

On worker1, worker2 and finally cp1:

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d /var/lib/calico ~/.kube
sudo iptables -F
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -X
sudo systemctl reboot

The reboot removes the Calico network interfaces. Afterwards, on every VM:

sudo ls /etc/kubernetes/manifests | wc -l
ip -o link | grep -c -E 'cali|vxlan'

Expected: 0 twice. containerd, kubeadm and kubelet stay installed.

Save your progress? Sign up for free

Drill 2: Prepare the VIP

First check that the address is free:

ping -c2 -W1 172.16.0.245

Expected: 100% packet loss.

On cp1, generate the kube-vip manifest – the image ships with a generator for this:

KV=v1.3.1
sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KV
sudo mkdir -p /etc/kubernetes/manifests
sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KV vip /kube-vip manifest pod \
  --interface eth0 --address 172.16.0.245 --controlplane --arp --leaderElection \
  | sudo tee /etc/kubernetes/manifests/kube-vip.yaml >/dev/null
sudo grep -n -E 'image:|value: 172|admin.conf' /etc/kubernetes/manifests/kube-vip.yaml

Expected:

42:      value: 172.16.0.245
45:    image: ghcr.io/kube-vip/kube-vip:v1.3.1
57:    - mountPath: /etc/kubernetes/admin.conf
66:      path: /etc/kubernetes/admin.conf

A trap on the first node: admin.conf only gets permissions after the API server has started – but kube-vip is supposed to provide the VIP before that. So for kubeadm init, temporarily use super-admin.conf:

sudo sed -i 's#\(path: /etc/kubernetes/\)admin.conf#\1super-admin.conf#' \
  /etc/kubernetes/manifests/kube-vip.yaml

Save your progress? Sign up for free

Drill 3: The First Control Plane

sudo kubeadm init --kubernetes-version=v1.37.2 \
  --control-plane-endpoint=172.16.0.245:6443 --upload-certs \
  --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=172.16.0.126

Expected (abridged):

You can now join any number of control-plane nodes running the following command on each
as root:

  kubeadm join 172.16.0.245:6443 --token e8f79e.2x4p86m4py4k91ba \
    --discovery-token-ca-cert-hash sha256:a0f7dc6016fe9fffd2de088f9eb2162cb1f936bec... \
    --control-plane --certificate-key 64441cac60641c6bf7c5e4e3676a04cbb2417210bde38e2...

Please note that the certificate-key gives access to cluster sensitive data, keep it
secret!

--control-plane-endpoint writes the VIP into all certificates and kubeconfigs. --upload-certs stores the CA keys encrypted in the cluster so that additional control planes can fetch them with the certificate-key – for two hours.

Switch kube-vip back to admin.conf, set up kubectl, install Calico:

sudo sed -i 's#\(path: /etc/kubernetes/\)super-admin.conf#\1admin.conf#' \
  /etc/kubernetes/manifests/kube-vip.yaml
mkdir -p ~/.kube
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
k config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Expected: https://172.16.0.245:6443

The switch restarts kube-vip; for a few seconds, kubectl then reports no route to host. After that:

C=https://raw.githubusercontent.com/projectcalico/calico/v3.33.1/manifests
k create -f $C/operator-crds.yaml
k create -f $C/tigera-operator.yaml
k create -f $C/custom-resources.yaml
ip -o -4 addr show eth0

Expected (abridged):

2: eth0    inet 172.16.0.126/24 metric 100 brd 172.16.0.255 scope global dynamic eth0
...
2: eth0    inet 172.16.0.245/32 scope global deprecated eth0 ...

The VIP sits on cp1 as a second address.

Save your progress? Sign up for free

Drill 4: Two More Control Planes

On cp1, build a fresh join command for control planes:

sudo kubeadm init phase upload-certs --upload-certs | tail -1
kubeadm token create --print-join-command

The first output is the certificate-key, the second the normal join command. On worker1, first prepare kube-vip (as in Drill 2, but without super-admin), then join:

KV=v1.3.1
sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KV
sudo mkdir -p /etc/kubernetes/manifests
sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KV vip /kube-vip manifest pod \
  --interface eth0 --address 172.16.0.245 --controlplane --arp --leaderElection \
  | sudo tee /etc/kubernetes/manifests/kube-vip.yaml >/dev/null
sudo kubeadm join 172.16.0.245:6443 --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane --certificate-key <KEY> --apiserver-advertise-address=172.16.0.228

Expected (abridged):

This node has joined the cluster and a new control plane instance was created:

Do the same on worker2 with --apiserver-advertise-address=172.16.0.83. Back on cp1:

k wait --for=condition=Ready nodes --all --timeout=300s
k get nodes

Expected:

NAME      STATUS   ROLES           AGE     VERSION
cp1       Ready    control-plane   4m33s   v1.37.2
worker1   Ready    control-plane   110s    v1.37.2
worker2   Ready    control-plane   35s     v1.37.2

Save your progress? Sign up for free

Drill 5: etcd with Three Members

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key member list -w table

Expected:

+------------------+---------+---------+---------------------------+--------------------
-------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS         |       CLIENT ADDRS
|
+------------------+---------+---------+---------------------------+--------------------
-------+
| 1ff2278c0fd14cc3 | started |     cp1 | https://172.16.0.126:2380 |
https://172.16.0.126:2379 |
| 465cb3aab965445d | started | worker2 |  https://172.16.0.83:2380 | 
https://172.16.0.83:2379 |
| eda1c7a40b588aae | started | worker1 | https://172.16.0.228:2380 |
https://172.16.0.228:2379 |
+------------------+---------+---------+---------------------------+--------------------
-------+

(Column IS LEARNER omitted.) So that workloads also run on all three nodes, remove the control-plane taints, and check who holds the VIP:

k taint nodes --all node-role.kubernetes.io/control-plane-
k get lease -n kube-system plndr-cp-lock -o jsonpath='{.spec.holderIdentity}{"\n"}'

Expected:

node/cp1 untainted
node/worker1 untainted
node/worker2 untainted
cp1

Save your progress? Sign up for free

Drill 6: Failure of a Node

Set up kubectl on worker1 as well – the kubeconfig there also points to the VIP:

mkdir -p ~/.kube
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

Now trigger the failure on cp1:

sudo systemctl reboot

Immediately afterwards, on worker1:

k get nodes
k get lease -n kube-system plndr-cp-lock -o jsonpath='{.spec.holderIdentity}{"\n"}'
k create deployment during-outage --image=nginx:1.29

Expected (abridged):

NAME      STATUS   ROLES           AGE     VERSION
cp1       Ready    control-plane   5m4s    v1.37.2
...
worker1
deployment.apps/during-outage created

The VIP has moved to worker1, the API responds, writes work – with two of three members, etcd still has the majority. cp1 still shows Ready because the node controller waits about 40 seconds. After the reboot, cp1 comes back on its own:

k wait --for=condition=Ready node/cp1 --timeout=180s
k delete deployment during-outage

And what if two of three fail? Stop etcd on worker1 and worker2:

sudo mv /etc/kubernetes/manifests/etcd.yaml /root/

On cp1, after about 30 seconds:

k get nodes --request-timeout=10s

Expected:

Unable to connect to the server: dial tcp 172.16.0.245:6443: connect: no route to host

Without quorum, kube-vip cannot renew its lease and gives up the VIP; etcd itself reports Error: unhealthy cluster. Back on worker1 and worker2:

sudo mv /root/etcd.yaml /etc/kubernetes/manifests/

After about a minute, the cluster responds again.

Save your progress? Sign up for free

Drill 7: Bring Back the Add-ons

The rebuild removed everything from Days 6–12. You know the commands – here they are bundled. metrics-server (Day 6):

M=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.1
k apply -f $M/components.yaml
k patch deployment metrics-server -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-",
  "value":"--kubelet-insecure-tls"}]'

MetalLB with a pool (Day 7):

ML=https://raw.githubusercontent.com/metallb/metallb/v0.16.2/config/manifests
k apply -f $ML/metallb-native.yaml
k wait -n metallb-system --for=condition=Ready pod --all --timeout=180s
cat <<'EOF' | k apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab
  namespace: metallb-system
spec:
  addresses:
  - 172.16.0.242-172.16.0.244
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab
  namespace: metallb-system
spec:
  ipAddressPools:
  - lab
EOF

NFS CSI and default StorageClass (Day 10; the NFS server on cp1 is still running):

helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system \
  --version 4.13.6
cat <<'EOF' | k apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: nfs.csi.k8s.io
parameters:
  server: 172.16.0.126
  share: /srv/nfs/k8s
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
- nfsvers=4.1
EOF

Gateway API CRDs and Traefik (Day 8):

G=https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0
k apply --server-side -f $G/standard-install.yaml
helm install traefik traefik/traefik --version 42.4.1 -n traefik --create-namespace \
  --set providers.kubernetesGateway.enabled=true --set gateway.enabled=false
k rollout status deployment traefik -n traefik
k get svc traefik -n traefik
k get sc

Expected (abridged):

deployment "traefik" successfully rolled out
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)
traefik   LoadBalancer   10.106.40.170   172.16.0.242   80:32671/TCP,443:30441/TCP
NAME            PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE  
ALLOWVOLUMEEXPANSION
nfs (default)   nfs.csi.k8s.io   Delete          Immediate           true

You no longer need cert-manager from Day 12.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which node is the etcd leader? Target time: 45 s
  2. How many etcd members may fail with five control planes? Target time: 10 s
  3. Show the address kubectl uses to reach the cluster. Target time: 15 s

Solution

E=https://172.16.0.126:2379,https://172.16.0.228:2379,https://172.16.0.83:2379
sudo etcdctl --endpoints=$E \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint status \
  | awk -F', ' '{print $1, $9}'
k config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Expected:

https://172.16.0.126:2379 false
https://172.16.0.228:2379 false
https://172.16.0.83:2379 true
https://172.16.0.245:6443

The ninth column of endpoint status is IS LEADER. The leader can change – after the quorum test from Drill 6, it was worker2 here.

On 2: two. Quorum is (n / 2) + 1, rounded down – so with five, it is three.

Save your progress? Sign up for free

Checkpoint

k get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l

Expected: 3

k get pods -n kube-system -l component=etcd --no-headers | grep -c Running

Expected: 3

k get pods -n kube-system | grep -c kube-vip

Expected: 3 – kube-vip Pods carry no labels, so here we match by name.

Challenge

A new colleague is supposed to add a fourth control plane. The certificate-key from Drill 4 has expired. Generate a complete, fresh join command for a control plane – without running it.

Hint: Two commands; the second one knows the --certificate-key option.

Solution

KEY=$(sudo kubeadm init phase upload-certs --upload-certs | tail -1)
kubeadm token create --print-join-command --certificate-key $KEY

Expected (abridged):

kubeadm join 172.16.0.245:6443 --token ... --discovery-token-ca-cert-hash sha256:...
--control-plane --certificate-key ...

Save your progress? Sign up for free

Exam Tip

  • In the exam, the HA task is often phrased as “join this node as an additional control plane”. What matters: --control-plane and --certificate-key.
  • kubeadm init phase upload-certs --upload-certs generates a new key when the old one (two hours) has expired.
  • Quorum: with 3 members, 1 failure; with 5 members, 2. Even numbers gain you nothing.

Enterprise Note

  • Instead of kube-vip, often an external load balancer (HAProxy + keepalived, F5, cloud LB) in front of the API servers.
  • Spread control planes across separate hosts, racks or zones – three VMs on the same hypervisor are not high availability.
  • Large clusters separate etcd from the API servers (“external etcd”).