Skip to content
Hands-on IT IT Drills

Day 15: Control plane and etcd

Goal: You check the API server and etcd, back up etcd before every intervention, find the cause when the API server no longer starts, and recognize expired or wrong certificates by their message. · Duration: approx. 75 min · Prerequisite: Day 14

Briefing

  • The API server’s /readyz and /livez describe its own state; [-] marks the failed check.
  • If the API server is gone, kubectl is no help: crictl, journalctl and the files under /var/log/pods on cp1 work without Kubernetes.
  • Before every intervention on the control plane: an etcd snapshot.
  • On the client, certificate errors give themselves away by their message: Unauthorized (your client certificate – only the API server log knows the reason), x509: … expired (the server’s certificate), unknown authority (the CA in your kubeconfig).
Fault Symptom Where the answer is
API server does not start connection … refused crictl logs, manifest
Client certificate expired Unauthorized openssl x509, API log
Wrong CA in the kubeconfig unknown authority certificate-authority-data

Drill 1: Is the API healthy?

The API server checks itself. /livez answers “is the process alive?”, /readyz “can it serve requests, does etcd respond?”. ?verbose lists every individual check:

# on cp1
mkdir -p ~/debug/tag15 && cd ~/debug/tag15
kubectl get --raw '/readyz?verbose' | head -4
kubectl get --raw '/readyz?verbose' | grep -c 'ok$'
kubectl get --raw '/readyz?verbose' | grep -v 'ok$'
kubectl get --raw '/livez?verbose' | tail -1
kubectl get --raw /readyz/etcd; echo

Expected:

[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
39
readyz check passed
livez check passed
ok

39 checks, none with [-]. A failed one would show up as [-]etcd failed: reason withheld – the API server writes the reason only to its log. You query individual checks directly, as with /readyz/etcd here.

Save your progress? Sign up for free

Drill 2: An etcd snapshot before every intervention

etcd is the cluster’s memory. Before you tinker with the control plane, you take a snapshot:

# on cp1
sudo -i rke2 etcd-snapshot save --name tag15-start 2>&1 | grep -o 'Snapshot .* saved.'
sudo -i rke2 etcd-snapshot ls 2>/dev/null | awk '{print $1, $NF}'

Expected:

Snapshot tag15-start-cp1-1795185570 saved.
Name Created
nach-tag12-cp1-1795167810 2026-11-20T09:43:30Z
vor-tag15-cp1-1795174399 2026-11-20T11:33:19Z
etcd-snapshot-cp1-1795176005 2026-11-20T12:00:05Z
tag15-start-cp1-1795185570 2026-11-20T14:39:30Z

nach-tag12 and vor-tag15 come from setting up the lab, etcd-snapshot-… was taken by RKE2 itself at 12:00 – RKE2 backs up automatically at 00:00 and 12:00 and keeps five of them. Your list shows your own. The snapshots are stored under /var/lib/rancher/rke2/server/db/snapshots/. Without 2>&1 | grep you see warnings such as Unknown flag --cni found in config.yaml, skipping – etcd-snapshot does not know all server options; harmless.

This is how you would restore – read only, do not run: systemctl stop rke2-server, then rke2 server --cluster-reset --cluster-reset-restore-path=<snapshot>, then systemctl start rke2-server. The cluster jumps back to the state of the snapshot; everything after it is gone.

Save your progress? Sign up for free

Drill 3: Ask etcd itself

RKE2 runs etcd as a static Pod; etcdctl is in the image, the client certificates are on the node:

# on cp1
E=$(sudo -i crictl ps --name etcd -q)
T=/var/lib/rancher/rke2/server/tls/etcd
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
  --key $T/server-client.key endpoint health
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
  --key $T/server-client.key endpoint status -w json \
  | jq -c '.[0].Status | {version, dbSize, dbSizeInUse, raftIndex}'
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
  --key $T/server-client.key member list -w simple | cut -c1-88
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
  --key $T/server-client.key alarm list | wc -l

Expected:

127.0.0.1:2379 is healthy: successfully committed proposal: took = 28.185044ms
{"version":"3.7.1","dbSize":59551744,"dbSizeInUse":21426176,"raftIndex":180212}
1ff2278c0fd14cc3, started, cp1-6c528aa2, https://172.16.0.126:2380, https://172.16.0.126
0

One member, healthy, a 60 MB database, a good 20 MB of it in use. alarm list prints one line per active alarm – none here. The exit code does not help: it is 0 even with an active alarm. A NOSPACE would mean: quota full, etcd no longer accepts writes. With etcd 3.7, -w table is over 200 characters wide, hence JSON.

Save your progress? Sign up for free

Drill 4: The API server does not start

Break it: A feature gate that does not exist – just like one that an upgrade has removed:

Warning: Afterwards the control plane is gone until you fix the fault. Check the snapshot from Drill 2 beforehand.

# on cp1
sudo mkdir -p /etc/rancher/rke2/config.yaml.d
echo 'kube-apiserver-arg: [feature-gates=DebugBuchGate=true]' \
  | sudo tee /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
sudo systemctl restart rke2-server --no-block

Expected:

kube-apiserver-arg: [feature-gates=DebugBuchGate=true]

--no-block returns immediately instead of waiting for a start that never finishes.

Symptom (after about a minute):

# on cp1
sleep 60
kubectl get nodes
systemctl is-active rke2-server

Expected (line wrapped):

The connection to the server 127.0.0.1:6443 was refused - did you specify the right
  host or port?
activating

Investigate: crictl asks containerd directly, without any API server:

# on cp1
sudo -i crictl ps -a --name kube-apiserver | tr -s ' ' | cut -c1-88
sudo -i crictl logs $(sudo -i crictl ps -a --name kube-apiserver -q | head -1) 2>&1 \
  | tail -1 | cut -c1-88
sudo sh -c 'tail -qn1 /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log' \
  | sed 's/^[^ ]* [^ ]* F //' | cut -c1-88
sudo grep feature-gates /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
sudo journalctl -u rke2-server -b --no-pager | grep -o 'Polling for API server.*' \
  | tail -1 | cut -c1-88

Expected (abridged):

CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
4f2b11200f3d4 6d31c161dcb90 10 seconds ago Exited kube-apiserver 3 a1664c15bb8a3 kube-ap
E1120 14:40:25.164233       1 run.go:72] "command failed" err="unrecognized feature gate
E1120 14:40:25.164233       1 run.go:72] "command failed" err="unrecognized feature gate
    - --feature-gates=DebugBuchGate=true
Polling for API server readiness: GET /readyz failed: Get \"https://127.0.0.1:6443/ready

The container log, the same line as a file under /var/log/pods/ (it survives even the deletion of the container), the static Pod manifest rendered by RKE2, and the journal of rke2-server, which is waiting for the API server.

Cause: kube-apiserver aborts while parsing its flags: unrecognized feature gate: DebugBuchGate. The kubelet keeps restarting the static Pod (ATTEMPT goes up). Without the API server, controllers, scheduler and kubectl stand still – the running Pods keep working, though.

Fix: RKE2 rewrites the manifest on every start, so fixing the configuration is enough:

# on cp1
sudo rm /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
sudo rmdir /etc/rancher/rke2/config.yaml.d
sudo systemctl restart rke2-server

Verify:

# on cp1
kubectl get --raw /readyz; echo
sudo -i crictl ps --name kube-apiserver | tr -s ' ' | cut -c1-88

Expected:

ok
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
86397b7cb87e9 6d31c161dcb90 36 seconds ago Running kube-apiserver 0 85353367e72ff kube-a

In the lab, restarting rke2-server takes just under a minute.

Save your progress? Sign up for free

Drill 5: The certificate that expires after ten minutes

Break it: The user anna gets a client certificate from the Kubernetes CA – with the shortest permitted lifetime of 600 seconds:

# on cp1
kubectl create namespace tag15
kubectl -n tag15 create configmap hallo --from-literal=gruss=moin
kubectl -n tag15 create rolebinding anna-view --clusterrole=view --user=anna
openssl genrsa -out anna.key 2048
openssl req -new -key anna.key -subj '/CN=anna' -out anna.csr
cat > anna-csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: {name: anna}
spec:
  request: $(base64 -w0 anna.csr)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 600
  usages: [client auth]
EOF
kubectl apply -f anna-csr.yaml
kubectl certificate approve anna
kubectl get csr anna

Expected (abridged):

namespace/tag15 created
configmap/hallo created
rolebinding.rbac.authorization.k8s.io/anna-view created
certificatesigningrequest.certificates.k8s.io/anna created
certificatesigningrequest.certificates.k8s.io/anna approved
NAME   AGE   SIGNERNAME                            REQUESTOR      REQUESTEDDURATION ...
anna   0s    kubernetes.io/kube-apiserver-client   system:admin   10m               ...

This time EOF has no quotes: the shell is supposed to substitute $(base64 …). The certificate becomes a kubeconfig of its own:

# on cp1
kubectl get csr anna -o jsonpath='{.status.certificate}' | base64 -d > anna.crt
kubectl config view --minify --raw \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
K=anna.kubeconfig
kubectl config set-cluster debug --server=https://127.0.0.1:6443 \
  --certificate-authority=ca.crt --embed-certs --kubeconfig=$K
kubectl config set-credentials anna --client-certificate=anna.crt \
  --client-key=anna.key --embed-certs --kubeconfig=$K
kubectl config set-context anna --cluster=debug --user=anna --namespace=tag15 \
  --kubeconfig=$K
kubectl config use-context anna --kubeconfig=$K
kubectl --kubeconfig=$K get configmaps

Expected:

Cluster "debug" set.
User "anna" set.
Context "anna" created.
Switched to context "anna".
NAME               DATA   AGE
hallo              1      4s
kube-root-ca.crt   1      4s

Symptom: Ten minutes later, with nothing changed in the kubeconfig:

# on cp1
sleep 600
kubectl --kubeconfig=anna.kubeconfig get configmaps

Expected:

error: You must be logged in to the server (Unauthorized)

Investigate: Unauthorized means: the API server does not know who you are. With client certificates, the first suspect is the expiry date:

# on cp1
kubectl config view --kubeconfig=anna.kubeconfig --raw \
  -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -subject -enddate
date -u
kubectl -n kube-system logs kube-apiserver-cp1 --since=2m \
  | grep -m1 -o 'x509: certificate has expired[^,]*' | head -1 | fold -w 86

Expected:

subject=CN = anna
notAfter=Nov 20 14:51:26 2026 GMT
Fri Nov 20 14:51:29 UTC 2026
x509: certificate has expired or is not yet valid: current time 2026-11-20T14:51:28Z i
s after 2026-11-20T14:51:26Z

Cause: The client certificate has expired. The API server rejects it on every request, but for security reasons tells the client only Unauthorized; it writes the reason to its own log, which you can read with admin access.

Fix: A new certificate the same way. The old CSR has to go first, and this time the lifetime is one day – otherwise you will be back here in ten minutes:

# on cp1
kubectl delete csr anna
sed -i 's/expirationSeconds: 600/expirationSeconds: 86400/' anna-csr.yaml
kubectl apply -f anna-csr.yaml
kubectl certificate approve anna
kubectl get csr anna -o jsonpath='{.status.certificate}' | base64 -d > anna.crt
kubectl config set-credentials anna --client-certificate=anna.crt \
  --client-key=anna.key --embed-certs --kubeconfig=anna.kubeconfig

Expected:

certificatesigningrequest.certificates.k8s.io "anna" deleted
certificatesigningrequest.certificates.k8s.io/anna created
certificatesigningrequest.certificates.k8s.io/anna approved
User "anna" set.

Verify:

# on cp1
openssl x509 -in anna.crt -noout -enddate
kubectl --kubeconfig=anna.kubeconfig get configmap hallo

Expected:

notAfter=Nov 21 14:46:30 2026 GMT
NAME    DATA   AGE
hallo   1      10m

The new certificate is valid until tomorrow; the signer counts the lifetime from five minutes before issuance, hence 14:46 instead of 14:51.

Message on the client What is wrong
Unauthorized your client certificate or token
x509: certificate has expired … the server’s serving certificate or your clock
x509: … signed by unknown authority the CA in your kubeconfig

Save your progress? Sign up for free

Drill 6: When do the cluster certificates expire?

RKE2 manages its own certificates and shows them with a single command:

# on cp1
sudo -i rke2 certificate check --output json 2>/dev/null \
  | jq -r '.Certificates[] | select(.Usages[0] != "CertSign")
      | "\(.ExpiryTime) \(.Status) \(.Filename)"' | sort | head -4
sudo -i rke2 certificate check --output json 2>/dev/null \
  | jq -r '.Certificates[].Status' | sort | uniq -c

Expected:

2027-11-20T05:29:51Z OK client-admin.crt
2027-11-20T05:29:51Z OK client-auth-proxy.crt
2027-11-20T05:29:51Z OK client-controller.crt
2027-11-20T05:29:51Z OK client-kube-apiserver.crt
     34 OK

Leaf certificates are valid for one year, the CAs for ten. On start, RKE2 renews every certificate that expires in less than 120 days. By hand: systemctl stop rke2-server, rke2 certificate rotate, systemctl start rke2-server – afterwards /etc/rancher/rke2/rke2.yaml is new as well. Copied kubeconfigs keep working until the old expiry date – then copy them again.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show only the failed checks of /readyz. Target time: 20 s
  2. Show all etcd snapshots with their names. Target time: 20 s
  3. Show the subject and expiry date of the client certificate in ~/.kube/config. Target time: 45 s

Solution

# on cp1
kubectl get --raw '/readyz?verbose' | grep '^\[-\]'; echo "rc=$?"
sudo -i rke2 etcd-snapshot ls 2>/dev/null | awk '{print $1}'
kubectl config view --raw --minify \
  -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -subject -enddate

Expected:

rc=1
Name
nach-tag12-cp1-1795167810
vor-tag15-cp1-1795174399
etcd-snapshot-cp1-1795176005
tag15-start-cp1-1795185570
subject=O = system:masters, CN = system:admin
notAfter=Nov 20 05:29:51 2027 GMT

rc=1 means: grep found no line with [-] – all green. The RKE2 kubeconfig belongs to system:admin in system:masters (Day 11).

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl get --raw /readyz; echo
sudo ls /etc/rancher/rke2/

Expected:

ok
config.yaml  registries.yaml  rke2-pss.yaml  rke2.yaml

No more config.yaml.d – the broken drop-in from Drill 4 is gone.

Challenge

A colleague sends you his kubeconfig fremd.kubeconfig with the request: “Hasn’t worked since the reinstall.” Find out which certificate does not match, and repair the file without touching the credentials.

# on cp1
cp anna.kubeconfig fremd.kubeconfig
openssl req -x509 -newkey rsa:2048 -nodes -keyout fremd-ca.key -out fremd-ca.crt \
  -days 1 -subj '/CN=fremde-ca' 2>/dev/null
kubectl config set-cluster debug --certificate-authority=fremd-ca.crt --embed-certs \
  --kubeconfig=fremd.kubeconfig

Hint: Compare the CA in the file with the issuer of the certificate the API server presents.

Solution

# on cp1
kubectl --kubeconfig=fremd.kubeconfig get configmaps 2>&1 | fold -w 86
kubectl config view --kubeconfig=fremd.kubeconfig --raw \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
  | base64 -d | openssl x509 -noout -subject
echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null \
  | openssl x509 -noout -issuer
kubectl config set-cluster debug --certificate-authority=ca.crt --embed-certs \
  --kubeconfig=fremd.kubeconfig
kubectl --kubeconfig=fremd.kubeconfig get configmap hallo

Expected:

Unable to connect to the server: tls: failed to verify certificate: x509: certificate
signed by unknown authority
subject=CN = fremde-ca
issuer=CN = rke2-server-ca@1795156191
Cluster "debug" set.
NAME    DATA   AGE
hallo   1      21m

AGE depends on how long you have taken since Drill 5 – Anna’s certificate is now valid for one day. The kubeconfig trusts fremde-ca (‘foreign CA’), while the API server presents a certificate from rke2-server-ca@…. After a reinstall, the cluster has a new CA – old kubeconfigs then fail in exactly this way. The credentials (users) stay untouched; only clusters[].certificate-authority-data is replaced.

Save your progress? Sign up for free

Cleanup

The snapshots stay where they are – they are your way back for the next few days.

# on cp1
kubectl delete csr anna
kubectl delete namespace tag15
rm -f anna.key fremd-ca.key anna.kubeconfig fremd.kubeconfig
kubectl get csr

Expected:

certificatesigningrequest.certificates.k8s.io "anna" deleted
namespace "tag15" deleted
No resources found

Interview Note

  • The API server does not respond – what now? On the control plane nodes: crictl ps -a --name kube-apiserver, crictl logs, /var/log/pods, the service’s journal; the cause is almost always in the last line of the container log.
  • What do you do before an intervention on the control plane? Check etcd (endpoint health, alarm list) and take a snapshot; know the restore path before you need it.
  • Unauthorized vs. x509? Unauthorized: the server rejects your client certificate/token. x509 on the client: you do not trust the server (CA, expiry, clock).

Enterprise Note

  • Three or five etcd members, plus snapshots to S3 (etcd-s3 in RKE2); a snapshot on the same node does not help when the disk dies.
  • Practice the restore regularly – on a copy, not in production.
  • Monitor certificate expiry (apiserver_client_certificate_expiration_seconds, rke2 certificate check via cron); people get short-lived access via OIDC instead of client certificates.
  • Test changes to config.yaml on one server first, then roll them out one after the other – never on all control plane nodes at once.
  • On Talos, the same questions are answered via talosctl etcd, talosctl logs and kubernetesdynamiccerts (Days 18 and 19).