Day 15: Control plane and etcd
Goal: You check the API server and etcd, back up etcd before every intervention, find the cause when the API server no longer starts, and recognize expired or wrong certificates by their message. · Duration: approx. 75 min · Prerequisite: Day 14
Briefing
- The API server’s
/readyzand/livezdescribe its own state;[-]marks the failed check. - If the API server is gone,
kubectlis no help:crictl,journalctland the files under/var/log/podsoncp1work without Kubernetes. - Before every intervention on the control plane: an etcd snapshot.
- On the client, certificate errors give themselves away by their
message:
Unauthorized(your client certificate – only the API server log knows the reason),x509: … expired(the server’s certificate),unknown authority(the CA in your kubeconfig).
| Fault | Symptom | Where the answer is |
|---|---|---|
| API server does not start | connection … refused |
crictl logs, manifest |
| Client certificate expired | Unauthorized |
openssl x509, API log |
| Wrong CA in the kubeconfig | unknown authority |
certificate-authority-data |
Drill 1: Is the API healthy?
The API server checks itself. /livez answers “is the
process alive?”, /readyz “can it serve requests, does etcd
respond?”. ?verbose lists every individual check:
# on cp1
mkdir -p ~/debug/tag15 && cd ~/debug/tag15
kubectl get --raw '/readyz?verbose' | head -4
kubectl get --raw '/readyz?verbose' | grep -c 'ok$'
kubectl get --raw '/readyz?verbose' | grep -v 'ok$'
kubectl get --raw '/livez?verbose' | tail -1
kubectl get --raw /readyz/etcd; echoExpected:
[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
39
readyz check passed
livez check passed
ok
39 checks, none with [-]. A failed one would show up as
[-]etcd failed: reason withheld – the API server writes the
reason only to its log. You query individual checks directly, as with
/readyz/etcd here.
Save your progress? Sign up for free
Drill 2: An etcd snapshot before every intervention
etcd is the cluster’s memory. Before you tinker with the control plane, you take a snapshot:
# on cp1
sudo -i rke2 etcd-snapshot save --name tag15-start 2>&1 | grep -o 'Snapshot .* saved.'
sudo -i rke2 etcd-snapshot ls 2>/dev/null | awk '{print $1, $NF}'Expected:
Snapshot tag15-start-cp1-1795185570 saved.
Name Created
nach-tag12-cp1-1795167810 2026-11-20T09:43:30Z
vor-tag15-cp1-1795174399 2026-11-20T11:33:19Z
etcd-snapshot-cp1-1795176005 2026-11-20T12:00:05Z
tag15-start-cp1-1795185570 2026-11-20T14:39:30Z
nach-tag12 and vor-tag15 come from setting
up the lab, etcd-snapshot-… was taken by RKE2 itself at
12:00 – RKE2 backs up automatically at 00:00 and 12:00 and keeps five of
them. Your list shows your own. The snapshots are stored under
/var/lib/rancher/rke2/server/db/snapshots/. Without
2>&1 | grep you see warnings such as
Unknown flag --cni found in config.yaml, skipping –
etcd-snapshot does not know all server options;
harmless.
This is how you would restore – read only, do not
run: systemctl stop rke2-server, then
rke2 server --cluster-reset --cluster-reset-restore-path=<snapshot>,
then systemctl start rke2-server. The cluster jumps back to
the state of the snapshot; everything after it is gone.
Save your progress? Sign up for free
Drill 3: Ask etcd itself
RKE2 runs etcd as a static Pod; etcdctl is in the image,
the client certificates are on the node:
# on cp1
E=$(sudo -i crictl ps --name etcd -q)
T=/var/lib/rancher/rke2/server/tls/etcd
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
--key $T/server-client.key endpoint health
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
--key $T/server-client.key endpoint status -w json \
| jq -c '.[0].Status | {version, dbSize, dbSizeInUse, raftIndex}'
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
--key $T/server-client.key member list -w simple | cut -c1-88
sudo -i crictl exec $E etcdctl --cacert $T/server-ca.crt --cert $T/server-client.crt \
--key $T/server-client.key alarm list | wc -lExpected:
127.0.0.1:2379 is healthy: successfully committed proposal: took = 28.185044ms
{"version":"3.7.1","dbSize":59551744,"dbSizeInUse":21426176,"raftIndex":180212}
1ff2278c0fd14cc3, started, cp1-6c528aa2, https://172.16.0.126:2380, https://172.16.0.126
0
One member, healthy, a 60 MB database, a good 20 MB of it in use.
alarm list prints one line per active alarm – none here.
The exit code does not help: it is 0 even with an active alarm. A
NOSPACE would mean: quota full, etcd no longer accepts
writes. With etcd 3.7, -w table is over 200 characters
wide, hence JSON.
Save your progress? Sign up for free
Drill 4: The API server does not start
Break it: A feature gate that does not exist – just like one that an upgrade has removed:
Warning: Afterwards the control plane is gone until you fix the fault. Check the snapshot from Drill 2 beforehand.
# on cp1
sudo mkdir -p /etc/rancher/rke2/config.yaml.d
echo 'kube-apiserver-arg: [feature-gates=DebugBuchGate=true]' \
| sudo tee /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
sudo systemctl restart rke2-server --no-blockExpected:
kube-apiserver-arg: [feature-gates=DebugBuchGate=true]
--no-block returns immediately instead of waiting for a
start that never finishes.
Symptom (after about a minute):
# on cp1
sleep 60
kubectl get nodes
systemctl is-active rke2-serverExpected (line wrapped):
The connection to the server 127.0.0.1:6443 was refused - did you specify the right
host or port?
activating
Investigate: crictl asks containerd
directly, without any API server:
# on cp1
sudo -i crictl ps -a --name kube-apiserver | tr -s ' ' | cut -c1-88
sudo -i crictl logs $(sudo -i crictl ps -a --name kube-apiserver -q | head -1) 2>&1 \
| tail -1 | cut -c1-88
sudo sh -c 'tail -qn1 /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log' \
| sed 's/^[^ ]* [^ ]* F //' | cut -c1-88
sudo grep feature-gates /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml
sudo journalctl -u rke2-server -b --no-pager | grep -o 'Polling for API server.*' \
| tail -1 | cut -c1-88Expected (abridged):
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
4f2b11200f3d4 6d31c161dcb90 10 seconds ago Exited kube-apiserver 3 a1664c15bb8a3 kube-ap
E1120 14:40:25.164233 1 run.go:72] "command failed" err="unrecognized feature gate
E1120 14:40:25.164233 1 run.go:72] "command failed" err="unrecognized feature gate
- --feature-gates=DebugBuchGate=true
Polling for API server readiness: GET /readyz failed: Get \"https://127.0.0.1:6443/ready
The container log, the same line as a file under
/var/log/pods/ (it survives even the deletion of the
container), the static Pod manifest rendered by RKE2, and the journal of
rke2-server, which is waiting for the API server.
Cause: kube-apiserver aborts while parsing its
flags: unrecognized feature gate: DebugBuchGate. The
kubelet keeps restarting the static Pod (ATTEMPT goes up).
Without the API server, controllers, scheduler and kubectl
stand still – the running Pods keep working, though.
Fix: RKE2 rewrites the manifest on every start, so fixing the configuration is enough:
# on cp1
sudo rm /etc/rancher/rke2/config.yaml.d/90-kaputt.yaml
sudo rmdir /etc/rancher/rke2/config.yaml.d
sudo systemctl restart rke2-serverVerify:
# on cp1
kubectl get --raw /readyz; echo
sudo -i crictl ps --name kube-apiserver | tr -s ' ' | cut -c1-88Expected:
ok
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
86397b7cb87e9 6d31c161dcb90 36 seconds ago Running kube-apiserver 0 85353367e72ff kube-a
In the lab, restarting rke2-server takes just under a
minute.
Save your progress? Sign up for free
Drill 5: The certificate that expires after ten minutes
Break it: The user anna gets a client
certificate from the Kubernetes CA – with the shortest permitted
lifetime of 600 seconds:
# on cp1
kubectl create namespace tag15
kubectl -n tag15 create configmap hallo --from-literal=gruss=moin
kubectl -n tag15 create rolebinding anna-view --clusterrole=view --user=anna
openssl genrsa -out anna.key 2048
openssl req -new -key anna.key -subj '/CN=anna' -out anna.csr
cat > anna-csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: {name: anna}
spec:
request: $(base64 -w0 anna.csr)
signerName: kubernetes.io/kube-apiserver-client
expirationSeconds: 600
usages: [client auth]
EOF
kubectl apply -f anna-csr.yaml
kubectl certificate approve anna
kubectl get csr annaExpected (abridged):
namespace/tag15 created
configmap/hallo created
rolebinding.rbac.authorization.k8s.io/anna-view created
certificatesigningrequest.certificates.k8s.io/anna created
certificatesigningrequest.certificates.k8s.io/anna approved
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION ...
anna 0s kubernetes.io/kube-apiserver-client system:admin 10m ...
This time EOF has no quotes: the shell is supposed to
substitute $(base64 …). The certificate becomes a
kubeconfig of its own:
# on cp1
kubectl get csr anna -o jsonpath='{.status.certificate}' | base64 -d > anna.crt
kubectl config view --minify --raw \
-o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
K=anna.kubeconfig
kubectl config set-cluster debug --server=https://127.0.0.1:6443 \
--certificate-authority=ca.crt --embed-certs --kubeconfig=$K
kubectl config set-credentials anna --client-certificate=anna.crt \
--client-key=anna.key --embed-certs --kubeconfig=$K
kubectl config set-context anna --cluster=debug --user=anna --namespace=tag15 \
--kubeconfig=$K
kubectl config use-context anna --kubeconfig=$K
kubectl --kubeconfig=$K get configmapsExpected:
Cluster "debug" set.
User "anna" set.
Context "anna" created.
Switched to context "anna".
NAME DATA AGE
hallo 1 4s
kube-root-ca.crt 1 4s
Symptom: Ten minutes later, with nothing changed in the kubeconfig:
# on cp1
sleep 600
kubectl --kubeconfig=anna.kubeconfig get configmapsExpected:
error: You must be logged in to the server (Unauthorized)
Investigate: Unauthorized means: the
API server does not know who you are. With client certificates, the
first suspect is the expiry date:
# on cp1
kubectl config view --kubeconfig=anna.kubeconfig --raw \
-o jsonpath='{.users[0].user.client-certificate-data}' \
| base64 -d | openssl x509 -noout -subject -enddate
date -u
kubectl -n kube-system logs kube-apiserver-cp1 --since=2m \
| grep -m1 -o 'x509: certificate has expired[^,]*' | head -1 | fold -w 86Expected:
subject=CN = anna
notAfter=Nov 20 14:51:26 2026 GMT
Fri Nov 20 14:51:29 UTC 2026
x509: certificate has expired or is not yet valid: current time 2026-11-20T14:51:28Z i
s after 2026-11-20T14:51:26Z
Cause: The client certificate has expired. The API
server rejects it on every request, but for security reasons tells the
client only Unauthorized; it writes the reason to its own
log, which you can read with admin access.
Fix: A new certificate the same way. The old CSR has to go first, and this time the lifetime is one day – otherwise you will be back here in ten minutes:
# on cp1
kubectl delete csr anna
sed -i 's/expirationSeconds: 600/expirationSeconds: 86400/' anna-csr.yaml
kubectl apply -f anna-csr.yaml
kubectl certificate approve anna
kubectl get csr anna -o jsonpath='{.status.certificate}' | base64 -d > anna.crt
kubectl config set-credentials anna --client-certificate=anna.crt \
--client-key=anna.key --embed-certs --kubeconfig=anna.kubeconfigExpected:
certificatesigningrequest.certificates.k8s.io "anna" deleted
certificatesigningrequest.certificates.k8s.io/anna created
certificatesigningrequest.certificates.k8s.io/anna approved
User "anna" set.
Verify:
# on cp1
openssl x509 -in anna.crt -noout -enddate
kubectl --kubeconfig=anna.kubeconfig get configmap halloExpected:
notAfter=Nov 21 14:46:30 2026 GMT
NAME DATA AGE
hallo 1 10m
The new certificate is valid until tomorrow; the signer counts the lifetime from five minutes before issuance, hence 14:46 instead of 14:51.
| Message on the client | What is wrong |
|---|---|
Unauthorized |
your client certificate or token |
x509: certificate has expired … |
the server’s serving certificate or your clock |
x509: … signed by unknown authority |
the CA in your kubeconfig |
Save your progress? Sign up for free
Drill 6: When do the cluster certificates expire?
RKE2 manages its own certificates and shows them with a single command:
# on cp1
sudo -i rke2 certificate check --output json 2>/dev/null \
| jq -r '.Certificates[] | select(.Usages[0] != "CertSign")
| "\(.ExpiryTime) \(.Status) \(.Filename)"' | sort | head -4
sudo -i rke2 certificate check --output json 2>/dev/null \
| jq -r '.Certificates[].Status' | sort | uniq -cExpected:
2027-11-20T05:29:51Z OK client-admin.crt
2027-11-20T05:29:51Z OK client-auth-proxy.crt
2027-11-20T05:29:51Z OK client-controller.crt
2027-11-20T05:29:51Z OK client-kube-apiserver.crt
34 OK
Leaf certificates are valid for one year, the CAs for ten. On start,
RKE2 renews every certificate that expires in less than 120 days. By
hand: systemctl stop rke2-server,
rke2 certificate rotate,
systemctl start rke2-server – afterwards
/etc/rancher/rke2/rke2.yaml is new as well. Copied
kubeconfigs keep working until the old expiry date – then copy them
again.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show only the failed checks of
/readyz. Target time: 20 s - Show all etcd snapshots with their names. Target time: 20 s
- Show the subject and expiry date of the client certificate in
~/.kube/config. Target time: 45 s
Solution
# on cp1
kubectl get --raw '/readyz?verbose' | grep '^\[-\]'; echo "rc=$?"
sudo -i rke2 etcd-snapshot ls 2>/dev/null | awk '{print $1}'
kubectl config view --raw --minify \
-o jsonpath='{.users[0].user.client-certificate-data}' \
| base64 -d | openssl x509 -noout -subject -enddateExpected:
rc=1
Name
nach-tag12-cp1-1795167810
vor-tag15-cp1-1795174399
etcd-snapshot-cp1-1795176005
tag15-start-cp1-1795185570
subject=O = system:masters, CN = system:admin
notAfter=Nov 20 05:29:51 2027 GMT
rc=1 means: grep found no line with
[-] – all green. The RKE2 kubeconfig belongs to
system:admin in system:masters (Day 11).
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl get --raw /readyz; echo
sudo ls /etc/rancher/rke2/Expected:
ok
config.yaml registries.yaml rke2-pss.yaml rke2.yaml
No more config.yaml.d – the broken drop-in from Drill 4
is gone.
Challenge
A colleague sends you his kubeconfig fremd.kubeconfig
with the request: “Hasn’t worked since the reinstall.” Find out which
certificate does not match, and repair the file without touching the
credentials.
# on cp1
cp anna.kubeconfig fremd.kubeconfig
openssl req -x509 -newkey rsa:2048 -nodes -keyout fremd-ca.key -out fremd-ca.crt \
-days 1 -subj '/CN=fremde-ca' 2>/dev/null
kubectl config set-cluster debug --certificate-authority=fremd-ca.crt --embed-certs \
--kubeconfig=fremd.kubeconfigHint: Compare the CA in the file with the issuer of the certificate the API server presents.
Solution
# on cp1
kubectl --kubeconfig=fremd.kubeconfig get configmaps 2>&1 | fold -w 86
kubectl config view --kubeconfig=fremd.kubeconfig --raw \
-o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
| base64 -d | openssl x509 -noout -subject
echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null \
| openssl x509 -noout -issuer
kubectl config set-cluster debug --certificate-authority=ca.crt --embed-certs \
--kubeconfig=fremd.kubeconfig
kubectl --kubeconfig=fremd.kubeconfig get configmap halloExpected:
Unable to connect to the server: tls: failed to verify certificate: x509: certificate
signed by unknown authority
subject=CN = fremde-ca
issuer=CN = rke2-server-ca@1795156191
Cluster "debug" set.
NAME DATA AGE
hallo 1 21m
AGE depends on how long you have taken since Drill 5 –
Anna’s certificate is now valid for one day. The kubeconfig trusts
fremde-ca (‘foreign CA’), while the API server presents a
certificate from rke2-server-ca@…. After a reinstall, the
cluster has a new CA – old kubeconfigs then fail in exactly this way.
The credentials (users) stay untouched; only
clusters[].certificate-authority-data is replaced.
Save your progress? Sign up for free
Cleanup
The snapshots stay where they are – they are your way back for the next few days.
# on cp1
kubectl delete csr anna
kubectl delete namespace tag15
rm -f anna.key fremd-ca.key anna.kubeconfig fremd.kubeconfig
kubectl get csrExpected:
certificatesigningrequest.certificates.k8s.io "anna" deleted
namespace "tag15" deleted
No resources found
Interview Note
- The API server does not respond – what now? On the control
plane nodes:
crictl ps -a --name kube-apiserver,crictl logs,/var/log/pods, the service’s journal; the cause is almost always in the last line of the container log. - What do you do before an intervention on the control plane?
Check etcd (
endpoint health,alarm list) and take a snapshot; know the restore path before you need it. Unauthorizedvs.x509?Unauthorized: the server rejects your client certificate/token.x509on the client: you do not trust the server (CA, expiry, clock).
Enterprise Note
- Three or five etcd members, plus snapshots to S3
(
etcd-s3in RKE2); a snapshot on the same node does not help when the disk dies. - Practice the restore regularly – on a copy, not in production.
- Monitor certificate expiry
(
apiserver_client_certificate_expiration_seconds,rke2 certificate checkvia cron); people get short-lived access via OIDC instead of client certificates. - Test changes to
config.yamlon one server first, then roll them out one after the other – never on all control plane nodes at once. - On Talos, the same questions are answered via
talosctl etcd,talosctl logsandkubernetesdynamiccerts(Days 18 and 19).