Day 14: Troubleshooting I – Broken Nodes and Components
Goal: You find and fix faults in the kubelet, container runtime, control plane, and kubeconfig – even when
kubectlno longer responds. · Duration: approx. 90 min · Prerequisite: Day 13
Briefing
CKA: Troubleshooting (30 %) – “Troubleshoot clusters and nodes”, “Troubleshoot cluster components”. Today you break the cluster on purpose – and fix it right away.
Symptom first look then
Node NotReady --> describe node --> ssh, systemctl, journalctl
Pods stay Pending --> scheduler pod --> kubectl logs / crictl logs
Deployment w/o pods --> controller manager --> check manifest
connection refused --> kubeconfig, API server --> crictl ps -a, crictl logs
Rule for every drill: back up first, then break things. Every fault is fixed in the same drill.
k create ns tag14Drill 1: The kubelet Is Down
On worker2:
sudo systemctl stop kubeletOn cp1, after about 50 seconds:
k get nodes
k describe node worker2 | grep -A6 '^Conditions'Expected (abbreviated):
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 92m v1.37.2
worker1 Ready <none> 91m v1.37.2
worker2 NotReady <none> 91m v1.37.2
Conditions:
Type Status ... Reason Message
NetworkUnavailable False ... CalicoIsUp Calico is running on ...
MemoryPressure Unknown ... NodeStatusUnknown Kubelet stopped posting ...
...
Ready Unknown ... NodeStatusUnknown Kubelet stopped posting ...
Unknown means: the kubelet is no longer reporting in. On
worker2:
systemctl status kubelet --no-pager | grep Active
sudo systemctl start kubelet
systemctl is-enabled kubeletExpected:
Active: inactive (dead) since Tue 2026-11-17 22:44:13 UTC; 54s ago
enabled
Always check is-enabled as well – a disabled service is
gone again after the next reboot.
Save your progress? Sign up for free
Drill 2: The kubelet Won’t Start
On worker1, back up and introduce a typo:
sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.bak
sudo sed -i 's#/etc/kubernetes/pki/ca.crt#/etc/kubernetes/pki/ca.cert#' \
/var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubeletExpected: activating – the kubelet
starts, dies, and starts again. The reason is in the journal:
sudo journalctl -u kubelet --no-pager -n 40 | grep -m2 -E 'error|failed'Expected (abbreviated):
... run.go:72] "command failed" err="failed to run Kubelet: open
/etc/kubernetes/pki/ca.cert:
no such file or directory"
... systemd[1]: kubelet.service: Failed with result 'exit-code'.
Fix it:
sudo sed -i 's#/etc/kubernetes/pki/ca.cert#/etc/kubernetes/pki/ca.crt#' \
/var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubeletExpected: active
The kubelet’s configuration lives in three places:
/var/lib/kubelet/config.yaml,
/var/lib/kubelet/kubeadm-flags.env, and the systemd
drop-ins (systemctl cat kubelet).
Save your progress? Sign up for free
Drill 3: The Container Runtime Is Gone
Still on worker1:
sudo systemctl stop containerdOn cp1, after about 45 seconds:
k get node worker1
k describe node worker1 | grep -E '^\s+Ready'Expected (abbreviated):
NAME STATUS ROLES AGE VERSION
worker1 NotReady <none> 93m v1.37.2
Ready False ... KubeletNotReady container runtime is down
This time the kubelet is running and reporting in – but it has no runtime. On worker1:
sudo crictl psExpected (abbreviated):
... level=fatal msg="validate service connection: validate CRI v1 runtime API for
endpoint
\"unix:///run/containerd/containerd.sock\": ... connect: no such file or directory\""
sudo systemctl start containerdAfter about 20 seconds, worker1 is Ready
again. By the way, running containers survive a containerd restart – the
shims keep running.
Save your progress? Sign up for free
Drill 4: Pods Stay Pending – the Scheduler
On cp1:
sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.bak
sudo sed -i 's#--kubeconfig=/etc/kubernetes/scheduler.conf#&f#' \
/etc/kubernetes/manifests/kube-scheduler.yamlAfter about 40 seconds, start a pod:
k run probe -n tag14 --image=nginx:1.29
k get pod probe -n tag14
k describe pod probe -n tag14 | tail -1Expected:
NAME READY STATUS RESTARTS AGE
probe 0/1 Pending 0 10s
Events: <none>
Pending without events means: nobody
even tried to place the pod. With a capacity problem you would see
FailedScheduling there (Day 5).
k get pods -n kube-system -l component=kube-scheduler
k logs -n kube-system kube-scheduler-cp1 | tail -1Expected:
NAME READY STATUS RESTARTS AGE
kube-scheduler-cp1 0/1 CrashLoopBackOff 2 (25s ago) 41s
E1117 22:47:34.120554 1 run.go:72] "command failed" err="stat
/etc/kubernetes/scheduler.conff:
no such file or directory"
Fix it:
sudo sed -i 's#scheduler.conff#scheduler.conf#' \
/etc/kubernetes/manifests/kube-scheduler.yamlAfter about 30 seconds the scheduler is running, and
probe is placed on a worker.
Save your progress? Sign up for free
Drill 5: Deployment Without Pods – the Controller Manager
sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /root/
k create deployment ghost -n tag14 --image=nginx:1.29 --replicas=2
k get deploy,rs,pods -n tag14 -l app=ghost
k get pods -n kube-system -l component=kube-controller-managerExpected:
deployment.apps/ghost created
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ghost 0/2 0 0 11s
No resources found in kube-system namespace.
The Deployment exists, but there is no ReplicaSet: the Deployment controller that would create it lives in the controller manager. Fix it:
sudo mv /root/kube-controller-manager.yaml /etc/kubernetes/manifests/After about 40 seconds:
k get deploy ghost -n tag14Expected: READY shows
2/2.
Save your progress? Sign up for free
Drill 6: The API Server Is Dead
Now the hardest fault – without an API server, kubectl
no longer helps:
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo sed -i 's#127.0.0.1:2379#127.0.0.1:2399#' \
/etc/kubernetes/manifests/kube-apiserver.yamlAfter about a minute:
k get nodesExpected (abbreviated):
... dial tcp 172.16.0.126:6443: connect: connection refused ...
One level deeper – ask the runtime:
sudo crictl ps -a --name kube-apiserverExpected:
CONTAINER IMAGE CREATED STATE NAME ...
a1dbdea6509df 7a918dfe48696 Less than a second ago Created kube-apiserver ...
7686936e813f5 7a918dfe48696 22 seconds ago Exited kube-apiserver ...
-a also shows exited containers – their logs reveal the
reason:
ID=$(sudo crictl ps -a --name kube-apiserver --state exited -q | head -1)
sudo crictl logs $ID 2>&1 | tail -1Expected:
E1117 22:49:41.420449 1 run.go:72] "command failed" err="error creating storage
factory:
etcd grpc connection not ready: context deadline exceeded"
Further up there are lines with 127.0.0.1:2399 – the
wrong port. The same logs are also available as files under
/var/log/pods/kube-system_kube-apiserver-cp1_*/. Fix
it:
sudo sed -i 's#127.0.0.1:2399#127.0.0.1:2379#' \
/etc/kubernetes/manifests/kube-apiserver.yamlAfter one to two minutes, k get nodes responds
again.
Save your progress? Sign up for free
Drill 7: kubectl Talks Into the Void
Not every connection refused is a dead API server:
cp ~/.kube/config /tmp/kubeconfig.bak
sed -i 's#:6443#:6444#' ~/.kube/config
k get nodes
k config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'Expected:
The connection to the server 172.16.0.126:6444 was refused - did you specify the right
host
or port?
https://172.16.0.126:6444
The port is in the error message – compare it with the manifest
(--secure-port=6443). Fix it:
sed -i 's#:6444#:6443#' ~/.kube/config
k get --raw='/readyz?verbose' | tail -2Expected:
[+]shutdown ok
readyz check passed
/readyz?verbose checks all parts of the API server
including etcd – the fastest health check.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the last 20 lines of the kubelet journal on
worker1. Target time: 20 s - List all control plane pods with their restarts. Target time: 20 s
- Check whether etcd is healthy from the API server’s point of view. Target time: 15 s
Solution
On worker1:
sudo journalctl -u kubelet --no-pager -n 20On cp1:
k get pods -n kube-system -l tier=control-plane
k get --raw='/livez/etcd'; echoExpected (excerpt):
NAME READY STATUS RESTARTS AGE
etcd-cp1 1/1 Running 1 (17m ago) 16m
kube-apiserver-cp1 1/1 Running 0 16m
kube-controller-manager-cp1 1/1 Running 2 (82s ago) 31s
kube-scheduler-cp1 1/1 Running 1 (3m ago) 3m30s
ok
Save your progress? Sign up for free
Checkpoint
k get nodesExpected: all three nodes Ready.
k get pods -n kube-system -l tier=control-plane --no-headers | grep -vc RunningExpected: 0
k get pod probe -n tag14 -o jsonpath='{.status.phase}{"\n"}'Expected: Running
Challenge
A colleague reports: “No new pods start on worker2, the
old ones keep running.” Reproduce the fault by changing the socket in
/var/lib/kubelet/config.yaml on worker2 to
unix:///var/run/containerd/containerd.sockx and restarting
the kubelet. Find the fault using only node tools and fix it.
Hint: journalctl -u kubelet and the key
containerRuntimeEndpoint.
Solution
On worker2:
sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.bak
sudo sed -i 's#containerd.sock$#containerd.sockx#' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
sudo journalctl -u kubelet --no-pager -n 20 | grep -m1 -i 'sockx'
sudo grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
sudo sed -i 's#containerd.sockx#containerd.sock#' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubeletExpected (abbreviated):
... grpc: addrConn.createTransport failed to connect to
{Addr: "/var/run/containerd/containerd.sockx", ServerName: "localhost", }. ...
containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sockx
active
Clean up:
k delete ns tag14Save your progress? Sign up for free
Exam Tip
- Node
NotReady:sshto the node, thensystemctl status kubelet,journalctl -u kubelet,systemctl status containerd– in that order. - Control plane broken: check
/etc/kubernetes/manifestsfirst, thencrictl ps -aandcrictl logs. A typo in the manifest is the most common cause. - After fixing a manifest, wait one to two minutes – the kubelet needs time to recreate the pod.
Enterprise Note
- In production, you detect node problems through monitoring
(node-exporter, kube-state-metrics, alerts on
NotReady), not through user complaints. - The node-problem-detector reports kernel and runtime problems as node conditions.
- Changes to control plane manifests belong in the kubeadm configuration; otherwise the next upgrade overwrites them (Day 13).