Skip to content
Hands-on IT IT Drills

Day 14: Troubleshooting I – Broken Nodes and Components

Goal: You find and fix faults in the kubelet, container runtime, control plane, and kubeconfig – even when kubectl no longer responds. · Duration: approx. 90 min · Prerequisite: Day 13

Briefing

CKA: Troubleshooting (30 %) – “Troubleshoot clusters and nodes”, “Troubleshoot cluster components”. Today you break the cluster on purpose – and fix it right away.

  Symptom                    first look                  then
  Node NotReady          --> describe node          --> ssh, systemctl, journalctl
  Pods stay Pending      --> scheduler pod          --> kubectl logs / crictl logs
  Deployment w/o pods    --> controller manager     --> check manifest
  connection refused     --> kubeconfig, API server --> crictl ps -a, crictl logs

Rule for every drill: back up first, then break things. Every fault is fixed in the same drill.

k create ns tag14

Drill 1: The kubelet Is Down

On worker2:

sudo systemctl stop kubelet

On cp1, after about 50 seconds:

k get nodes
k describe node worker2 | grep -A6 '^Conditions'

Expected (abbreviated):

NAME      STATUS     ROLES           AGE   VERSION
cp1       Ready      control-plane   92m   v1.37.2
worker1   Ready      <none>          91m   v1.37.2
worker2   NotReady   <none>          91m   v1.37.2
Conditions:
  Type                 Status    ...   Reason              Message
  NetworkUnavailable   False     ...   CalicoIsUp          Calico is running on ...
  MemoryPressure       Unknown   ...   NodeStatusUnknown   Kubelet stopped posting ...
  ...
  Ready                Unknown   ...   NodeStatusUnknown   Kubelet stopped posting ...

Unknown means: the kubelet is no longer reporting in. On worker2:

systemctl status kubelet --no-pager | grep Active
sudo systemctl start kubelet
systemctl is-enabled kubelet

Expected:

     Active: inactive (dead) since Tue 2026-11-17 22:44:13 UTC; 54s ago
enabled

Always check is-enabled as well – a disabled service is gone again after the next reboot.

Save your progress? Sign up for free

Drill 2: The kubelet Won’t Start

On worker1, back up and introduce a typo:

sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.bak
sudo sed -i 's#/etc/kubernetes/pki/ca.crt#/etc/kubernetes/pki/ca.cert#' \
  /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet

Expected: activating – the kubelet starts, dies, and starts again. The reason is in the journal:

sudo journalctl -u kubelet --no-pager -n 40 | grep -m2 -E 'error|failed'

Expected (abbreviated):

... run.go:72] "command failed" err="failed to run Kubelet: open
/etc/kubernetes/pki/ca.cert:
no such file or directory"
... systemd[1]: kubelet.service: Failed with result 'exit-code'.

Fix it:

sudo sed -i 's#/etc/kubernetes/pki/ca.cert#/etc/kubernetes/pki/ca.crt#' \
  /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet

Expected: active

The kubelet’s configuration lives in three places: /var/lib/kubelet/config.yaml, /var/lib/kubelet/kubeadm-flags.env, and the systemd drop-ins (systemctl cat kubelet).

Save your progress? Sign up for free

Drill 3: The Container Runtime Is Gone

Still on worker1:

sudo systemctl stop containerd

On cp1, after about 45 seconds:

k get node worker1
k describe node worker1 | grep -E '^\s+Ready'

Expected (abbreviated):

NAME      STATUS     ROLES    AGE   VERSION
worker1   NotReady   <none>   93m   v1.37.2
  Ready   False   ...   KubeletNotReady   container runtime is down

This time the kubelet is running and reporting in – but it has no runtime. On worker1:

sudo crictl ps

Expected (abbreviated):

... level=fatal msg="validate service connection: validate CRI v1 runtime API for
endpoint
\"unix:///run/containerd/containerd.sock\": ... connect: no such file or directory\""
sudo systemctl start containerd

After about 20 seconds, worker1 is Ready again. By the way, running containers survive a containerd restart – the shims keep running.

Save your progress? Sign up for free

Drill 4: Pods Stay Pending – the Scheduler

On cp1:

sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.bak
sudo sed -i 's#--kubeconfig=/etc/kubernetes/scheduler.conf#&f#' \
  /etc/kubernetes/manifests/kube-scheduler.yaml

After about 40 seconds, start a pod:

k run probe -n tag14 --image=nginx:1.29
k get pod probe -n tag14
k describe pod probe -n tag14 | tail -1

Expected:

NAME    READY   STATUS    RESTARTS   AGE
probe   0/1     Pending   0          10s
Events:                      <none>

Pending without events means: nobody even tried to place the pod. With a capacity problem you would see FailedScheduling there (Day 5).

k get pods -n kube-system -l component=kube-scheduler
k logs -n kube-system kube-scheduler-cp1 | tail -1

Expected:

NAME                 READY   STATUS             RESTARTS      AGE
kube-scheduler-cp1   0/1     CrashLoopBackOff   2 (25s ago)   41s
E1117 22:47:34.120554  1 run.go:72] "command failed" err="stat
/etc/kubernetes/scheduler.conff:
no such file or directory"

Fix it:

sudo sed -i 's#scheduler.conff#scheduler.conf#' \
  /etc/kubernetes/manifests/kube-scheduler.yaml

After about 30 seconds the scheduler is running, and probe is placed on a worker.

Save your progress? Sign up for free

Drill 5: Deployment Without Pods – the Controller Manager

sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /root/
k create deployment ghost -n tag14 --image=nginx:1.29 --replicas=2
k get deploy,rs,pods -n tag14 -l app=ghost
k get pods -n kube-system -l component=kube-controller-manager

Expected:

deployment.apps/ghost created
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ghost   0/2     0            0           11s
No resources found in kube-system namespace.

The Deployment exists, but there is no ReplicaSet: the Deployment controller that would create it lives in the controller manager. Fix it:

sudo mv /root/kube-controller-manager.yaml /etc/kubernetes/manifests/

After about 40 seconds:

k get deploy ghost -n tag14

Expected: READY shows 2/2.

Save your progress? Sign up for free

Drill 6: The API Server Is Dead

Now the hardest fault – without an API server, kubectl no longer helps:

sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo sed -i 's#127.0.0.1:2379#127.0.0.1:2399#' \
  /etc/kubernetes/manifests/kube-apiserver.yaml

After about a minute:

k get nodes

Expected (abbreviated):

... dial tcp 172.16.0.126:6443: connect: connection refused ...

One level deeper – ask the runtime:

sudo crictl ps -a --name kube-apiserver

Expected:

CONTAINER       IMAGE           CREATED                  STATE     NAME             ...
a1dbdea6509df   7a918dfe48696   Less than a second ago   Created   kube-apiserver   ...
7686936e813f5   7a918dfe48696   22 seconds ago           Exited    kube-apiserver   ...

-a also shows exited containers – their logs reveal the reason:

ID=$(sudo crictl ps -a --name kube-apiserver --state exited -q | head -1)
sudo crictl logs $ID 2>&1 | tail -1

Expected:

E1117 22:49:41.420449  1 run.go:72] "command failed" err="error creating storage
factory:
etcd grpc connection not ready: context deadline exceeded"

Further up there are lines with 127.0.0.1:2399 – the wrong port. The same logs are also available as files under /var/log/pods/kube-system_kube-apiserver-cp1_*/. Fix it:

sudo sed -i 's#127.0.0.1:2399#127.0.0.1:2379#' \
  /etc/kubernetes/manifests/kube-apiserver.yaml

After one to two minutes, k get nodes responds again.

Save your progress? Sign up for free

Drill 7: kubectl Talks Into the Void

Not every connection refused is a dead API server:

cp ~/.kube/config /tmp/kubeconfig.bak
sed -i 's#:6443#:6444#' ~/.kube/config
k get nodes
k config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Expected:

The connection to the server 172.16.0.126:6444 was refused - did you specify the right
host
or port?
https://172.16.0.126:6444

The port is in the error message – compare it with the manifest (--secure-port=6443). Fix it:

sed -i 's#:6444#:6443#' ~/.kube/config
k get --raw='/readyz?verbose' | tail -2

Expected:

[+]shutdown ok
readyz check passed

/readyz?verbose checks all parts of the API server including etcd – the fastest health check.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the last 20 lines of the kubelet journal on worker1. Target time: 20 s
  2. List all control plane pods with their restarts. Target time: 20 s
  3. Check whether etcd is healthy from the API server’s point of view. Target time: 15 s

Solution

On worker1:

sudo journalctl -u kubelet --no-pager -n 20

On cp1:

k get pods -n kube-system -l tier=control-plane
k get --raw='/livez/etcd'; echo

Expected (excerpt):

NAME                          READY   STATUS    RESTARTS      AGE
etcd-cp1                      1/1     Running   1 (17m ago)   16m
kube-apiserver-cp1            1/1     Running   0             16m
kube-controller-manager-cp1   1/1     Running   2 (82s ago)   31s
kube-scheduler-cp1            1/1     Running   1 (3m ago)    3m30s
ok

Save your progress? Sign up for free

Checkpoint

k get nodes

Expected: all three nodes Ready.

k get pods -n kube-system -l tier=control-plane --no-headers | grep -vc Running

Expected: 0

k get pod probe -n tag14 -o jsonpath='{.status.phase}{"\n"}'

Expected: Running

Challenge

A colleague reports: “No new pods start on worker2, the old ones keep running.” Reproduce the fault by changing the socket in /var/lib/kubelet/config.yaml on worker2 to unix:///var/run/containerd/containerd.sockx and restarting the kubelet. Find the fault using only node tools and fix it.

Hint: journalctl -u kubelet and the key containerRuntimeEndpoint.

Solution

On worker2:

sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.bak
sudo sed -i 's#containerd.sock$#containerd.sockx#' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
sudo journalctl -u kubelet --no-pager -n 20 | grep -m1 -i 'sockx'
sudo grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
sudo sed -i 's#containerd.sockx#containerd.sock#' /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
systemctl is-active kubelet

Expected (abbreviated):

... grpc: addrConn.createTransport failed to connect to
{Addr: "/var/run/containerd/containerd.sockx", ServerName: "localhost", }. ...
containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sockx
active

Clean up:

k delete ns tag14

Save your progress? Sign up for free

Exam Tip

  • Node NotReady: ssh to the node, then systemctl status kubelet, journalctl -u kubelet, systemctl status containerd – in that order.
  • Control plane broken: check /etc/kubernetes/manifests first, then crictl ps -a and crictl logs. A typo in the manifest is the most common cause.
  • After fixing a manifest, wait one to two minutes – the kubelet needs time to recreate the pod.

Enterprise Note

  • In production, you detect node problems through monitoring (node-exporter, kube-state-metrics, alerts on NotReady), not through user complaints.
  • The node-problem-detector reports kernel and runtime problems as node conditions.
  • Changes to control plane manifests belong in the kubeadm configuration; otherwise the next upgrade overwrites them (Day 13).