Skip to content
Hands-on IT IT Drills

Day 2: Understanding and Managing k0s

Goal: You know which processes run where, where k0s keeps its files, why kubectl logs can hang, and you change the configuration only via k0sctl apply. · Duration: approx. 45 min · Prerequisite: Day 1

Briefing

  • One systemd service per node: k0scontroller or k0sworker. All components are child processes, all logs are in its journal.
  • Everything lives under /var/lib/k0s – including the kubelet. For CSI drivers, this is the most important difference from other distributions.
  • konnectivity: Because the controller is not a node, the workers build a tunnel to it. logs, exec, port-forward and metrics run through it.
  • Changes to the cluster belong in k0sctl.yaml. Whatever you change by hand, k0s overwrites on the next restart.
 kubectl get ──────────> kube-apiserver ──> etcd           (always works)
 kubectl logs/exec ────> kube-apiserver ──> konnectivity-server :8132
                                              ^ tunnel from every worker
                                              └─ konnectivity-agent ──> kubelet

Drill 1: One service per node

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag02 && cd ~/k0s-lab/tag02
systemctl list-units 'k0s*' --no-legend
ps -eo comm --no-headers | sort -u | grep -E 'kube|etcd|konn'
ssh k0s-w1 "systemctl list-units 'k0s*' --no-legend"
ssh k0s-w1 "ps -eo comm --no-headers | sort -u | grep -E 'kubelet|containerd$'"
ssh k0s-w1 'systemctl cat k0sworker | grep ExecStart'

Expected (abridged):

  k0scontroller.service loaded active running k0s - Zero Friction Kubernetes
etcd
konnectivity-se
kube-apiserver
kube-controller
kube-scheduler
  k0sworker.service loaded active running k0s - Zero Friction Kubernetes
containerd
kubelet
ExecStart=/usr/local/bin/k0s worker --data-dir=/var/lib/k0s --labels=topology.kuber...
Node Service underneath
Controller k0scontroller etcd, kube-apiserver, scheduler, controller manager, konnectivity-server
Worker k0sworker containerd, kubelet; as Pods kube-proxy, kube-router, konnectivity-agent

If a node causes trouble, ask systemd first, not kubectl. A worker with a stopped service is merely NotReady in Kubernetes – without a reason.

Save your progress? Sign up for free

Drill 2: Where k0s keeps its files

# on k0s-ctrl
sudo ls /var/lib/k0s
sudo grep -A3 '^  api:' /etc/k0s/k0s.yaml
ssh k0s-w1 'sudo ls /var/lib/k0s; sudo ls /var/lib/k0s/kubelet | head -4'
ssh k0s-w1 'ls /var/lib/kubelet'

Expected (abridged):

bin
etcd
konnectivity.conf
manifests
pki
  api:
    address: 172.16.0.176
    sans:
    - 172.16.0.176
bin
containerd
images
kubelet
kubelet.conf
worker-profile.yaml
actuated_pods_state
allocated_pods_state
checkpoints
cpu_manager_state
device-plugins
Path Content
/etc/k0s/k0s.yaml cluster configuration on the controller, written by k0sctl
/var/lib/k0s/pki, /var/lib/k0s/etcd certificates and etcd data – the cluster state
/var/lib/k0s/kubelet the kubelet’s working directory (not /var/lib/kubelet)
/var/lib/k0s/containerd images and container file systems

/var/lib/kubelet exists anyway, but only with device-plugins. A CSI driver that puts its socket there therefore finds a directory and reports no error – but the kubelet looks elsewhere. On Day 7, you walk right into this trap.

Save your progress? Sign up for free

Drill 3: Tools in the binary

k0s ships with kubectl, ctr and a system check. You do not need to install anything extra on the nodes.

# on k0s-ctrl
sudo k0s kubectl get nodes
ssh k0s-w1 'sudo k0s ctr -n k8s.io images ls -q | head -3'
ssh k0s-w1 'sudo k0s sysinfo | head -4'
ssh k0s-w1 'sudo k0s sysinfo | grep -v "(pass)"'

Expected:

NAME     STATUS   ROLES    AGE   VERSION
k0s-w1   Ready    <none>   13m   v1.36.6+k0s
k0s-w2   Ready    <none>   13m   v1.36.6+k0s
k0s-w3   Ready    <none>   13m   v1.36.6+k0s
quay.io/k0sproject/apiserver-network-proxy-agent:v0.36.0-k0s.2
quay.io/k0sproject/apiserver-network-proxy-agent@sha256:f39f4e902eeef002c4e9f4d8c34...
quay.io/k0sproject/cni-node:v1.9.1-k0s.2
Total memory: 3.8 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 24.4 GiB (pass)
Relative disk space available for /var/lib/k0s: 86% (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT)...
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT)...
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)

k0s sysinfo is the first command on a node that refuses to start. The four warnings are harmless: these kernel options no longer exist under that name since Linux 4.19 and 5.1 respectively.

Save your progress? Sign up for free

Drill 4: One journal for everything

# on k0s-ctrl
sudo journalctl -u k0scontroller --since -10min -o cat \
  | grep -o 'component=[a-z-]*' | sort | uniq -c | sort -rn | head -8

Expected (numbers vary):

   2243 component=grpc
    631 component=kube-apiserver
    428 component=kube-controller-manager
    244 component=konnectivity
    150 component=controller-runtime
     91 component=kube-scheduler
     77 component=etcd
     55 component=cfssl

Every line carries component=. With it, you filter the journal for API server, etcd or konnectivity without knowing which process currently has which PID.

Save your progress? Sign up for free

Drill 5: When logs hangs

On each worker, a konnectivity-agent keeps a connection to port 8132 on the controller. You cut it with a firewall rule and see what still works.

# on k0s-ctrl
sudo ss -tn state established '( sport = :8132 )' | tail -n +2 | wc -l
sudo iptables -I INPUT -p tcp --dport 8132 -j DROP
kubectl get nodes --no-headers | wc -l
kubectl -n kube-system logs ds/kube-router -c kube-router --tail=1 --request-timeout=20s

Expected:

3
3
Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded...

get responds immediately, logs runs into the timeout after 20 seconds. The message “Unable to connect to the server” is misleading: the API server is reachable, only the path from it to the kubelet is missing. The journal says it more clearly:

# on k0s-ctrl
sleep 30
sudo journalctl -u k0scontroller --since -1min -o cat | grep component=konnectivity \
  | grep -o 'No agent available' | uniq -c
kubectl -n kube-system get pods -l k8s-app=konnectivity-agent

Expected:

     22 No agent available
NAME                       READY   STATUS    RESTARTS   AGE
konnectivity-agent-4249z   1/1     Running   0          14m
konnectivity-agent-9rhdh   0/1     Running   0          14m
konnectivity-agent-hnlbn   0/1     Running   0          14m

Remove the rule and wait:

# on k0s-ctrl
sudo iptables -D INPUT -p tcp --dport 8132 -j DROP
sleep 15
kubectl -n kube-system get pods -l k8s-app=konnectivity-agent
kubectl -n kube-system logs ds/kube-router -c kube-router --tail=1 | cut -c1-60

Expected (abridged):

NAME                       READY   STATUS    RESTARTS   AGE
konnectivity-agent-4249z   1/1     Running   0          15m
konnectivity-agent-9rhdh   1/1     Running   0          15m
konnectivity-agent-hnlbn   1/1     Running   0          15m
I1119 23:52:31.283921    3076 bgp_policies.go:991] Ensuring t...

After about ten seconds, the tunnels are back. You see the same picture after every controller restart, until all agents have reconnected.

Save your progress? Sign up for free

Drill 6: Manual changes do not last

# on k0s-ctrl
kubectl label node k0s-w3 topology.kubernetes.io/zone=b --overwrite
kubectl get node k0s-w3 -L topology.kubernetes.io/zone
ssh k0s-w3 sudo systemctl restart k0sworker
sleep 30
kubectl get node k0s-w3 -L topology.kubernetes.io/zone

Expected:

node/k0s-w3 labeled
NAME     STATUS   ROLES    AGE   VERSION       ZONE
k0s-w3   Ready    <none>   15m   v1.36.6+k0s   b
NAME     STATUS   ROLES    AGE   VERSION       ZONE
k0s-w3   Ready    <none>   16m   v1.36.6+k0s   a

On startup, the kubelet resets the labels from --labels. Your kubectl label only lasted until the next restart. For a zone, that is dangerous: between two restarts, placement and topology would believe something different from LINSTOR.

Save your progress? Sign up for free

Drill 7: Change the configuration via k0sctl

For storage nodes, graceful node shutdown pays off: on shutdown, the kubelet first terminates the Pods so that DRBD can release cleanly. This setting belongs in the kubelet configuration. k0s distributes it as a worker profile.

# on k0s-ctrl
cp ../tag01/k0sctl.yaml .
sed -i 's|zone=\([ab]\)\]|zone=\1, --profile=storage]|' k0sctl.yaml
cat >> k0sctl.yaml <<'EOF'
        workerProfiles:
        - name: storage
          values:
            shutdownGracePeriod: 60s
            shutdownGracePeriodCriticalPods: 20s
EOF
diff ../tag01/k0sctl.yaml k0sctl.yaml

Expected:

11c11
<     installFlags: [--labels=topology.kubernetes.io/zone=a]
---
>     installFlags: [--labels=topology.kubernetes.io/zone=a, --profile=storage]
14c14
...
34a35,39
>         workerProfiles:
>         - name: storage
>           values:
>             shutdownGracePeriod: 60s
>             shutdownGracePeriodCriticalPods: 20s

First, see what would happen:

# on k0s-ctrl
k0sctl apply --config k0sctl.yaml --dry-run 2>&1 | grep dry-run: | cut -c1-80

Expected (abridged):

dry-run:  - restart the k0s service
dry-run: * 172.16.0.186:22 :
dry-run:  - reinstall k0s using `k0s install worker --labels=topology.kubernetes...
dry-run:  - restart k0s service
...

Restart the controller, reinstall all workers. On a large cluster, you do not want that on all workers at the same time, so one after the other:

# on k0s-ctrl
k0sctl apply --config k0sctl.yaml --concurrency 1 2>&1 \
  | grep -E 'Reinstall|reinstalling'
kubectl get --raw /api/v1/nodes/k0s-w1/proxy/configz | python3 -m json.tool \
  | grep shutdownGrace
ssh k0s-w1 'systemd-inhibit --list --no-pager | grep -c kubelet'

Expected:

level=info msg="==> Running phase: Reinstall"
level=info msg="Reinstalling max 1 workers in parallel"
level=info msg="172.16.0.186:22: reinstalling k0s"
level=info msg="172.16.0.36:22: reinstalling k0s"
level=info msg="172.16.0.72:22: reinstalling k0s"
        "shutdownGracePeriod": "1m0s",
        "shutdownGracePeriodCriticalPods": "20s",
0

The configuration has arrived, but the kubelet has registered no shutdown inhibitor lock with systemd-logind. Without it, nobody waits for the Pods on shutdown.

# on k0s-ctrl
ssh k0s-w1 'sudo journalctl -u k0sworker -o cat' \
  | grep -o 'Failed to start node shut.*' | cut -c1-88
ssh k0s-w1 'systemd-analyze cat-config systemd/logind.conf \
  | grep -E "^# /.*conf$|^Inhibit"'

Expected (abridged):

Failed to start node shutdown manager\" err=\"node shutdown manager was timed out af...
# /etc/systemd/logind.conf
# /etc/systemd/logind.conf.d/99-kubelet.conf
InhibitDelayMaxSec=60
# /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf
InhibitDelayMaxSec=30

The kubelet wrote its 60 seconds into 99-kubelet.conf. But systemd reads the drop-ins sorted by file name across all directories, and unattended-upgrades-… comes after 99-…. The 30 seconds from the unattended-upgrades package win, and the kubelet gives up. A file of your own that sorts last solves this:

# on k0s-w1
sudo tee /etc/systemd/logind.conf.d/zz-kubelet.conf >/dev/null <<'EOF'
[Login]
InhibitDelayMaxSec=60
EOF
sudo systemctl restart systemd-logind
sudo systemctl restart k0sworker
sleep 25
systemd-inhibit --list --no-pager | grep kubelet

Expected (abridged):

kubelet                      0   root 6348 kubelet         shutdown Kubelet needs ti...

Repeat this on k0s-w2 and k0s-w3. From now on, ~/k0s-lab/tag02/k0sctl.yaml is the valid cluster description. The logind file does not belong in k0sctl but in host preparation – on Day 16, Ansible takes it over.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show since when k0scontroller has been running. Target time: 20 s
  2. Show the last five journal lines of k0sworker on k0s-w2. Target time: 20 s
  3. Count the running containers on k0s-w1 using containerd tools. Target time: 30 s
  4. Show the shutdown settings from the running kubelet configuration of k0s-w2. Target time: 40 s

Solution

# on k0s-ctrl
systemctl show -p ActiveEnterTimestamp k0scontroller
ssh k0s-w2 'sudo journalctl -u k0sworker -n 5 --no-pager -o cat'
ssh k0s-w1 'sudo k0s ctr -n k8s.io tasks ls | grep -c RUNNING'
kubectl get --raw /api/v1/nodes/k0s-w2/proxy/configz | python3 -m json.tool \
  | grep -i shutdownGrace

Expected (abridged):

ActiveEnterTimestamp=Fri 2026-11-20 00:01:15 UTC
time="2026-11-20 00:06:02" level=info msg="I1120 00:06:02.951511    6402 prober.go:1...
...
8
        "shutdownGracePeriod": "1m0s",
        "shutdownGracePeriodCriticalPods": "20s",

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zone

Expected: three nodes Ready, zones a, b, a.

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  ssh $h 'systemd-inhibit --list | grep -c kubelet'
done

Expected: 1 three times.

# on k0s-ctrl
sudo iptables -S INPUT

Expected: only -P INPUT ACCEPT – the rule from Drill 5 is gone.

# on k0s-ctrl
kubectl -n kube-system get pods -l k8s-app=konnectivity-agent --no-headers | grep -c 1/1

Expected: 3

Challenge

Stop the service on k0s-w3 and measure how long Kubernetes takes until the node is NotReady. Find out whether the containers on k0s-w3 keep running meanwhile. Start the service again afterwards.

Hint: When stopping, the journal of k0sworker tells you what is left behind.

Solution

# on k0s-ctrl
ssh k0s-w3 sudo systemctl stop k0sworker
s=$(date +%s)
until kubectl get node k0s-w3 --no-headers | grep -q NotReady; do sleep 2; done
echo "NotReady nach $(( $(date +%s) - s ))s"
ssh k0s-w3 'sudo journalctl -u k0sworker -n 3 --no-pager -o cat'
ssh k0s-w3 sudo systemctl start k0sworker

Expected:

NotReady nach 50s
k0sworker.service: Unit process 3508 (containerd-shim) remains running after unit st...
Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
k0sworker.service: Consumed 8.099s CPU time, 739.7M memory peak, 0B memory swap peak.

After 50 seconds (node-monitor-grace-period), the controller marks the node as NotReady. The containers keep running: the containerd-shim processes survive the service stop. For storage, this means: a stopped worker keeps its volumes open and may still be writing. NotReady is no proof that a node is dead – this becomes important on Day 10. After the start, k0s-w3 is Ready again within a few seconds.

Save your progress? Sign up for free

Interview Note

  • kubectl get works, kubectl logs hangs – what is the cause? The path from the API server to the kubelet. With k0s, that is konnectivity: check the agents, port 8132 on the controller, the journal with component=konnectivity.
  • Where is the kubelet directory in k0s? /var/lib/k0s/kubelet. CSI drivers and backup agents must be configured for it.
  • How do you change the kubelet configuration? Via worker profiles in k0sctl.yaml and k0sctl apply, with --concurrency for many nodes.

Enterprise Note

  • Graceful node shutdown on all nodes with databases. Check the inhibitor lock with systemd-inhibit --list – otherwise a silent misconfiguration like this one only shows up in the next maintenance window.
  • Collect the journal centrally (Loki, Elastic), since one service bundles all components.
  • Firewalls between workers and controllers: 6443 (API), 8132 (konnectivity), 9443 (join). If 8132 is missing, logs, exec and metrics fail first.
  • Changes to k0sctl.yaml via pull request, with the --dry-run output included in the review.