Day 2: Understanding and Managing k0s
Goal: You know which processes run where, where k0s keeps its files, why
kubectl logscan hang, and you change the configuration only viak0sctl apply. · Duration: approx. 45 min · Prerequisite: Day 1
Briefing
- One systemd service per node:
k0scontrollerork0sworker. All components are child processes, all logs are in its journal. - Everything lives under
/var/lib/k0s– including the kubelet. For CSI drivers, this is the most important difference from other distributions. - konnectivity: Because the controller is not a node,
the workers build a tunnel to it.
logs,exec,port-forwardand metrics run through it. - Changes to the cluster belong in
k0sctl.yaml. Whatever you change by hand, k0s overwrites on the next restart.
kubectl get ──────────> kube-apiserver ──> etcd (always works)
kubectl logs/exec ────> kube-apiserver ──> konnectivity-server :8132
^ tunnel from every worker
└─ konnectivity-agent ──> kubelet
Drill 1: One service per node
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag02 && cd ~/k0s-lab/tag02
systemctl list-units 'k0s*' --no-legend
ps -eo comm --no-headers | sort -u | grep -E 'kube|etcd|konn'
ssh k0s-w1 "systemctl list-units 'k0s*' --no-legend"
ssh k0s-w1 "ps -eo comm --no-headers | sort -u | grep -E 'kubelet|containerd$'"
ssh k0s-w1 'systemctl cat k0sworker | grep ExecStart'Expected (abridged):
k0scontroller.service loaded active running k0s - Zero Friction Kubernetes
etcd
konnectivity-se
kube-apiserver
kube-controller
kube-scheduler
k0sworker.service loaded active running k0s - Zero Friction Kubernetes
containerd
kubelet
ExecStart=/usr/local/bin/k0s worker --data-dir=/var/lib/k0s --labels=topology.kuber...
| Node | Service | underneath |
|---|---|---|
| Controller | k0scontroller |
etcd, kube-apiserver, scheduler, controller manager, konnectivity-server |
| Worker | k0sworker |
containerd, kubelet; as Pods kube-proxy, kube-router, konnectivity-agent |
If a node causes trouble, ask systemd first, not kubectl. A worker
with a stopped service is merely NotReady in Kubernetes –
without a reason.
Save your progress? Sign up for free
Drill 2: Where k0s keeps its files
# on k0s-ctrl
sudo ls /var/lib/k0s
sudo grep -A3 '^ api:' /etc/k0s/k0s.yaml
ssh k0s-w1 'sudo ls /var/lib/k0s; sudo ls /var/lib/k0s/kubelet | head -4'
ssh k0s-w1 'ls /var/lib/kubelet'Expected (abridged):
bin
etcd
konnectivity.conf
manifests
pki
api:
address: 172.16.0.176
sans:
- 172.16.0.176
bin
containerd
images
kubelet
kubelet.conf
worker-profile.yaml
actuated_pods_state
allocated_pods_state
checkpoints
cpu_manager_state
device-plugins
| Path | Content |
|---|---|
/etc/k0s/k0s.yaml |
cluster configuration on the controller, written by k0sctl |
/var/lib/k0s/pki, /var/lib/k0s/etcd |
certificates and etcd data – the cluster state |
/var/lib/k0s/kubelet |
the kubelet’s working directory (not
/var/lib/kubelet) |
/var/lib/k0s/containerd |
images and container file systems |
/var/lib/kubelet exists anyway, but only with
device-plugins. A CSI driver that puts its socket there
therefore finds a directory and reports no error – but the kubelet looks
elsewhere. On Day 7, you walk right into this trap.
Save your progress? Sign up for free
Drill 3: Tools in the binary
k0s ships with kubectl, ctr and a system check. You do
not need to install anything extra on the nodes.
# on k0s-ctrl
sudo k0s kubectl get nodes
ssh k0s-w1 'sudo k0s ctr -n k8s.io images ls -q | head -3'
ssh k0s-w1 'sudo k0s sysinfo | head -4'
ssh k0s-w1 'sudo k0s sysinfo | grep -v "(pass)"'Expected:
NAME STATUS ROLES AGE VERSION
k0s-w1 Ready <none> 13m v1.36.6+k0s
k0s-w2 Ready <none> 13m v1.36.6+k0s
k0s-w3 Ready <none> 13m v1.36.6+k0s
quay.io/k0sproject/apiserver-network-proxy-agent:v0.36.0-k0s.2
quay.io/k0sproject/apiserver-network-proxy-agent@sha256:f39f4e902eeef002c4e9f4d8c34...
quay.io/k0sproject/cni-node:v1.9.1-k0s.2
Total memory: 3.8 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 24.4 GiB (pass)
Relative disk space available for /var/lib/k0s: 86% (pass)
CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT)...
CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT)...
CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
k0s sysinfo is the first command on a node that refuses
to start. The four warnings are harmless: these kernel options no longer
exist under that name since Linux 4.19 and 5.1 respectively.
Save your progress? Sign up for free
Drill 4: One journal for everything
# on k0s-ctrl
sudo journalctl -u k0scontroller --since -10min -o cat \
| grep -o 'component=[a-z-]*' | sort | uniq -c | sort -rn | head -8Expected (numbers vary):
2243 component=grpc
631 component=kube-apiserver
428 component=kube-controller-manager
244 component=konnectivity
150 component=controller-runtime
91 component=kube-scheduler
77 component=etcd
55 component=cfssl
Every line carries component=. With it, you filter the
journal for API server, etcd or konnectivity without knowing which
process currently has which PID.
Save your progress? Sign up for free
Drill 5: When logs hangs
On each worker, a konnectivity-agent keeps a connection
to port 8132 on the controller. You cut it with a firewall rule and see
what still works.
# on k0s-ctrl
sudo ss -tn state established '( sport = :8132 )' | tail -n +2 | wc -l
sudo iptables -I INPUT -p tcp --dport 8132 -j DROP
kubectl get nodes --no-headers | wc -l
kubectl -n kube-system logs ds/kube-router -c kube-router --tail=1 --request-timeout=20sExpected:
3
3
Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded...
get responds immediately, logs runs into
the timeout after 20 seconds. The message “Unable to connect to the
server” is misleading: the API server is reachable, only the path from
it to the kubelet is missing. The journal says it more clearly:
# on k0s-ctrl
sleep 30
sudo journalctl -u k0scontroller --since -1min -o cat | grep component=konnectivity \
| grep -o 'No agent available' | uniq -c
kubectl -n kube-system get pods -l k8s-app=konnectivity-agentExpected:
22 No agent available
NAME READY STATUS RESTARTS AGE
konnectivity-agent-4249z 1/1 Running 0 14m
konnectivity-agent-9rhdh 0/1 Running 0 14m
konnectivity-agent-hnlbn 0/1 Running 0 14m
Remove the rule and wait:
# on k0s-ctrl
sudo iptables -D INPUT -p tcp --dport 8132 -j DROP
sleep 15
kubectl -n kube-system get pods -l k8s-app=konnectivity-agent
kubectl -n kube-system logs ds/kube-router -c kube-router --tail=1 | cut -c1-60Expected (abridged):
NAME READY STATUS RESTARTS AGE
konnectivity-agent-4249z 1/1 Running 0 15m
konnectivity-agent-9rhdh 1/1 Running 0 15m
konnectivity-agent-hnlbn 1/1 Running 0 15m
I1119 23:52:31.283921 3076 bgp_policies.go:991] Ensuring t...
After about ten seconds, the tunnels are back. You see the same picture after every controller restart, until all agents have reconnected.
Save your progress? Sign up for free
Drill 6: Manual changes do not last
# on k0s-ctrl
kubectl label node k0s-w3 topology.kubernetes.io/zone=b --overwrite
kubectl get node k0s-w3 -L topology.kubernetes.io/zone
ssh k0s-w3 sudo systemctl restart k0sworker
sleep 30
kubectl get node k0s-w3 -L topology.kubernetes.io/zoneExpected:
node/k0s-w3 labeled
NAME STATUS ROLES AGE VERSION ZONE
k0s-w3 Ready <none> 15m v1.36.6+k0s b
NAME STATUS ROLES AGE VERSION ZONE
k0s-w3 Ready <none> 16m v1.36.6+k0s a
On startup, the kubelet resets the labels from --labels.
Your kubectl label only lasted until the next restart. For
a zone, that is dangerous: between two restarts, placement and topology
would believe something different from LINSTOR.
Save your progress? Sign up for free
Drill 7: Change the configuration via k0sctl
For storage nodes, graceful node shutdown pays off: on shutdown, the kubelet first terminates the Pods so that DRBD can release cleanly. This setting belongs in the kubelet configuration. k0s distributes it as a worker profile.
# on k0s-ctrl
cp ../tag01/k0sctl.yaml .
sed -i 's|zone=\([ab]\)\]|zone=\1, --profile=storage]|' k0sctl.yaml
cat >> k0sctl.yaml <<'EOF'
workerProfiles:
- name: storage
values:
shutdownGracePeriod: 60s
shutdownGracePeriodCriticalPods: 20s
EOF
diff ../tag01/k0sctl.yaml k0sctl.yamlExpected:
11c11
< installFlags: [--labels=topology.kubernetes.io/zone=a]
---
> installFlags: [--labels=topology.kubernetes.io/zone=a, --profile=storage]
14c14
...
34a35,39
> workerProfiles:
> - name: storage
> values:
> shutdownGracePeriod: 60s
> shutdownGracePeriodCriticalPods: 20s
First, see what would happen:
# on k0s-ctrl
k0sctl apply --config k0sctl.yaml --dry-run 2>&1 | grep dry-run: | cut -c1-80Expected (abridged):
dry-run: - restart the k0s service
dry-run: * 172.16.0.186:22 :
dry-run: - reinstall k0s using `k0s install worker --labels=topology.kubernetes...
dry-run: - restart k0s service
...
Restart the controller, reinstall all workers. On a large cluster, you do not want that on all workers at the same time, so one after the other:
# on k0s-ctrl
k0sctl apply --config k0sctl.yaml --concurrency 1 2>&1 \
| grep -E 'Reinstall|reinstalling'
kubectl get --raw /api/v1/nodes/k0s-w1/proxy/configz | python3 -m json.tool \
| grep shutdownGrace
ssh k0s-w1 'systemd-inhibit --list --no-pager | grep -c kubelet'Expected:
level=info msg="==> Running phase: Reinstall"
level=info msg="Reinstalling max 1 workers in parallel"
level=info msg="172.16.0.186:22: reinstalling k0s"
level=info msg="172.16.0.36:22: reinstalling k0s"
level=info msg="172.16.0.72:22: reinstalling k0s"
"shutdownGracePeriod": "1m0s",
"shutdownGracePeriodCriticalPods": "20s",
0
The configuration has arrived, but the kubelet has registered no shutdown inhibitor lock with systemd-logind. Without it, nobody waits for the Pods on shutdown.
# on k0s-ctrl
ssh k0s-w1 'sudo journalctl -u k0sworker -o cat' \
| grep -o 'Failed to start node shut.*' | cut -c1-88
ssh k0s-w1 'systemd-analyze cat-config systemd/logind.conf \
| grep -E "^# /.*conf$|^Inhibit"'Expected (abridged):
Failed to start node shutdown manager\" err=\"node shutdown manager was timed out af...
# /etc/systemd/logind.conf
# /etc/systemd/logind.conf.d/99-kubelet.conf
InhibitDelayMaxSec=60
# /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf
InhibitDelayMaxSec=30
The kubelet wrote its 60 seconds into 99-kubelet.conf.
But systemd reads the drop-ins sorted by file name across all
directories, and unattended-upgrades-… comes after
99-…. The 30 seconds from the unattended-upgrades package
win, and the kubelet gives up. A file of your own that sorts last solves
this:
# on k0s-w1
sudo tee /etc/systemd/logind.conf.d/zz-kubelet.conf >/dev/null <<'EOF'
[Login]
InhibitDelayMaxSec=60
EOF
sudo systemctl restart systemd-logind
sudo systemctl restart k0sworker
sleep 25
systemd-inhibit --list --no-pager | grep kubeletExpected (abridged):
kubelet 0 root 6348 kubelet shutdown Kubelet needs ti...
Repeat this on k0s-w2 and k0s-w3. From now
on, ~/k0s-lab/tag02/k0sctl.yaml is the valid cluster
description. The logind file does not belong in k0sctl but in host
preparation – on Day 16, Ansible takes it over.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show since when
k0scontrollerhas been running. Target time: 20 s - Show the last five journal lines of
k0sworkeronk0s-w2. Target time: 20 s - Count the running containers on
k0s-w1using containerd tools. Target time: 30 s - Show the shutdown settings from the running kubelet configuration of
k0s-w2. Target time: 40 s
Solution
# on k0s-ctrl
systemctl show -p ActiveEnterTimestamp k0scontroller
ssh k0s-w2 'sudo journalctl -u k0sworker -n 5 --no-pager -o cat'
ssh k0s-w1 'sudo k0s ctr -n k8s.io tasks ls | grep -c RUNNING'
kubectl get --raw /api/v1/nodes/k0s-w2/proxy/configz | python3 -m json.tool \
| grep -i shutdownGraceExpected (abridged):
ActiveEnterTimestamp=Fri 2026-11-20 00:01:15 UTC
time="2026-11-20 00:06:02" level=info msg="I1120 00:06:02.951511 6402 prober.go:1...
...
8
"shutdownGracePeriod": "1m0s",
"shutdownGracePeriodCriticalPods": "20s",
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zoneExpected: three nodes Ready, zones a,
b, a.
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
ssh $h 'systemd-inhibit --list | grep -c kubelet'
doneExpected: 1 three times.
# on k0s-ctrl
sudo iptables -S INPUTExpected: only -P INPUT ACCEPT – the
rule from Drill 5 is gone.
# on k0s-ctrl
kubectl -n kube-system get pods -l k8s-app=konnectivity-agent --no-headers | grep -c 1/1Expected: 3
Challenge
Stop the service on k0s-w3 and measure how long
Kubernetes takes until the node is NotReady. Find out
whether the containers on k0s-w3 keep running meanwhile.
Start the service again afterwards.
Hint: When stopping, the journal of
k0sworker tells you what is left behind.
Solution
# on k0s-ctrl
ssh k0s-w3 sudo systemctl stop k0sworker
s=$(date +%s)
until kubectl get node k0s-w3 --no-headers | grep -q NotReady; do sleep 2; done
echo "NotReady nach $(( $(date +%s) - s ))s"
ssh k0s-w3 'sudo journalctl -u k0sworker -n 3 --no-pager -o cat'
ssh k0s-w3 sudo systemctl start k0sworkerExpected:
NotReady nach 50s
k0sworker.service: Unit process 3508 (containerd-shim) remains running after unit st...
Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
k0sworker.service: Consumed 8.099s CPU time, 739.7M memory peak, 0B memory swap peak.
After 50 seconds (node-monitor-grace-period), the
controller marks the node as NotReady. The containers keep
running: the containerd-shim processes survive the service
stop. For storage, this means: a stopped worker keeps its volumes open
and may still be writing. NotReady is no proof that a node
is dead – this becomes important on Day 10. After the start,
k0s-w3 is Ready again within a few
seconds.
Save your progress? Sign up for free
Interview Note
kubectl getworks,kubectl logshangs – what is the cause? The path from the API server to the kubelet. With k0s, that is konnectivity: check the agents, port 8132 on the controller, the journal withcomponent=konnectivity.- Where is the kubelet directory in k0s?
/var/lib/k0s/kubelet. CSI drivers and backup agents must be configured for it. - How do you change the kubelet configuration? Via worker
profiles in
k0sctl.yamlandk0sctl apply, with--concurrencyfor many nodes.
Enterprise Note
- Graceful node shutdown on all nodes with databases. Check the
inhibitor lock with
systemd-inhibit --list– otherwise a silent misconfiguration like this one only shows up in the next maintenance window. - Collect the journal centrally (Loki, Elastic), since one service bundles all components.
- Firewalls between workers and controllers: 6443 (API), 8132
(konnectivity), 9443 (join). If 8132 is missing,
logs,execand metrics fail first. - Changes to
k0sctl.yamlvia pull request, with the--dry-runoutput included in the review.