Skip to content
Hands-on IT IT Drills

Day 1: Building k0s with k0sctl

Goal: A k0s v1.36.6 cluster with one controller and three workers is running, the workers carry their zone as a label, and kubectl on k0s-ctrl talks to it. · Duration: approx. 45 min · Prerequisite: Day 0

Briefing

  • k0s is a single binary with API server, scheduler, etcd, containerd, kubelet and CNI. From the operating system, it only needs the kernel.
  • k0sctl distributes k0s via SSH: download, configuration, join tokens, later updates. It runs on the workstation and needs access to all hosts from there.
  • A pure controller is not a node: no kubelet, no Pods, not in kubectl get nodes.
  • The cluster description k0sctl.yaml is the single source of truth. k0sctl apply reconciles the cluster with it, as often as you like.
  • You deliberately build v1.36.6. You practice the update to 1.37 on Day 15.
 k0s-ctrl: k0sctl ──SSH──> k0s-ctrl  k0scontroller (etcd, API, scheduler)
                  ──SSH──> k0s-w1/2/3 k0sworker (containerd, kubelet)
           kubectl ──6443──> API      konnectivity: worker ──8132──> controller

Drill 1: Download k0sctl

In this book, you always download binaries matching the architecture. dpkg --print-architecture returns amd64 or arm64 – exactly what the release files are called.

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag01 && cd ~/k0s-lab/tag01
ARCH=$(dpkg --print-architecture); echo $ARCH
V=v0.33.1
curl -sSfLO https://github.com/k0sproject/k0sctl/releases/download/$V/k0sctl-linux-$ARCH
curl -sSfLO https://github.com/k0sproject/k0sctl/releases/download/$V/checksums.txt
grep "k0sctl-linux-$ARCH\$" checksums.txt | sha256sum -c
sudo install -m 755 k0sctl-linux-$ARCH /usr/local/bin/k0sctl
k0sctl version

Expected:

amd64
k0sctl-linux-amd64: OK
version: v0.33.1
commit: c4bfa95

The \$ in the grep anchors the name at the end of the line: sha256sum -c gets exactly the one line for your file.

Save your progress? Sign up for free

Drill 2: kubectl matching the cluster

kubectl may differ from the API server by at most one minor version. You take exactly the cluster’s version.

# on k0s-ctrl
K=v1.36.6
curl -sSfLO https://dl.k8s.io/release/$K/bin/linux/$ARCH/kubectl
curl -sSfLO https://dl.k8s.io/release/$K/bin/linux/$ARCH/kubectl.sha256
echo "$(cat kubectl.sha256)  kubectl" | sha256sum -c
sudo install -m 755 kubectl /usr/local/bin/kubectl
kubectl version --client

Expected:

kubectl: OK
Client Version: v1.36.6
Kustomize Version: v5.8.1

On the controller, there would also be sudo k0s kubectl. But that runs as root with admin rights, and only there. For everyday work, a normal kubectl with a kubeconfig is cleaner.

Save your progress? Sign up for free

Drill 3: The skeleton from k0sctl init

# on k0s-ctrl
k0sctl init --k0s -C 1 -i ~/.ssh/id_ed25519 [email protected] \
  [email protected] [email protected] [email protected] > init.yaml
wc -l init.yaml
grep role: init.yaml
grep -A1 telemetry init.yaml
grep -c version: init.yaml

Expected:

88 init.yaml
    role: controller
    role: worker
    role: worker
    role: worker
        telemetry:
          enabled: true
0

-C 1 makes the first address the controller and all others workers. The skeleton writes out every default value – 88 lines you do not want to maintain. Telemetry is on, and two things are missing entirely: a fixed version (0 matches) and the zones.

Save your progress? Sign up for free

Drill 4: Your own k0sctl.yaml

# on k0s-ctrl
cat > k0sctl.yaml <<'EOF'
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-lab
spec:
  hosts:
  - role: controller
    ssh: {address: 172.16.0.176, user: ubuntu, keyPath: ~/.ssh/id_ed25519}
  - role: worker
    ssh: {address: 172.16.0.186, user: ubuntu, keyPath: ~/.ssh/id_ed25519}
    installFlags: [--labels=topology.kubernetes.io/zone=a]
  - role: worker
    ssh: {address: 172.16.0.36, user: ubuntu, keyPath: ~/.ssh/id_ed25519}
    installFlags: [--labels=topology.kubernetes.io/zone=b]
  - role: worker
    ssh: {address: 172.16.0.72, user: ubuntu, keyPath: ~/.ssh/id_ed25519}
    installFlags: [--labels=topology.kubernetes.io/zone=a]
  k0s:
    version: v1.36.6+k0s.0
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s-lab
      spec:
        api:
          address: 172.16.0.176
          sans: [172.16.0.176]
        network:
          provider: kuberouter
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/12
        telemetry:
          enabled: false
EOF
  • role: controller means control plane without workloads. controller+worker lets the controller carry Pods as well – meant for small clusters, not wanted here.
  • version pins k0s. Without that line, k0sctl takes the newest one, and a later apply updates the cluster without asking. The suffix +k0s.0 is part of it.
  • installFlags end up in the worker’s systemd unit. That way, the zone label is back on the node after every start and every re-registration. A kubectl label does not survive that (Day 2).
  • topology.kubernetes.io/zone is the standard label that the scheduler and CSI drivers know. In LINSTOR, you additionally set the zone as Aux/zone on Day 4.

Save your progress? Sign up for free

Drill 5: The cluster comes to life

# on k0s-ctrl
k0sctl apply --config k0sctl.yaml

Expected (abridged):

level=info msg="==> Running phase: Gather host facts"
level=info msg="172.16.0.72:22: detected amd64 architecture"
...
level=info msg="==> Running phase: Validate hosts"
level=info msg="validating clock skew"
...
level=info msg="172.16.0.72:22: downloading k0s v1.36.6+k0s.0"
...
level=info msg="==> Running phase: Initialize the k0s cluster"
level=info msg="172.16.0.176:22: installing k0s controller"
level=info msg="172.16.0.176:22: wait for kubernetes to reach ready state"
level=info msg="==> Running phase: Install workers"
level=info msg="172.16.0.176:22: generating a join token for worker 1"
...
level=info msg="172.16.0.186:22: waiting for node to become ready"
level=info msg="==> Finished in 2m38s"
level=info msg="k0s cluster version v1.36.6+k0s.0 is now installed"

k0sctl detects each host’s architecture and downloads the matching k0s binary directly there. Before that, it checks the clock on all hosts – that is why NTP mattered on Day 0.

Save your progress? Sign up for free

Drill 6: Connect kubectl

# on k0s-ctrl
mkdir -p ~/.kube
k0sctl kubeconfig --config k0sctl.yaml > ~/.kube/config
chmod 600 ~/.kube/config
grep server ~/.kube/config
kubectl get nodes -o wide

Expected (abridged):

    server: https://172.16.0.176:6443
NAME     STATUS   ROLES    AGE   VERSION       INTERNAL-IP    ... CONTAINER-RUNTIME
k0s-w1   Ready    <none>   50s   v1.36.6+k0s   172.16.0.186   ... containerd://2.3.6
k0s-w2   Ready    <none>   50s   v1.36.6+k0s   172.16.0.36    ... containerd://2.3.6
k0s-w3   Ready    <none>   50s   v1.36.6+k0s   172.16.0.72    ... containerd://2.3.6

Three nodes, not four: k0s-ctrl is missing, and that is correct. Wait a minute, then the system Pods are ready:

# on k0s-ctrl
kubectl get pods -A

Expected:

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-7c5b856895-4dxld          1/1     Running   0          3m51s
kube-system   coredns-7c5b856895-l4zk8          1/1     Running   0          3m51s
kube-system   konnectivity-agent-4249z          1/1     Running   0          3m57s
kube-system   konnectivity-agent-9rhdh          1/1     Running   0          3m57s
kube-system   konnectivity-agent-hnlbn          1/1     Running   0          3m56s
kube-system   kube-proxy-5brnr                  1/1     Running   0          3m57s
kube-system   kube-proxy-kl9cr                  1/1     Running   0          3m57s
kube-system   kube-proxy-vz59j                  1/1     Running   0          3m57s
kube-system   kube-router-b9zjl                 1/1     Running   0          3m57s
kube-system   kube-router-t9dhd                 1/1     Running   0          3m57s
kube-system   kube-router-zh4sq                 1/1     Running   0          3m57s
kube-system   metrics-server-5749cd655b-dpx5r   1/1     Running   0          4m1s

API server, scheduler and etcd are missing from the list. They run on k0s-ctrl as normal processes under systemd, not as Pods. Right after the build, you briefly see three coredns Pods: k0s rescales coredns as soon as more than one node exists.

Tip: kubectl -n kube-system wait --for=condition=Ready pod --all hangs here until the timeout, because it also waits for the coredns Pod that was just deleted. Use kubectl get pods -A or rollout status for individual Deployments.

Save your progress? Sign up for free

Drill 7: Look on the hosts

# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zone
sudo k0s status
sudo k0s etcd member-list
ssh k0s-w1 sudo k0s status

Expected (abridged):

NAME     STATUS   ROLES    AGE     VERSION       ZONE
k0s-w1   Ready    <none>   4m46s   v1.36.6+k0s   a
k0s-w2   Ready    <none>   4m46s   v1.36.6+k0s   b
k0s-w3   Ready    <none>   4m46s   v1.36.6+k0s   a
Version: v1.36.6+k0s.0
Process ID: 2604
Role: controller
Workloads: false
SingleNode: false
{"members":{"k0s-ctrl":"https://172.16.0.176:2380"}}
Version: v1.36.6+k0s.0
Process ID: 2551
Role: worker
Workloads: true
SingleNode: false
Kube-api probing successful: true

Workloads: false confirms: Pods never run on the controller. But a single etcd member also means: if k0s-ctrl fails, the Pods keep running, but nobody can change anything anymore. How three controllers solve this is covered on Day 19.

And one more apply, without changing anything:

# on k0s-ctrl
k0sctl apply --config k0sctl.yaml 2>&1 | grep -E 'phase|running k0s|Finished'

Expected (abridged):

level=info msg="==> Running phase: Gather k0s facts"
level=info msg="172.16.0.176:22: is running k0s controller version v1.36.6+k0s.0"
level=info msg="172.16.0.186:22: is running k0s worker version v1.36.6+k0s.0"
...
level=info msg="==> Running phase: Run After Apply Hooks"
level=info msg="==> Running phase: Disconnect from hosts"
level=info msg="==> Finished in 9s"

No install phase, done after nine seconds. apply is idempotent: you can run it safely at any time.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the nodes with their zone as a separate column. Target time: 15 s
  2. Show the k0s version on k0s-w2. Target time: 15 s
  3. Show all Pods running on k0s-w2. Target time: 30 s
  4. Show which API address kubectl talks to. Target time: 20 s

Solution

# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zone
ssh k0s-w2 sudo k0s version
kubectl get pods -A -o wide --field-selector spec.nodeName=k0s-w2
kubectl config view -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Expected (abridged):

...
v1.36.6+k0s.0
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE ...
kube-system   konnectivity-agent-9rhdh          1/1     Running   0          4m49s
kube-system   kube-proxy-vz59j                  1/1     Running   0          4m49s
kube-system   kube-router-zh4sq                 1/1     Running   0          4m49s
kube-system   metrics-server-5749cd655b-dpx5r   1/1     Running   0          4m53s
https://172.16.0.176:6443

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zone

Expected: three nodes Ready, v1.36.6+k0s, zones a, b, a.

# on k0s-ctrl
kubectl get pods -A --no-headers | grep -vc Running

Expected: 0

# on k0s-ctrl
sudo k0s status | grep -E 'Version|Workloads'

Expected:

Version: v1.36.6+k0s.0
Workloads: false

Challenge

Start a Deployment spread with four pause Pods that spread evenly across the two zones, and delete it afterwards.

Hint: topologySpreadConstraints with the zone label as topologyKey.

Solution

# on k0s-ctrl
cat > spread.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: spread, namespace: default}
spec:
  replicas: 4
  selector: {matchLabels: {app: spread}}
  template:
    metadata: {labels: {app: spread}}
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector: {matchLabels: {app: spread}}
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.10
EOF
kubectl apply -f spread.yaml
kubectl rollout status deploy/spread --timeout=120s
kubectl get pods -l app=spread -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Expected:

deployment.apps/spread created
...
deployment "spread" successfully rolled out
      1 k0s-w1
      2 k0s-w2
      1 k0s-w3

Two Pods in zone a (spread across k0s-w1 and k0s-w3), two in zone b – which only has k0s-w2. You will meet the same imbalance again on Day 12: zone b must carry one copy of every volume.

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
kubectl delete deploy spread

Interview Note

  • What distinguishes k0s from kubeadm? A single binary without dependencies, the control plane as processes under systemd instead of static Pods, controllers without a kubelet by default.
  • How do you distribute k0s to several hosts? With k0sctl and a declarative k0sctl.yaml. apply is idempotent and also serves for expansion and updates.
  • Why pin the version? Without a pin, every apply pulls the newest version – an unplanned update in the middle of operations.

Enterprise Note

  • Three controllers (etcd quorum) behind a load balancer or with k0s node-local load balancing. With an external load balancer, api.externalAddress points to its address.
  • k0sctl.yaml belongs in Git, changes go through review. The SSH key of the automation host is a critical secret.
  • Air gap: k0sctl can upload binaries and images from the workstation (uploadBinary, airgap bundle) instead of downloading them on every host.
  • Do not distribute the admin’s kubeconfig. Use OIDC for humans, dedicated ServiceAccounts with RBAC for automation.