Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: Turn three empty Ubuntu VMs into the RKE2 cluster debug, where from tomorrow on you break things and track the faults down. · Duration: approx. 90 min · Prerequisite: three VMs with Ubuntu 24.04, SSH access and sudo

Briefing

  • You build an RKE2 cluster with three nodes: cp1 is the server (control plane, etcd) and also your workstation, worker1 and worker2 are agents.
  • RKE2 ships Canal as CNI, Traefik, CoreDNS and metrics-server. You add a StorageClass yourself, but deliberately not as the default.
  • Today you get to know the places where the answers will be later: the journal of rke2-server and rke2-agent, crictl on the node and the kubeconfig.
  • Talos only joins on Day 18, as a separate cluster on the VM argo.
  Management network 172.16.0.0/24
 ────┬──────────────────┬──────────────────┬────
 cp1 .126           worker1 .228       worker2 .83
 rke2-server        rke2-agent         rke2-agent
 etcd, API server   kubelet            kubelet
 = workstation      containerd         containerd
 kubectl k9s helm

Drill 1: The Starting Point

The book was tested on three VMs under Proxmox: cp1 with 4 vCPU and 11 GiB RAM, the two workers with 2 vCPU and 4 GiB each. All run Ubuntu 24.04 on amd64.

# on cp1
lsb_release -ds
nproc
free -h | head -2
dpkg --print-architecture
sudo ufw status

Expected:

Ubuntu 24.04.5 LTS
4
               total        used        free      shared  buff/cache   available
Mem:            11Gi       434Mi        11Gi       1.1Mi       297Mi        11Gi
amd64
Status: inactive

If ufw is active, you need rules for 6443 and 9345 (TCP, server), 10250 and 30000–32767 (TCP, all nodes; NodePorts, from Day 4), 8472 (UDP, Canal’s VXLAN) and 9099 (TCP, Canal’s health check). The list comes from the RKE2 docs; on a worker with ufw active, everything worked with it (Pods, Services, NodePorts, DNS). The challenge shows what a blocked 9345 looks like.

For the node days you go from cp1 to the workers via SSH. Add the names and create a key if cp1 does not have one yet:

# on cp1
sudo tee -a /etc/hosts <<'EOF'
172.16.0.126 cp1
172.16.0.228 worker1
172.16.0.83  worker2
EOF
[ -f ~/.ssh/id_ed25519 ] || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub

Append the public key to ~/.ssh/authorized_keys on both workers, the same way you deployed your own key. Then:

# on cp1
ssh -o StrictHostKeyChecking=accept-new worker1 hostname
ssh -o StrictHostKeyChecking=accept-new worker2 hostname

Expected:

worker1
worker2

Save your progress? Sign up for free

Drill 2: The Server Configuration

On its first start, RKE2 reads /etc/rancher/rke2/config.yaml. Each key corresponds to an option of rke2 server. The file must be in place before the service starts for the first time.

# on cp1
mkdir -p ~/debug/tag00 && cd ~/debug/tag00
openssl rand -hex 32 > rke2-token
chmod 600 rke2-token
sudo mkdir -p /etc/rancher/rke2
sudo tee /etc/rancher/rke2/config.yaml >/dev/null <<EOF
node-name: cp1
node-ip: 172.16.0.126
token: $(cat rke2-token)
tls-san:
  - 172.16.0.126
  - cp1
write-kubeconfig-mode: "0600"
cni: canal
EOF
sudo chmod 600 /etc/rancher/rke2/config.yaml
sudo sed 's/^token: .*/token: <gekürzt>/' /etc/rancher/rke2/config.yaml

Expected:

node-name: cp1
node-ip: 172.16.0.126
token: <gekürzt>
tls-san:
  - 172.16.0.126
  - cp1
write-kubeconfig-mode: "0600"
cni: canal

Why? The heredoc marker EOF is unquoted here so that the shell substitutes $(cat rke2-token). token is the shared secret of server and agents, tls-san adds the IP and the name to the API server certificate. (<gekürzt> just means ‘redacted’.)

Save your progress? Sign up for free

Drill 3: Start the Server and Read Its Log

# on cp1
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.37.2+rke2r1 sh -
time sudo systemctl enable --now rke2-server.service

Expected (abridged):

[INFO]  using v1.37.2+rke2r1 as release
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → ...

real    1m29.932s

systemctl only returns once etcd and the API server are running – on the first start, including loading the images, about a minute and a half. The log is the first place you look when it takes longer:

# on cp1
sudo journalctl -u rke2-server -b -o cat \
  | grep -E 'Starting rke2|Wrote kubeconfig|level=error' | cut -c1-88

Expected (abridged):

time="2026-11-20T06:29:51Z" level=info msg="Starting rke2 v1.37.2+rke2r1 (972ca61add47ea
time="2026-11-20T06:29:56Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:
time="2026-11-20T06:29:56Z" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml
time="2026-11-20T06:30:24Z" level=error msg="Failed to process image event: failed to im
time="2026-11-20T06:31:19Z" level=error msg="Failed to process image event: failed to im

Why? Not every level=error is a fault. The 503 … runtime core not ready lines come from the first seconds, when the supervisor is already receiving requests but etcd is not ready yet. Failed to process image event … not found is reported by the import of the image lists; the kubelet pulls the images afterwards anyway. What matters is whether Pods run afterwards – Drill 4 shows that.

Tip: journalctl -u rke2-server -f follows the log live. Errors in config.yaml (unknown key, wrong indentation) show up right at the beginning.

Save your progress? Sign up for free

Drill 4: kubectl, Kubeconfig and the Three Variables

RKE2 puts kubectl, crictl and ctr into /var/lib/rancher/rke2/bin and writes the admin kubeconfig to /etc/rancher/rke2/rke2.yaml, readable by root only. crictl only finds the containerd socket with CRI_CONFIG_FILE.

# on cp1
sudo tee /etc/profile.d/rke2.sh >/dev/null <<'EOF'
export PATH=$PATH:/var/lib/rancher/rke2/bin
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
if [ "$(id -u)" = 0 ]; then export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; fi
EOF
sudo ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
mkdir -p ~/.kube
sudo install -m 600 -o "$USER" -g "$USER" /etc/rancher/rke2/rke2.yaml ~/.kube/config
kubectl get nodes

Expected:

NAME   STATUS     ROLES                AGE   VERSION
cp1    NotReady   control-plane,etcd   49s   v1.37.2+rke2r1

NotReady is normal in the first minute: Canal has to start before the kubelet reports the Pod network. Now the same from containerd’s point of view. sudo -i opens a login shell and reads /etc/profile.d/rke2.sh along the way:

# on cp1
sudo -i crictl ps -o table | head -5 | cut -c1-88

Expected (abridged):

CONTAINER           IMAGE               CREATED              STATE               NAME
9b0b5d0288151       ddf8337da7bdb       40 seconds ago       Running             cloud-c
c568c35ccde1d       6d31c161dcb90       44 seconds ago       Running             kube-sc
924396c58fd99       6d31c161dcb90       44 seconds ago       Running             kube-co
a35a7f4fd0f0b       6d31c161dcb90       54 seconds ago       Running             kube-ap

Why? In RKE2 the control plane runs as static Pods that the kubelet starts by itself. If the API server goes down, you no longer see them with kubectl, but you still do with crictl (Day 15).

Save your progress? Sign up for free

Drill 5: The Agents

The agent registers with the server via the supervisor port 9345, not via 6443. Over this connection it fetches certificates and configuration. First, each worker gets the token:

# on cp1
cd ~/debug/tag00
scp -q rke2-token worker1: && scp -q rke2-token worker2:
# on worker1
sudo mkdir -p /etc/rancher/rke2
sudo tee /etc/rancher/rke2/config.yaml >/dev/null <<EOF
server: https://172.16.0.126:9345
node-name: worker1
node-ip: 172.16.0.228
token: $(cat ~/rke2-token)
EOF
sudo chmod 600 /etc/rancher/rke2/config.yaml
rm ~/rke2-token
curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_VERSION=v1.37.2+rke2r1 INSTALL_RKE2_TYPE=agent sh -
sudo systemctl enable --now rke2-agent.service

Do the same on worker2 with node-name: worker2 and node-ip: 172.16.0.83. Both workers also need the variables from Drill 4: create the same file /etc/profile.d/rke2.sh there (the KUBECONFIG line has no effect on agents and does no harm). The agent’s log:

# on worker1
sudo journalctl -u rke2-agent -b -o cat \
  | grep -E 'Starting rke2|Remotedialer|Running kubelet' | cut -c1-88

Expected:

time="2026-11-20T06:32:43Z" level=info msg="Starting rke2 agent v1.37.2+rke2r1 (972ca61a
time="2026-11-20T06:32:58Z" level=info msg="Remotedialer connected to proxy" url="wss://
time="2026-11-20T06:34:00Z" level=info msg="Running kubelet --alsologtostderr=false --cl

Remotedialer connected means the tunnel to the supervisor is up. If the line is missing, the problem lies between the agent and port 9345.

# on cp1
kubectl get nodes -o wide | cut -c1-88

Expected:

NAME      STATUS   ROLES                AGE     VERSION          INTERNAL-IP    EXTERNAL
cp1       Ready    control-plane,etcd   8m49s   v1.37.2+rke2r1   172.16.0.126   <none>
worker1   Ready    <none>               6m      v1.37.2+rke2r1   172.16.0.228   <none>
worker2   Ready    <none>               4m38s   v1.37.2+rke2r1   172.16.0.83    <none>

Save your progress? Sign up for free

Drill 6: What RKE2 Ships

RKE2 installs its add-on components as Helm charts via its own controller. Each chart is an object of type HelmChart, installed by a Job helm-install-<name>.

# on cp1
kubectl -n kube-system get helmcharts
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
RUNTIME:.status.nodeInfo.containerRuntimeVersion

Expected (abridged):

NAME                           REPO   CHART   VERSION   TARGETNAMESPACE   BOOTSTRAP ...
rke2-canal                                                                true      ...
rke2-coredns                                                              true      ...
rke2-gateway-api-crd                                                                ...
rke2-metrics-server                                                                 ...
rke2-runtimeclasses                                                                 ...
rke2-security-responder                                                             ...
rke2-snapshot-controller                                                            ...
rke2-snapshot-controller-crd                                                        ...
rke2-traefik                                                                        ...
rke2-traefik-crd                                                                    ...
NAME      RUNTIME
cp1       containerd://2.3.7-k3s1
worker1   containerd://2.3.7-k3s1
worker2   containerd://2.3.7-k3s1

Why? In RKE2 1.37, Traefik is the default ingress controller, and the Gateway API CRDs come from a chart of their own. If a helm-install-… Pod gets stuck in Error, the reason is in its log (Day 9).

Talos: Talos ships neither an ingress controller nor metrics-server; Days 18 and 19 show what that means for troubleshooting.

Save your progress? Sign up for free

Drill 7: The Tools

You install helm, k9s and the Kyverno CLI in an architecture-neutral way. The Kyverno CLI calls amd64 x86_64, hence the substitution in KARCH.

# on cp1
cd ~/debug/tag00
ARCH=$(dpkg --print-architecture)
curl -sSfLO https://get.helm.sh/helm-v4.3.2-linux-${ARCH}.tar.gz
tar xzf helm-v4.3.2-linux-${ARCH}.tar.gz linux-${ARCH}/helm
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
K9S=https://github.com/derailed/k9s/releases/download/v0.51.0
curl -sSfLO ${K9S}/k9s_Linux_${ARCH}.tar.gz
tar xzf k9s_Linux_${ARCH}.tar.gz k9s
sudo install -m 755 k9s /usr/local/bin/k9s
KARCH=${ARCH/amd64/x86_64}
KYV=https://github.com/kyverno/kyverno/releases/download/v1.20.0
curl -sSfLO ${KYV}/kyverno-cli_v1.20.0_linux_${KARCH}.tar.gz
tar xzf kyverno-cli_v1.20.0_linux_${KARCH}.tar.gz kyverno
sudo install -m 755 kyverno /usr/local/bin/kyverno
rm -rf linux-${ARCH} k9s kyverno *.tar.gz
helm version --short
k9s version | head -1
kyverno version | head -1

Expected:

v4.3.2+g8346aaa
Version              v0.51.0
Version: v1.20.0

Ubuntu 24.04 already ships jq, curl and openssl.

Save your progress? Sign up for free

Drill 8: A StorageClass, but No Default

RKE2 has no StorageClass. You install the local-path-provisioner. Its manifest does not set the default marker, and it stays that way: on Days 3 and 10 you need the fault “PVC without a StorageClass”.

# on cp1
kubectl apply -f https://raw.githubusercontent.com/rancher/\
local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
kubectl -n local-path-storage rollout status deploy/local-path-provisioner
kubectl get storageclass

Expected (abridged):

namespace/local-path-storage created
...
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
deployment "local-path-provisioner" successfully rolled out
NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ...
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   ...

There is no (default) after local-path. A small test shows whether a volume is really created:

# on cp1
cat > pvc-test.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag00}
spec:
  storageClassName: local-path
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: schreiber, namespace: tag00}
spec:
  containers:
  - name: schreiber
    image: busybox:1.37
    command: [sh, -c, 'date > /daten/hallo.txt; cat /daten/hallo.txt; sleep 3600']
    volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
  - name: daten
    persistentVolumeClaim: {claimName: daten}
EOF
kubectl create namespace tag00
kubectl apply -f pvc-test.yaml
kubectl -n tag00 wait pod/schreiber --for=condition=Ready --timeout=120s
kubectl -n tag00 get pod schreiber -o wide | cut -c1-72
kubectl get pv -o custom-columns=PATH:.spec.hostPath.path | cut -c1-88

Expected:

namespace/tag00 created
persistentvolumeclaim/daten created
pod/schreiber created
pod/schreiber condition met
NAME        READY   STATUS    RESTARTS   AGE   IP          NODE      NOMINATED NODE
schreiber   1/1     Running   0          13s   10.42.1.5   worker1   <none>
PATH
/opt/local-path-provisioner/pvc-6626be9a-953e-42e6-98d6-8c30e89d4f60_tag00_daten

The directory lives on the node shown in the NODE column, which may be worker2 in your cluster:

# on worker1
sudo cat /opt/local-path-provisioner/*_tag00_daten/hallo.txt

Expected:

Fri Nov 20 06:37:38 UTC 2026

Tip: Take a snapshot of all three VMs now. From Day 3 on you break things; every day cleans up after itself, but a snapshot is the fastest way back.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show name, kubelet version and container runtime of all nodes. Target time: 30 s
  2. From cp1, show the Flannel containers on worker2. Target time: 30 s
  3. Count the level=error lines of rke2-server since the last boot. Target time: 30 s
  4. Check whether the API server is ready and show only the last line. Target time: 20 s

Solution

# on cp1
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
VERSION:.status.nodeInfo.kubeletVersion,\
RUNTIME:.status.nodeInfo.containerRuntimeVersion
ssh worker2 sudo -i crictl ps --name flannel | cut -c1-88
sudo journalctl -u rke2-server -b -o cat | grep -c level=error
kubectl get --raw='/readyz?verbose' | tail -1

Expected (abridged):

NAME      VERSION          RUNTIME
cp1       v1.37.2+rke2r1   containerd://2.3.7-k3s1
worker1   v1.37.2+rke2r1   containerd://2.3.7-k3s1
worker2   v1.37.2+rke2r1   containerd://2.3.7-k3s1
CONTAINER           IMAGE               CREATED             STATE               NAME
d27d5570b3822       4f23b420044ae       5 minutes ago       Running             kube-fla
6
readyz check passed

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl get nodes

Expected: three nodes, all Ready, version v1.37.2+rke2r1.

# on cp1
kubectl get storageclass -o name

Expected:

storageclass.storage.k8s.io/local-path
# on cp1
kubectl get storageclass local-path \
  -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}'

Expected: no output – local-path is not the default.

# on cp1
ssh worker1 sudo -i crictl version | tail -2

Expected:

RuntimeVersion:  v2.3.7-k3s1
RuntimeApiVersion:  v1

Challenge

On cp1, block port 9345 for worker2 only, restart rke2-agent there and find the message that reveals the reason. Remove the block afterwards.

Hint: The agent talks to the supervisor through a local load balancer on port 6444 – so the message does not only mention 9345.

Solution

# on cp1
sudo iptables -I INPUT -p tcp --dport 9345 -s 172.16.0.83 -j REJECT
# on worker2
sudo systemctl restart rke2-agent --no-block
systemctl is-active rke2-agent
sudo journalctl -u rke2-agent --since -1min -o cat | grep level=error | tail -2
nc -zv -w3 172.16.0.126 9345
nc -zv -w3 172.16.0.126 6443

Expected (lines wrapped):

activating
time="2026-11-20T06:38:32Z" level=error msg="Failed to validate connection to
  cluster at https://172.16.0.126:9345: failed to get CA certs:
  Get \"https://127.0.0.1:6444/cacerts\": EOF"
nc: connect to 172.16.0.126 port 9345 (tcp) failed: Connection refused
Connection to 172.16.0.126 6443 port [tcp/*] succeeded!

The service hangs in activating, and after about 50 seconds worker2 shows up as NotReady in kubectl get nodes. Port 6443 is open, but that is not enough for the agent. Remove the block:

# on cp1
sudo iptables -D INPUT -p tcp --dport 9345 -s 172.16.0.83 -j REJECT
kubectl get nodes

Expected: after about 30 seconds all three nodes are Ready again.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag00
rm ~/debug/tag00/rke2-token
kubectl get pv

Expected:

namespace "tag00" deleted
No resources found

You no longer need the file to add more nodes: the token is in /etc/rancher/rke2/config.yaml on cp1.

Interview Note

  • How does an RKE2 agent register? Via the supervisor port 9345 with the shared token; only then does it get certificates and start the kubelet.
  • Where do you look first when an RKE2 node does not come up? journalctl -u rke2-server or rke2-agent, then crictl ps -a on the node.
  • Why no default StorageClass? A PVC without storageClassName should visibly get stuck instead of silently landing on a local path.

Enterprise Note

  • In production, run three servers (etcd quorum) behind a fixed registration address (DNS or VIP) in server: and tls-san, never the IP of a single server.
  • The token belongs in a secret store; new nodes get it through automation, not via scp.
  • Air gap: images as a tarball under /var/lib/rancher/rke2/agent/images/ and your own registry in registries.yaml (Day 4).
  • Maintain firewall rules for 6443, 9345, 10250, NodePorts and 8472/UDP as code; most “agent does not come up” tickets are about ports.
  • For production storage, use a distributed system (Longhorn, Ceph, cloud CSI) instead of local-path; local-path ties Pods to nodes.