Day 0: The Training Ground
Goal: Turn three empty Ubuntu VMs into the RKE2 cluster
debug, where from tomorrow on you break things and track the faults down. · Duration: approx. 90 min · Prerequisite: three VMs with Ubuntu 24.04, SSH access andsudo
Briefing
- You build an RKE2 cluster with three nodes:
cp1is the server (control plane, etcd) and also your workstation,worker1andworker2are agents. - RKE2 ships Canal as CNI, Traefik, CoreDNS and metrics-server. You add a StorageClass yourself, but deliberately not as the default.
- Today you get to know the places where the answers will be later:
the journal of
rke2-serverandrke2-agent,crictlon the node and the kubeconfig. - Talos only joins on Day 18, as a separate cluster on the VM
argo.
Management network 172.16.0.0/24
────┬──────────────────┬──────────────────┬────
cp1 .126 worker1 .228 worker2 .83
rke2-server rke2-agent rke2-agent
etcd, API server kubelet kubelet
= workstation containerd containerd
kubectl k9s helm
Drill 1: The Starting Point
The book was tested on three VMs under Proxmox: cp1 with
4 vCPU and 11 GiB RAM, the two workers with 2 vCPU and 4 GiB each. All
run Ubuntu 24.04 on amd64.
# on cp1
lsb_release -ds
nproc
free -h | head -2
dpkg --print-architecture
sudo ufw statusExpected:
Ubuntu 24.04.5 LTS
4
total used free shared buff/cache available
Mem: 11Gi 434Mi 11Gi 1.1Mi 297Mi 11Gi
amd64
Status: inactive
If ufw is active, you need rules for 6443 and 9345 (TCP,
server), 10250 and 30000–32767 (TCP, all nodes; NodePorts, from Day 4),
8472 (UDP, Canal’s VXLAN) and 9099 (TCP, Canal’s health check). The list
comes from the RKE2 docs; on a worker with ufw active,
everything worked with it (Pods, Services, NodePorts, DNS). The
challenge shows what a blocked 9345 looks like.
For the node days you go from cp1 to the workers via
SSH. Add the names and create a key if cp1 does not have
one yet:
# on cp1
sudo tee -a /etc/hosts <<'EOF'
172.16.0.126 cp1
172.16.0.228 worker1
172.16.0.83 worker2
EOF
[ -f ~/.ssh/id_ed25519 ] || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pubAppend the public key to ~/.ssh/authorized_keys on both
workers, the same way you deployed your own key. Then:
# on cp1
ssh -o StrictHostKeyChecking=accept-new worker1 hostname
ssh -o StrictHostKeyChecking=accept-new worker2 hostnameExpected:
worker1
worker2
Save your progress? Sign up for free
Drill 2: The Server Configuration
On its first start, RKE2 reads
/etc/rancher/rke2/config.yaml. Each key corresponds to an
option of rke2 server. The file must be in place
before the service starts for the first time.
# on cp1
mkdir -p ~/debug/tag00 && cd ~/debug/tag00
openssl rand -hex 32 > rke2-token
chmod 600 rke2-token
sudo mkdir -p /etc/rancher/rke2
sudo tee /etc/rancher/rke2/config.yaml >/dev/null <<EOF
node-name: cp1
node-ip: 172.16.0.126
token: $(cat rke2-token)
tls-san:
- 172.16.0.126
- cp1
write-kubeconfig-mode: "0600"
cni: canal
EOF
sudo chmod 600 /etc/rancher/rke2/config.yaml
sudo sed 's/^token: .*/token: <gekürzt>/' /etc/rancher/rke2/config.yamlExpected:
node-name: cp1
node-ip: 172.16.0.126
token: <gekürzt>
tls-san:
- 172.16.0.126
- cp1
write-kubeconfig-mode: "0600"
cni: canal
Why? The heredoc marker EOF is unquoted here so that the
shell substitutes $(cat rke2-token). token is
the shared secret of server and agents, tls-san adds the IP
and the name to the API server certificate.
(<gekürzt> just means ‘redacted’.)
Save your progress? Sign up for free
Drill 3: Start the Server and Read Its Log
# on cp1
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.37.2+rke2r1 sh -
time sudo systemctl enable --now rke2-server.serviceExpected (abridged):
[INFO] using v1.37.2+rke2r1 as release
[INFO] verifying tarball
[INFO] unpacking tarball file to /usr/local
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → ...
real 1m29.932s
systemctl only returns once etcd and the API server are
running – on the first start, including loading the images, about a
minute and a half. The log is the first place you look when it takes
longer:
# on cp1
sudo journalctl -u rke2-server -b -o cat \
| grep -E 'Starting rke2|Wrote kubeconfig|level=error' | cut -c1-88Expected (abridged):
time="2026-11-20T06:29:51Z" level=info msg="Starting rke2 v1.37.2+rke2r1 (972ca61add47ea
time="2026-11-20T06:29:56Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:
time="2026-11-20T06:29:56Z" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml
time="2026-11-20T06:30:24Z" level=error msg="Failed to process image event: failed to im
time="2026-11-20T06:31:19Z" level=error msg="Failed to process image event: failed to im
Why? Not every level=error is a fault. The
503 … runtime core not ready lines come from the first
seconds, when the supervisor is already receiving requests but etcd is
not ready yet. Failed to process image event … not found is
reported by the import of the image lists; the kubelet pulls the images
afterwards anyway. What matters is whether Pods run afterwards – Drill 4
shows that.
Tip:
journalctl -u rke2-server -ffollows the log live. Errors inconfig.yaml(unknown key, wrong indentation) show up right at the beginning.
Save your progress? Sign up for free
Drill 4: kubectl, Kubeconfig and the Three Variables
RKE2 puts kubectl, crictl and
ctr into /var/lib/rancher/rke2/bin and writes
the admin kubeconfig to /etc/rancher/rke2/rke2.yaml,
readable by root only. crictl only finds the containerd
socket with CRI_CONFIG_FILE.
# on cp1
sudo tee /etc/profile.d/rke2.sh >/dev/null <<'EOF'
export PATH=$PATH:/var/lib/rancher/rke2/bin
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
if [ "$(id -u)" = 0 ]; then export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; fi
EOF
sudo ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
mkdir -p ~/.kube
sudo install -m 600 -o "$USER" -g "$USER" /etc/rancher/rke2/rke2.yaml ~/.kube/config
kubectl get nodesExpected:
NAME STATUS ROLES AGE VERSION
cp1 NotReady control-plane,etcd 49s v1.37.2+rke2r1
NotReady is normal in the first minute: Canal has to
start before the kubelet reports the Pod network. Now the same from
containerd’s point of view. sudo -i opens a login shell and
reads /etc/profile.d/rke2.sh along the way:
# on cp1
sudo -i crictl ps -o table | head -5 | cut -c1-88Expected (abridged):
CONTAINER IMAGE CREATED STATE NAME
9b0b5d0288151 ddf8337da7bdb 40 seconds ago Running cloud-c
c568c35ccde1d 6d31c161dcb90 44 seconds ago Running kube-sc
924396c58fd99 6d31c161dcb90 44 seconds ago Running kube-co
a35a7f4fd0f0b 6d31c161dcb90 54 seconds ago Running kube-ap
Why? In RKE2 the control plane runs as static Pods that the kubelet
starts by itself. If the API server goes down, you no longer see them
with kubectl, but you still do with crictl
(Day 15).
Save your progress? Sign up for free
Drill 5: The Agents
The agent registers with the server via the supervisor port 9345, not via 6443. Over this connection it fetches certificates and configuration. First, each worker gets the token:
# on cp1
cd ~/debug/tag00
scp -q rke2-token worker1: && scp -q rke2-token worker2:# on worker1
sudo mkdir -p /etc/rancher/rke2
sudo tee /etc/rancher/rke2/config.yaml >/dev/null <<EOF
server: https://172.16.0.126:9345
node-name: worker1
node-ip: 172.16.0.228
token: $(cat ~/rke2-token)
EOF
sudo chmod 600 /etc/rancher/rke2/config.yaml
rm ~/rke2-token
curl -sfL https://get.rke2.io | \
sudo INSTALL_RKE2_VERSION=v1.37.2+rke2r1 INSTALL_RKE2_TYPE=agent sh -
sudo systemctl enable --now rke2-agent.serviceDo the same on worker2 with
node-name: worker2 and node-ip: 172.16.0.83.
Both workers also need the variables from Drill 4: create the same file
/etc/profile.d/rke2.sh there (the KUBECONFIG
line has no effect on agents and does no harm). The agent’s log:
# on worker1
sudo journalctl -u rke2-agent -b -o cat \
| grep -E 'Starting rke2|Remotedialer|Running kubelet' | cut -c1-88Expected:
time="2026-11-20T06:32:43Z" level=info msg="Starting rke2 agent v1.37.2+rke2r1 (972ca61a
time="2026-11-20T06:32:58Z" level=info msg="Remotedialer connected to proxy" url="wss://
time="2026-11-20T06:34:00Z" level=info msg="Running kubelet --alsologtostderr=false --cl
Remotedialer connected means the tunnel to the
supervisor is up. If the line is missing, the problem lies between the
agent and port 9345.
# on cp1
kubectl get nodes -o wide | cut -c1-88Expected:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL
cp1 Ready control-plane,etcd 8m49s v1.37.2+rke2r1 172.16.0.126 <none>
worker1 Ready <none> 6m v1.37.2+rke2r1 172.16.0.228 <none>
worker2 Ready <none> 4m38s v1.37.2+rke2r1 172.16.0.83 <none>
Save your progress? Sign up for free
Drill 6: What RKE2 Ships
RKE2 installs its add-on components as Helm charts via its own
controller. Each chart is an object of type HelmChart,
installed by a Job helm-install-<name>.
# on cp1
kubectl -n kube-system get helmcharts
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
RUNTIME:.status.nodeInfo.containerRuntimeVersionExpected (abridged):
NAME REPO CHART VERSION TARGETNAMESPACE BOOTSTRAP ...
rke2-canal true ...
rke2-coredns true ...
rke2-gateway-api-crd ...
rke2-metrics-server ...
rke2-runtimeclasses ...
rke2-security-responder ...
rke2-snapshot-controller ...
rke2-snapshot-controller-crd ...
rke2-traefik ...
rke2-traefik-crd ...
NAME RUNTIME
cp1 containerd://2.3.7-k3s1
worker1 containerd://2.3.7-k3s1
worker2 containerd://2.3.7-k3s1
Why? In RKE2 1.37, Traefik is the default ingress controller, and the
Gateway API CRDs come from a chart of their own. If a
helm-install-… Pod gets stuck in Error, the
reason is in its log (Day 9).
Talos: Talos ships neither an ingress controller nor metrics-server; Days 18 and 19 show what that means for troubleshooting.
Save your progress? Sign up for free
Drill 7: The Tools
You install helm, k9s and the Kyverno CLI
in an architecture-neutral way. The Kyverno CLI calls amd64
x86_64, hence the substitution in KARCH.
# on cp1
cd ~/debug/tag00
ARCH=$(dpkg --print-architecture)
curl -sSfLO https://get.helm.sh/helm-v4.3.2-linux-${ARCH}.tar.gz
tar xzf helm-v4.3.2-linux-${ARCH}.tar.gz linux-${ARCH}/helm
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
K9S=https://github.com/derailed/k9s/releases/download/v0.51.0
curl -sSfLO ${K9S}/k9s_Linux_${ARCH}.tar.gz
tar xzf k9s_Linux_${ARCH}.tar.gz k9s
sudo install -m 755 k9s /usr/local/bin/k9s
KARCH=${ARCH/amd64/x86_64}
KYV=https://github.com/kyverno/kyverno/releases/download/v1.20.0
curl -sSfLO ${KYV}/kyverno-cli_v1.20.0_linux_${KARCH}.tar.gz
tar xzf kyverno-cli_v1.20.0_linux_${KARCH}.tar.gz kyverno
sudo install -m 755 kyverno /usr/local/bin/kyverno
rm -rf linux-${ARCH} k9s kyverno *.tar.gz
helm version --short
k9s version | head -1
kyverno version | head -1Expected:
v4.3.2+g8346aaa
Version v0.51.0
Version: v1.20.0
Ubuntu 24.04 already ships jq, curl and
openssl.
Save your progress? Sign up for free
Drill 8: A StorageClass, but No Default
RKE2 has no StorageClass. You install the local-path-provisioner. Its manifest does not set the default marker, and it stays that way: on Days 3 and 10 you need the fault “PVC without a StorageClass”.
# on cp1
kubectl apply -f https://raw.githubusercontent.com/rancher/\
local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
kubectl -n local-path-storage rollout status deploy/local-path-provisioner
kubectl get storageclassExpected (abridged):
namespace/local-path-storage created
...
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
deployment "local-path-provisioner" successfully rolled out
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ...
local-path rancher.io/local-path Delete WaitForFirstConsumer ...
There is no (default) after local-path. A
small test shows whether a volume is really created:
# on cp1
cat > pvc-test.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag00}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: schreiber, namespace: tag00}
spec:
containers:
- name: schreiber
image: busybox:1.37
command: [sh, -c, 'date > /daten/hallo.txt; cat /daten/hallo.txt; sleep 3600']
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: daten}
EOF
kubectl create namespace tag00
kubectl apply -f pvc-test.yaml
kubectl -n tag00 wait pod/schreiber --for=condition=Ready --timeout=120s
kubectl -n tag00 get pod schreiber -o wide | cut -c1-72
kubectl get pv -o custom-columns=PATH:.spec.hostPath.path | cut -c1-88Expected:
namespace/tag00 created
persistentvolumeclaim/daten created
pod/schreiber created
pod/schreiber condition met
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
schreiber 1/1 Running 0 13s 10.42.1.5 worker1 <none>
PATH
/opt/local-path-provisioner/pvc-6626be9a-953e-42e6-98d6-8c30e89d4f60_tag00_daten
The directory lives on the node shown in the NODE
column, which may be worker2 in your cluster:
# on worker1
sudo cat /opt/local-path-provisioner/*_tag00_daten/hallo.txtExpected:
Fri Nov 20 06:37:38 UTC 2026
Tip: Take a snapshot of all three VMs now. From Day 3 on you break things; every day cleans up after itself, but a snapshot is the fastest way back.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show name, kubelet version and container runtime of all nodes. Target time: 30 s
- From
cp1, show the Flannel containers onworker2. Target time: 30 s - Count the
level=errorlines ofrke2-serversince the last boot. Target time: 30 s - Check whether the API server is ready and show only the last line. Target time: 20 s
Solution
# on cp1
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
VERSION:.status.nodeInfo.kubeletVersion,\
RUNTIME:.status.nodeInfo.containerRuntimeVersion
ssh worker2 sudo -i crictl ps --name flannel | cut -c1-88
sudo journalctl -u rke2-server -b -o cat | grep -c level=error
kubectl get --raw='/readyz?verbose' | tail -1Expected (abridged):
NAME VERSION RUNTIME
cp1 v1.37.2+rke2r1 containerd://2.3.7-k3s1
worker1 v1.37.2+rke2r1 containerd://2.3.7-k3s1
worker2 v1.37.2+rke2r1 containerd://2.3.7-k3s1
CONTAINER IMAGE CREATED STATE NAME
d27d5570b3822 4f23b420044ae 5 minutes ago Running kube-fla
6
readyz check passed
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl get nodesExpected: three nodes, all Ready,
version v1.37.2+rke2r1.
# on cp1
kubectl get storageclass -o nameExpected:
storageclass.storage.k8s.io/local-path
# on cp1
kubectl get storageclass local-path \
-o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}'Expected: no output – local-path is not
the default.
# on cp1
ssh worker1 sudo -i crictl version | tail -2Expected:
RuntimeVersion: v2.3.7-k3s1
RuntimeApiVersion: v1
Challenge
On cp1, block port 9345 for worker2 only,
restart rke2-agent there and find the message that reveals
the reason. Remove the block afterwards.
Hint: The agent talks to the supervisor through a local load balancer on port 6444 – so the message does not only mention 9345.
Solution
# on cp1
sudo iptables -I INPUT -p tcp --dport 9345 -s 172.16.0.83 -j REJECT# on worker2
sudo systemctl restart rke2-agent --no-block
systemctl is-active rke2-agent
sudo journalctl -u rke2-agent --since -1min -o cat | grep level=error | tail -2
nc -zv -w3 172.16.0.126 9345
nc -zv -w3 172.16.0.126 6443Expected (lines wrapped):
activating
time="2026-11-20T06:38:32Z" level=error msg="Failed to validate connection to
cluster at https://172.16.0.126:9345: failed to get CA certs:
Get \"https://127.0.0.1:6444/cacerts\": EOF"
nc: connect to 172.16.0.126 port 9345 (tcp) failed: Connection refused
Connection to 172.16.0.126 6443 port [tcp/*] succeeded!
The service hangs in activating, and after about 50
seconds worker2 shows up as NotReady in
kubectl get nodes. Port 6443 is open, but that is not
enough for the agent. Remove the block:
# on cp1
sudo iptables -D INPUT -p tcp --dport 9345 -s 172.16.0.83 -j REJECT
kubectl get nodesExpected: after about 30 seconds all three nodes are
Ready again.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag00
rm ~/debug/tag00/rke2-token
kubectl get pvExpected:
namespace "tag00" deleted
No resources found
You no longer need the file to add more nodes: the token is in
/etc/rancher/rke2/config.yaml on cp1.
Interview Note
- How does an RKE2 agent register? Via the supervisor port 9345 with the shared token; only then does it get certificates and start the kubelet.
- Where do you look first when an RKE2 node does not come up?
journalctl -u rke2-serverorrke2-agent, thencrictl ps -aon the node. - Why no default StorageClass? A PVC without
storageClassNameshould visibly get stuck instead of silently landing on a local path.
Enterprise Note
- In production, run three servers (etcd quorum) behind a fixed
registration address (DNS or VIP) in
server:andtls-san, never the IP of a single server. - The token belongs in a secret store; new nodes get it through
automation, not via
scp. - Air gap: images as a tarball under
/var/lib/rancher/rke2/agent/images/and your own registry inregistries.yaml(Day 4). - Maintain firewall rules for 6443, 9345, 10250, NodePorts and 8472/UDP as code; most “agent does not come up” tickets are about ports.
- For production storage, use a distributed system (Longhorn, Ceph, cloud CSI) instead of local-path; local-path ties Pods to nodes.