Skip to content
Hands-on IT IT Drills

Day 12: A Three-Node Cluster with RKE2

Goal: Start three VMs, build an RKE2 cluster on them with one server and two agents, control it from the Mac with kubectl and prepare it with storage and a Gateway for the move on Day 13. · Duration: approx. 90 min · Prerequisite: Day 11

Briefing

  • RKE2 is a Kubernetes distribution from Rancher (SUSE). A single binary brings containerd, kubelet, etcd and the control plane and runs as a systemd service.
  • A server carries the control plane and etcd, an agent only your Pods. Agents register with the server using a join token (port 9345).
  • Unlike kind, the nodes are real machines: they fail, restart and have their own disks. That is exactly what you practice from today on.
  • RKE2 ships no storage for PVCs. That is provided by the local-path-provisioner. The Gateway is once again Envoy Gateway, as on Day 11.
 Mac                         Multipass VMs (Ubuntu 24.04)
 ┌──────────────────┐        ┌─────────────────────┐
 │ ~/k8s-lab/       │ :6443  │ rke2-server         │ control plane, etcd
 │ kubectl, helm ───┼───────►│ rke2-server.service │
 │ rke2.yaml        │        └──────────▲──────────┘
 │                  │                   │ :9345 join token
 │ curl :30080 ─────┼──────► ┌──────────┴──────────┐ ┌─────────────────────┐
 └──────────────────┘        │ rke2-agent1         │ │ rke2-agent2         │
                             │ rke2-agent.service  │ │ rke2-agent.service  │
                             └─────────────────────┘ └─────────────────────┘

Drill 1: Three VMs

Together, the VMs need 10 GB of RAM. Stop the kind lab from Part 1; it no longer appears in this part. Multipass is available at multipass.run or via brew install --cask multipass.

colima stop
multipass launch 24.04 --name rke2-server --cpus 2 --memory 4G --disk 30G
multipass launch 24.04 --name rke2-agent1 --cpus 2 --memory 3G --disk 30G
multipass launch 24.04 --name rke2-agent2 --cpus 2 --memory 3G --disk 30G
multipass list

Expected (abridged, names, IPs and times will differ on your machine):

Launched: rke2-server
Launched: rke2-agent1
Launched: rke2-agent2
Name                    State             IPv4             Image
rke2-agent1             Running           …                Ubuntu 24.04 LTS
rke2-agent2             Running           …                Ubuntu 24.04 LTS
rke2-server             Running           …                Ubuntu 24.04 LTS

Any other Ubuntu 24.04 VM with SSH and sudo works too. Then ssh replaces the commands multipass shell and multipass exec, and scp replaces multipass transfer.

Save your progress? Sign up for free

Drill 2: Addresses and Day Folder

You need the IP addresses all the time. Read them from the IPv4 column of multipass list (Drill 1) and enter them into a new file ~/k8s-lab/setup/rke2.env. The same file switches kubectl to the new cluster.

# setup/rke2.env – Part 2: addresses of the VMs and kubeconfig
export SERVER=192.168.64.10   # IPv4 of rke2-server, different on your machine
export AGENT1=192.168.64.11   # IPv4 of rke2-agent1
export AGENT2=192.168.64.12   # IPv4 of rke2-agent2
export KUBECONFIG=$HOME/k8s-lab/rke2.yaml
source ~/k8s-lab/setup/env.sh
source "$LAB/setup/rke2.env"
echo "$SERVER $AGENT1 $AGENT2"
mkdir -p "$LAB/tag12" && cd "$LAB/tag12"
cp -R "$LAB/code/tag12/." .
ls rke2

Expected (abridged):

… … …
config-agent.yaml
config-server.yaml

From now on, every day starts with source ~/k8s-lab/setup/env.sh and source ~/k8s-lab/setup/rke2.env.

Save your progress? Sign up for free

Drill 3: The Server

The configuration must already be in /etc/rancher/rke2/config.yaml before the installation. The template contains the placeholder SERVER_IP.

# /etc/rancher/rke2/config.yaml on rke2-server – state at end of Day 12
# Template: Drill 3 replaces SERVER_IP with the real address via sed.
write-kubeconfig-mode: "0600"
tls-san:
  - SERVER_IP
# Traefik (default since v1.36) off: the Gateway is Envoy, as on Day 11.
ingress-controller: none

tls-san writes the IP into the API server’s certificate; otherwise kubectl on the Mac rejects the connection. Replace, copy to the server, install:

sed "s/SERVER_IP/$SERVER/" rke2/config-server.yaml > config-server.yaml
multipass transfer config-server.yaml rke2-server:/tmp/config.yaml
multipass exec rke2-server -- sudo install -D -m 600 /tmp/config.yaml \
  /etc/rancher/rke2/config.yaml
multipass exec rke2-server -- bash -c \
  'curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.36.5+rke2r1 sh -'
multipass exec rke2-server -- sudo systemctl enable --now rke2-server

Expected (abridged):

[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/…
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → …

systemctl enable --now only returns once the server is running. That takes about a minute and a half.

Warning: The version is fixed in the command. Without INSTALL_RKE2_VERSION, the installer takes the stable channel, and the upgrade on Day 14 no longer fits.

Save your progress? Sign up for free

Drill 4: The Server from the Inside

Go to the server and become root. RKE2 puts kubectl and crictl under /var/lib/rancher/rke2/bin, and the kubeconfig under /etc/rancher/rke2/rke2.yaml (readable only by root).

multipass shell rke2-server

In the VM:

sudo -i
export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
rke2 --version
kubectl get nodes
kubectl get pods -n kube-system

Expected (abridged):

rke2 version v1.36.5+rke2r1 (e1d699d2c3a3d3c1f3247c4f9eb140f20d369776)
go version go1.26.8-X:boringcrypto
NAME          STATUS   ROLES                AGE   VERSION
rke2-server   Ready    control-plane,etcd   91s   v1.36.5+rke2r1
NAME                                                   READY   STATUS      RESTARTS
cloud-controller-manager-rke2-server                   1/1     Running     0
etcd-rke2-server                                       1/1     Running     0
helm-install-rke2-canal-tc86x                          0/1     Completed   0
helm-install-rke2-coredns-v7xs9                        0/1     Completed   0
…
kube-apiserver-rke2-server                             1/1     Running     0
kube-controller-manager-rke2-server                    1/1     Running     0
kube-proxy-rke2-server                                 1/1     Running     0
kube-scheduler-rke2-server                             1/1     Running     0
rke2-canal-r8rmh                                       2/2     Running     0
rke2-coredns-rke2-coredns-747b86f67b-8jtpf             1/1     Running     0
…

Right after the start, the node is briefly NotReady until the Pod network (canal) is running. Two kinds of Pods stand out. The control plane runs as static Pods: the kubelet starts them from files, not the API server. RKE2 installs the add-on services as Helm charts via the helm-install-… Jobs.

ls /var/lib/rancher/rke2/agent/pod-manifests
kubectl get helmcharts -n kube-system
cat /var/lib/rancher/rke2/server/manifests/rke2-traefik.yaml

Expected:

cloud-controller-manager.yaml  kube-apiserver.yaml           kube-proxy.yaml
etcd.yaml                      kube-controller-manager.yaml  kube-scheduler.yaml
NAME                           REPO   CHART   …   BOOTSTRAP   FAILED
rke2-canal                                        true        False
rke2-coredns                                      true        False
rke2-metrics-server                                           False
rke2-runtimeclasses                                           False
rke2-snapshot-controller                                      False
rke2-snapshot-controller-crd                                  False
# disabled by configuration

Traefik is there, but empty: ingress-controller: none takes effect. Finally, the containers directly in containerd. crictl needs the path to RKE2’s socket:

export CONTAINER_RUNTIME_ENDPOINT=unix:///run/k3s/containerd/containerd.sock
crictl ps --name etcd
exit
exit

The first exit leaves the root shell, the second the VM.

Expected (abridged):

… level=warning msg="Config \"/etc/crictl.yaml\" does not exist, trying next: …
CONTAINER           IMAGE               CREATED             STATE     NAME   ATTEMPT
a366598c30332       20d464133993d       15 minutes ago      Running   etcd   0

The warning is harmless: crictl first looks for its configuration file and then takes the socket from the variable.

Save your progress? Sign up for free

Drill 5: kubectl from the Mac

The server’s kubeconfig points to 127.0.0.1 and calls everything default. Fetch it to the Mac, insert the server IP and name the cluster, user and context rke2:

multipass exec rke2-server -- sudo cat /etc/rancher/rke2/rke2.yaml \
  | sed -e "s/127.0.0.1/$SERVER/" -e 's/: default$/: rke2/' > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
kubectl config get-contexts
kubectl get nodes -o wide

Expected (abridged):

CURRENT   NAME   CLUSTER   AUTHINFO   NAMESPACE
*         rke2   rke2      rke2
NAME          STATUS   ROLES                AGE     VERSION          INTERNAL-IP   …
rke2-server   Ready    control-plane,etcd   4m53s   v1.36.5+rke2r1   …             …

Security: The file contains the cluster’s admin certificate. It belongs in ~/k8s-lab with permissions 600, never in a Git repository.

Save your progress? Sign up for free

Drill 6: Two Agents Join

An agent needs the address of the server and the join token. The token travels from the server into a file on the Mac and from there to the agents, without you seeing it:

multipass exec rke2-server -- sudo cat /var/lib/rancher/rke2/server/node-token \
  > "$LAB/rke2-token"
chmod 600 "$LAB/rke2-token"
TOKEN="$LAB/rke2-token"
cat rke2/config-agent.yaml
sed "s/SERVER_IP/$SERVER/" rke2/config-agent.yaml > config-agent.yaml

Expected:

# /etc/rancher/rke2/config.yaml auf rke2-agent1 und rke2-agent2 – Stand Ende Tag 12
server: https://SERVER_IP:9345
# Das Join-Token liegt in einer Datei, nie im Klartext in der Konfig.
token-file: /etc/rancher/rke2/token

Now the same for both agents. INSTALL_RKE2_TYPE=agent installs the service rke2-agent instead of rke2-server:

for vm in rke2-agent1 rke2-agent2; do
  multipass transfer "$TOKEN" config-agent.yaml "$vm":/tmp/
  multipass exec "$vm" -- sudo install -D -m 600 /tmp/rke2-token \
    /etc/rancher/rke2/token
  multipass exec "$vm" -- sudo install -m 600 /tmp/config-agent.yaml \
    /etc/rancher/rke2/config.yaml
  multipass exec "$vm" -- rm /tmp/rke2-token
  multipass exec "$vm" -- bash -c 'curl -sfL https://get.rke2.io \
    | sudo INSTALL_RKE2_VERSION=v1.36.5+rke2r1 INSTALL_RKE2_TYPE=agent sh -'
  multipass exec "$vm" -- sudo systemctl enable --now rke2-agent
done
kubectl get nodes

Expected (abridged):

NAME          STATUS     ROLES                AGE    VERSION
rke2-agent1   NotReady   <none>               4s     v1.36.5+rke2r1
rke2-agent2   NotReady   <none>               2s     v1.36.5+rke2r1
rke2-server   Ready      control-plane,etcd   3m5s   v1.36.5+rke2r1

The agents first load the Pod network. Wait until all three are ready:

kubectl wait --for=condition=Ready node --all --timeout=300s
kubectl get nodes

Expected (abridged):

node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met
NAME          STATUS   ROLES                AGE     VERSION
rke2-agent1   Ready    <none>               91s     v1.36.5+rke2r1
rke2-agent2   Ready    <none>               89s     v1.36.5+rke2r1
rke2-server   Ready    control-plane,etcd   4m32s   v1.36.5+rke2r1

ROLES <none> means: a pure worker. The server also accepts normal Pods, because it has no taint.

Save your progress? Sign up for free

Drill 7: Storage with local-path

In kind there was the StorageClass standard. Here there is none:

kubectl get sc

Expected:

No resources found

The local-path-provisioner creates a directory on one node for every PVC. Install it and make it the default StorageClass:

kubectl apply -f https://raw.githubusercontent.com/rancher/\
local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=true
kubectl -n local-path-storage rollout status deploy/local-path-provisioner
kubectl get sc

Expected (abridged):

…
storageclass.storage.k8s.io/local-path annotated
deployment "local-path-provisioner" successfully rolled out
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer

Try it with a PVC and a Pod:

cat > probe.yaml <<'EOF'
# probe.yaml – test for local-path (Day 12, delete afterwards)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: probe
  namespace: default
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 100Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: probe
  namespace: default
spec:
  containers:
    - name: probe
      image: busybox:1.37
      command: ["sh", "-c", "echo hallo > /d/f && sleep 3600"]
      volumeMounts:
        - {name: d, mountPath: /d}
  volumes:
    - name: d
      persistentVolumeClaim:
        claimName: probe
EOF
kubectl apply -f probe.yaml
kubectl -n default wait --for=condition=Ready pod/probe --timeout=120s
kubectl -n default get pvc probe
kubectl -n default get pod probe -o wide

Expected (abridged):

persistentvolumeclaim/probe created
pod/probe created
pod/probe condition met
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES
probe   Bound    pvc-bd543f6e-bc8b-416d-a605-78fcb2d9f9d7   100Mi      RWO
NAME    READY   STATUS    RESTARTS   AGE   IP          NODE          …
probe   1/1     Running   0          13s   10.42.2.5   rke2-agent2   …

WaitForFirstConsumer means: the PVC stays Pending until a Pod needs it. Only then is it decided on which node the directory is created. Look on that node (on your machine possibly rke2-agent1):

multipass exec rke2-agent2 -- sudo ls /opt/local-path-provisioner
multipass exec rke2-agent2 -- sh -c 'sudo cat /opt/local-path-provisioner/*/f'
kubectl delete -f probe.yaml

Expected (abridged):

pvc-bd543f6e-bc8b-416d-a605-78fcb2d9f9d7_default_probe
hallo
persistentvolumeclaim "probe" deleted from default namespace
pod "probe" deleted from default namespace

The data lives on the disk of this one node. What that means for Postgres when the node is under maintenance, you will see on Day 14.

Save your progress? Sign up for free

Drill 8: Envoy Gateway

The Gateway comes via Helm as on Day 11, plus the platform file k8s/gateway.yaml from Day 11 with the NodePort 30080:

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.3 \
  -n envoy-gateway-system --create-namespace --wait | grep -E '^(NAME|STATUS):'
kubectl apply -f "$LAB/code/tag11/notes/k8s/gateway.yaml"
kubectl get gatewayclass

Expected:

NAME: eg
STATUS: deployed
envoyproxy.gateway.envoyproxy.io/nodeport created
gatewayclass.gateway.networking.k8s.io/eg created
NAME   CONTROLLER                                      ACCEPTED   AGE
eg     gateway.envoyproxy.io/gatewayclass-controller   True       1s

A test Gateway without a route shows whether the port answers on all nodes:

cat > probe-gw.yaml <<'EOF'
# probe-gw.yaml – test for the NodePort (Day 12, delete afterwards)
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: probe
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - {name: http, protocol: HTTP, port: 80}
EOF
kubectl apply -f probe-gw.yaml
kubectl -n default wait --for=condition=Programmed gateway/probe --timeout=120s
curl -s -o /dev/null -w '%{http_code}\n' "http://$SERVER:30080/"
curl -s -o /dev/null -w '%{http_code}\n' "http://$AGENT1:30080/"
curl -s -o /dev/null -w '%{http_code}\n' "http://$AGENT2:30080/"
kubectl delete -f probe-gw.yaml

Expected:

gateway.gateway.networking.k8s.io/probe created
gateway.gateway.networking.k8s.io/probe condition met
404
404
404
gateway.gateway.networking.k8s.io "probe" deleted from default namespace

404 is correct: Envoy answers but doesn’t know any route yet. That all three nodes answer, even though Envoy only runs on one of them, is thanks to externalTrafficPolicy: Cluster from Day 11.

Save your progress? Sign up for free

Checkpoint

kubectl get nodes

Expected: three nodes, all Ready, version v1.36.5+rke2r1.

NAME          STATUS   ROLES                AGE   VERSION
rke2-agent1   Ready    <none>               12m   v1.36.5+rke2r1
rke2-agent2   Ready    <none>               12m   v1.36.5+rke2r1
rke2-server   Ready    control-plane,etcd   15m   v1.36.5+rke2r1
kubectl get sc

Expected: local-path with (default).

NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer
kubectl get gatewayclass eg

Expected: ACCEPTED True.

NAME   CONTROLLER                                      ACCEPTED   AGE
eg     gateway.envoyproxy.io/gatewayclass-controller   True       5m
kubectl get ns notes

Expected: the namespace doesn’t exist here yet; it only comes on Day 13.

Error from server (NotFound): namespaces "notes" not found

Challenge

What does the cluster see when an agent fails? Stop the service rke2-agent on rke2-agent1. Find out after how many seconds the node becomes NotReady and which taints it then carries. Start the service again afterwards.

Hint: Restarting the VM isn’t enough: it’s over faster than the cluster gets suspicious. kubectl wait can also wait for Ready=Unknown.

Solution

multipass exec rke2-agent1 -- sudo systemctl stop rke2-agent
time kubectl wait --for=condition=Ready=Unknown node/rke2-agent1 --timeout=180s
kubectl get nodes
kubectl get node rke2-agent1 -o jsonpath='{.spec.taints}'; echo
multipass exec rke2-agent1 -- sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent1 --timeout=180s

Expected (abridged):

node/rke2-agent1 condition met
real    0m42.3s
NAME          STATUS     ROLES                AGE     VERSION
rke2-agent1   NotReady   <none>               8m42s   v1.36.5+rke2r1
rke2-agent2   Ready      <none>               8m40s   v1.36.5+rke2r1
rke2-server   Ready      control-plane,etcd   11m     v1.36.5+rke2r1
[{"effect":"NoSchedule","key":"node.kubernetes.io/unreachable",…},
 {"effect":"NoExecute","key":"node.kubernetes.io/unreachable",…}]
node/rke2-agent1 condition met

After a good 40 seconds without a sign of life from the kubelet, the controller sets the node to Unknown. kubectl describe node names the reason: Kubelet stopped posting node status. The taint NoSchedule keeps new Pods away. NoExecute evicts running Pods after another five minutes; that is how long they tolerate it by default. After the start, the node is immediately Ready again.

Save your progress? Sign up for free

Enterprise Note

  • High availability: In production, three (or five) servers run so that etcd keeps a quorum. Agents and additional servers then point to a fixed address in front of the servers (load balancer or DNS), not to a single server.
  • Hardening: profile: cis in config.yaml turns on the CIS requirements (among other things, Pod Security restricted cluster-wide). Beforehand, a user etcd and some kernel parameters must be set up on every host; see docs.rke2.io.
  • Without internet: For data centers without access there are image archives and an “air-gapped” installation; the images then come from your own registry (Day 13).
  • Storage: local-path ties data to one node. For real resilience, Longhorn or external storage (Ceph, NFS filer, SAN with CSI driver) replicates the volumes.
  • Many clusters: Rancher Manager manages RKE2 clusters centrally, with permissions, upgrades and GitOps via Fleet.