Day 12: A Three-Node Cluster with RKE2
Goal: Start three VMs, build an RKE2 cluster on them with one server and two agents, control it from the Mac with
kubectland prepare it with storage and a Gateway for the move on Day 13. · Duration: approx. 90 min · Prerequisite: Day 11
Briefing
- RKE2 is a Kubernetes distribution from Rancher (SUSE). A single binary brings containerd, kubelet, etcd and the control plane and runs as a systemd service.
- A server carries the control plane and etcd, an agent only your Pods. Agents register with the server using a join token (port 9345).
- Unlike kind, the nodes are real machines: they fail, restart and have their own disks. That is exactly what you practice from today on.
- RKE2 ships no storage for PVCs. That is provided by the local-path-provisioner. The Gateway is once again Envoy Gateway, as on Day 11.
Mac Multipass VMs (Ubuntu 24.04)
┌──────────────────┐ ┌─────────────────────┐
│ ~/k8s-lab/ │ :6443 │ rke2-server │ control plane, etcd
│ kubectl, helm ───┼───────►│ rke2-server.service │
│ rke2.yaml │ └──────────▲──────────┘
│ │ │ :9345 join token
│ curl :30080 ─────┼──────► ┌──────────┴──────────┐ ┌─────────────────────┐
└──────────────────┘ │ rke2-agent1 │ │ rke2-agent2 │
│ rke2-agent.service │ │ rke2-agent.service │
└─────────────────────┘ └─────────────────────┘
Drill 1: Three VMs
Together, the VMs need 10 GB of RAM. Stop the kind lab from Part 1;
it no longer appears in this part. Multipass is available at
multipass.run or via
brew install --cask multipass.
colima stop
multipass launch 24.04 --name rke2-server --cpus 2 --memory 4G --disk 30G
multipass launch 24.04 --name rke2-agent1 --cpus 2 --memory 3G --disk 30G
multipass launch 24.04 --name rke2-agent2 --cpus 2 --memory 3G --disk 30G
multipass listExpected (abridged, names, IPs and times will differ on your machine):
Launched: rke2-server
Launched: rke2-agent1
Launched: rke2-agent2
Name State IPv4 Image
rke2-agent1 Running … Ubuntu 24.04 LTS
rke2-agent2 Running … Ubuntu 24.04 LTS
rke2-server Running … Ubuntu 24.04 LTS
Any other Ubuntu 24.04 VM with SSH and sudo works too.
Then ssh replaces the commands multipass shell
and multipass exec, and scp replaces
multipass transfer.
Save your progress? Sign up for free
Drill 2: Addresses and Day Folder
You need the IP addresses all the time. Read them from the
IPv4 column of multipass list (Drill 1) and
enter them into a new file ~/k8s-lab/setup/rke2.env. The
same file switches kubectl to the new cluster.
# setup/rke2.env – Part 2: addresses of the VMs and kubeconfig
export SERVER=192.168.64.10 # IPv4 of rke2-server, different on your machine
export AGENT1=192.168.64.11 # IPv4 of rke2-agent1
export AGENT2=192.168.64.12 # IPv4 of rke2-agent2
export KUBECONFIG=$HOME/k8s-lab/rke2.yamlsource ~/k8s-lab/setup/env.sh
source "$LAB/setup/rke2.env"
echo "$SERVER $AGENT1 $AGENT2"
mkdir -p "$LAB/tag12" && cd "$LAB/tag12"
cp -R "$LAB/code/tag12/." .
ls rke2Expected (abridged):
… … …
config-agent.yaml
config-server.yaml
From now on, every day starts with
source ~/k8s-lab/setup/env.sh and
source ~/k8s-lab/setup/rke2.env.
Save your progress? Sign up for free
Drill 3: The Server
The configuration must already be in
/etc/rancher/rke2/config.yaml before the installation. The
template contains the placeholder SERVER_IP.
# /etc/rancher/rke2/config.yaml on rke2-server – state at end of Day 12
# Template: Drill 3 replaces SERVER_IP with the real address via sed.
write-kubeconfig-mode: "0600"
tls-san:
- SERVER_IP
# Traefik (default since v1.36) off: the Gateway is Envoy, as on Day 11.
ingress-controller: nonetls-san writes the IP into the API server’s certificate;
otherwise kubectl on the Mac rejects the connection.
Replace, copy to the server, install:
sed "s/SERVER_IP/$SERVER/" rke2/config-server.yaml > config-server.yaml
multipass transfer config-server.yaml rke2-server:/tmp/config.yaml
multipass exec rke2-server -- sudo install -D -m 600 /tmp/config.yaml \
/etc/rancher/rke2/config.yaml
multipass exec rke2-server -- bash -c \
'curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.36.5+rke2r1 sh -'
multipass exec rke2-server -- sudo systemctl enable --now rke2-serverExpected (abridged):
[INFO] downloading tarball at https://github.com/rancher/rke2/releases/download/…
[INFO] verifying tarball
[INFO] unpacking tarball file to /usr/local
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → …
systemctl enable --now only returns once the server is
running. That takes about a minute and a half.
Warning: The version is fixed in the command. Without
INSTALL_RKE2_VERSION, the installer takes the stable channel, and the upgrade on Day 14 no longer fits.
Save your progress? Sign up for free
Drill 4: The Server from the Inside
Go to the server and become root. RKE2 puts kubectl and
crictl under /var/lib/rancher/rke2/bin, and
the kubeconfig under /etc/rancher/rke2/rke2.yaml (readable
only by root).
multipass shell rke2-serverIn the VM:
sudo -i
export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
rke2 --version
kubectl get nodes
kubectl get pods -n kube-systemExpected (abridged):
rke2 version v1.36.5+rke2r1 (e1d699d2c3a3d3c1f3247c4f9eb140f20d369776)
go version go1.26.8-X:boringcrypto
NAME STATUS ROLES AGE VERSION
rke2-server Ready control-plane,etcd 91s v1.36.5+rke2r1
NAME READY STATUS RESTARTS
cloud-controller-manager-rke2-server 1/1 Running 0
etcd-rke2-server 1/1 Running 0
helm-install-rke2-canal-tc86x 0/1 Completed 0
helm-install-rke2-coredns-v7xs9 0/1 Completed 0
…
kube-apiserver-rke2-server 1/1 Running 0
kube-controller-manager-rke2-server 1/1 Running 0
kube-proxy-rke2-server 1/1 Running 0
kube-scheduler-rke2-server 1/1 Running 0
rke2-canal-r8rmh 2/2 Running 0
rke2-coredns-rke2-coredns-747b86f67b-8jtpf 1/1 Running 0
…
Right after the start, the node is briefly NotReady
until the Pod network (canal) is running. Two kinds of Pods stand out.
The control plane runs as static Pods: the kubelet
starts them from files, not the API server. RKE2 installs the add-on
services as Helm charts via the
helm-install-… Jobs.
ls /var/lib/rancher/rke2/agent/pod-manifests
kubectl get helmcharts -n kube-system
cat /var/lib/rancher/rke2/server/manifests/rke2-traefik.yamlExpected:
cloud-controller-manager.yaml kube-apiserver.yaml kube-proxy.yaml
etcd.yaml kube-controller-manager.yaml kube-scheduler.yaml
NAME REPO CHART … BOOTSTRAP FAILED
rke2-canal true False
rke2-coredns true False
rke2-metrics-server False
rke2-runtimeclasses False
rke2-snapshot-controller False
rke2-snapshot-controller-crd False
# disabled by configuration
Traefik is there, but empty: ingress-controller: none
takes effect. Finally, the containers directly in containerd.
crictl needs the path to RKE2’s socket:
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/k3s/containerd/containerd.sock
crictl ps --name etcd
exit
exitThe first exit leaves the root shell, the second the
VM.
Expected (abridged):
… level=warning msg="Config \"/etc/crictl.yaml\" does not exist, trying next: …
CONTAINER IMAGE CREATED STATE NAME ATTEMPT
a366598c30332 20d464133993d 15 minutes ago Running etcd 0
The warning is harmless: crictl first looks for its configuration file and then takes the socket from the variable.
Save your progress? Sign up for free
Drill 5: kubectl from the
Mac
The server’s kubeconfig points to 127.0.0.1 and calls
everything default. Fetch it to the Mac, insert the server
IP and name the cluster, user and context rke2:
multipass exec rke2-server -- sudo cat /etc/rancher/rke2/rke2.yaml \
| sed -e "s/127.0.0.1/$SERVER/" -e 's/: default$/: rke2/' > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
kubectl config get-contexts
kubectl get nodes -o wideExpected (abridged):
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* rke2 rke2 rke2
NAME STATUS ROLES AGE VERSION INTERNAL-IP …
rke2-server Ready control-plane,etcd 4m53s v1.36.5+rke2r1 … …
Security: The file contains the cluster’s admin certificate. It belongs in
~/k8s-labwith permissions600, never in a Git repository.
Save your progress? Sign up for free
Drill 6: Two Agents Join
An agent needs the address of the server and the join token. The token travels from the server into a file on the Mac and from there to the agents, without you seeing it:
multipass exec rke2-server -- sudo cat /var/lib/rancher/rke2/server/node-token \
> "$LAB/rke2-token"
chmod 600 "$LAB/rke2-token"
TOKEN="$LAB/rke2-token"
cat rke2/config-agent.yaml
sed "s/SERVER_IP/$SERVER/" rke2/config-agent.yaml > config-agent.yamlExpected:
# /etc/rancher/rke2/config.yaml auf rke2-agent1 und rke2-agent2 – Stand Ende Tag 12
server: https://SERVER_IP:9345
# Das Join-Token liegt in einer Datei, nie im Klartext in der Konfig.
token-file: /etc/rancher/rke2/tokenNow the same for both agents. INSTALL_RKE2_TYPE=agent
installs the service rke2-agent instead of
rke2-server:
for vm in rke2-agent1 rke2-agent2; do
multipass transfer "$TOKEN" config-agent.yaml "$vm":/tmp/
multipass exec "$vm" -- sudo install -D -m 600 /tmp/rke2-token \
/etc/rancher/rke2/token
multipass exec "$vm" -- sudo install -m 600 /tmp/config-agent.yaml \
/etc/rancher/rke2/config.yaml
multipass exec "$vm" -- rm /tmp/rke2-token
multipass exec "$vm" -- bash -c 'curl -sfL https://get.rke2.io \
| sudo INSTALL_RKE2_VERSION=v1.36.5+rke2r1 INSTALL_RKE2_TYPE=agent sh -'
multipass exec "$vm" -- sudo systemctl enable --now rke2-agent
done
kubectl get nodesExpected (abridged):
NAME STATUS ROLES AGE VERSION
rke2-agent1 NotReady <none> 4s v1.36.5+rke2r1
rke2-agent2 NotReady <none> 2s v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 3m5s v1.36.5+rke2r1
The agents first load the Pod network. Wait until all three are ready:
kubectl wait --for=condition=Ready node --all --timeout=300s
kubectl get nodesExpected (abridged):
node/rke2-agent1 condition met
node/rke2-agent2 condition met
node/rke2-server condition met
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 91s v1.36.5+rke2r1
rke2-agent2 Ready <none> 89s v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 4m32s v1.36.5+rke2r1
ROLES <none> means: a pure worker. The server also
accepts normal Pods, because it has no taint.
Save your progress? Sign up for free
Drill 7: Storage with local-path
In kind there was the StorageClass standard. Here there
is none:
kubectl get scExpected:
No resources found
The local-path-provisioner creates a directory on one node for every PVC. Install it and make it the default StorageClass:
kubectl apply -f https://raw.githubusercontent.com/rancher/\
local-path-provisioner/v0.0.37/deploy/local-path-storage.yaml
kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=true
kubectl -n local-path-storage rollout status deploy/local-path-provisioner
kubectl get scExpected (abridged):
…
storageclass.storage.k8s.io/local-path annotated
deployment "local-path-provisioner" successfully rolled out
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer
Try it with a PVC and a Pod:
cat > probe.yaml <<'EOF'
# probe.yaml – test for local-path (Day 12, delete afterwards)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: probe
namespace: default
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 100Mi
---
apiVersion: v1
kind: Pod
metadata:
name: probe
namespace: default
spec:
containers:
- name: probe
image: busybox:1.37
command: ["sh", "-c", "echo hallo > /d/f && sleep 3600"]
volumeMounts:
- {name: d, mountPath: /d}
volumes:
- name: d
persistentVolumeClaim:
claimName: probe
EOF
kubectl apply -f probe.yaml
kubectl -n default wait --for=condition=Ready pod/probe --timeout=120s
kubectl -n default get pvc probe
kubectl -n default get pod probe -o wideExpected (abridged):
persistentvolumeclaim/probe created
pod/probe created
pod/probe condition met
NAME STATUS VOLUME CAPACITY ACCESS MODES
probe Bound pvc-bd543f6e-bc8b-416d-a605-78fcb2d9f9d7 100Mi RWO
NAME READY STATUS RESTARTS AGE IP NODE …
probe 1/1 Running 0 13s 10.42.2.5 rke2-agent2 …
WaitForFirstConsumer means: the PVC stays
Pending until a Pod needs it. Only then is it decided on
which node the directory is created. Look on that node (on your machine
possibly rke2-agent1):
multipass exec rke2-agent2 -- sudo ls /opt/local-path-provisioner
multipass exec rke2-agent2 -- sh -c 'sudo cat /opt/local-path-provisioner/*/f'
kubectl delete -f probe.yamlExpected (abridged):
pvc-bd543f6e-bc8b-416d-a605-78fcb2d9f9d7_default_probe
hallo
persistentvolumeclaim "probe" deleted from default namespace
pod "probe" deleted from default namespace
The data lives on the disk of this one node. What that means for Postgres when the node is under maintenance, you will see on Day 14.
Save your progress? Sign up for free
Drill 8: Envoy Gateway
The Gateway comes via Helm as on Day 11, plus the platform file
k8s/gateway.yaml from Day 11 with the NodePort 30080:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.3 \
-n envoy-gateway-system --create-namespace --wait | grep -E '^(NAME|STATUS):'
kubectl apply -f "$LAB/code/tag11/notes/k8s/gateway.yaml"
kubectl get gatewayclassExpected:
NAME: eg
STATUS: deployed
envoyproxy.gateway.envoyproxy.io/nodeport created
gatewayclass.gateway.networking.k8s.io/eg created
NAME CONTROLLER ACCEPTED AGE
eg gateway.envoyproxy.io/gatewayclass-controller True 1s
A test Gateway without a route shows whether the port answers on all nodes:
cat > probe-gw.yaml <<'EOF'
# probe-gw.yaml – test for the NodePort (Day 12, delete afterwards)
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: probe
namespace: default
spec:
gatewayClassName: eg
listeners:
- {name: http, protocol: HTTP, port: 80}
EOF
kubectl apply -f probe-gw.yaml
kubectl -n default wait --for=condition=Programmed gateway/probe --timeout=120s
curl -s -o /dev/null -w '%{http_code}\n' "http://$SERVER:30080/"
curl -s -o /dev/null -w '%{http_code}\n' "http://$AGENT1:30080/"
curl -s -o /dev/null -w '%{http_code}\n' "http://$AGENT2:30080/"
kubectl delete -f probe-gw.yamlExpected:
gateway.gateway.networking.k8s.io/probe created
gateway.gateway.networking.k8s.io/probe condition met
404
404
404
gateway.gateway.networking.k8s.io "probe" deleted from default namespace
404 is correct: Envoy answers but doesn’t know any route yet. That
all three nodes answer, even though Envoy only runs on one of them, is
thanks to externalTrafficPolicy: Cluster from Day 11.
Save your progress? Sign up for free
Checkpoint
kubectl get nodesExpected: three nodes, all Ready, version
v1.36.5+rke2r1.
NAME STATUS ROLES AGE VERSION
rke2-agent1 Ready <none> 12m v1.36.5+rke2r1
rke2-agent2 Ready <none> 12m v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 15m v1.36.5+rke2r1
kubectl get scExpected: local-path with (default).
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer
kubectl get gatewayclass egExpected: ACCEPTED True.
NAME CONTROLLER ACCEPTED AGE
eg gateway.envoyproxy.io/gatewayclass-controller True 5m
kubectl get ns notesExpected: the namespace doesn’t exist here yet; it only comes on Day 13.
Error from server (NotFound): namespaces "notes" not found
Challenge
What does the cluster see when an agent fails? Stop the service
rke2-agent on rke2-agent1. Find out after how
many seconds the node becomes NotReady and which taints it
then carries. Start the service again afterwards.
Hint: Restarting the VM isn’t enough: it’s over
faster than the cluster gets suspicious. kubectl wait can
also wait for Ready=Unknown.
Solution
multipass exec rke2-agent1 -- sudo systemctl stop rke2-agent
time kubectl wait --for=condition=Ready=Unknown node/rke2-agent1 --timeout=180s
kubectl get nodes
kubectl get node rke2-agent1 -o jsonpath='{.spec.taints}'; echo
multipass exec rke2-agent1 -- sudo systemctl start rke2-agent
kubectl wait --for=condition=Ready node/rke2-agent1 --timeout=180sExpected (abridged):
node/rke2-agent1 condition met
real 0m42.3s
NAME STATUS ROLES AGE VERSION
rke2-agent1 NotReady <none> 8m42s v1.36.5+rke2r1
rke2-agent2 Ready <none> 8m40s v1.36.5+rke2r1
rke2-server Ready control-plane,etcd 11m v1.36.5+rke2r1
[{"effect":"NoSchedule","key":"node.kubernetes.io/unreachable",…},
{"effect":"NoExecute","key":"node.kubernetes.io/unreachable",…}]
node/rke2-agent1 condition met
After a good 40 seconds without a sign of life from the kubelet, the
controller sets the node to Unknown.
kubectl describe node names the reason:
Kubelet stopped posting node status. The taint
NoSchedule keeps new Pods away. NoExecute
evicts running Pods after another five minutes; that is how long they
tolerate it by default. After the start, the node is immediately
Ready again.
Save your progress? Sign up for free
Enterprise Note
- High availability: In production, three (or five) servers run so that etcd keeps a quorum. Agents and additional servers then point to a fixed address in front of the servers (load balancer or DNS), not to a single server.
- Hardening:
profile: cisinconfig.yamlturns on the CIS requirements (among other things, Pod Securityrestrictedcluster-wide). Beforehand, a useretcdand some kernel parameters must be set up on every host; seedocs.rke2.io. - Without internet: For data centers without access there are image archives and an “air-gapped” installation; the images then come from your own registry (Day 13).
- Storage: local-path ties data to one node. For real resilience, Longhorn or external storage (Ceph, NFS filer, SAN with CSI driver) replicates the volumes.
- Many clusters: Rancher Manager manages RKE2 clusters centrally, with permissions, upgrades and GitOps via Fleet.