Day 8: RKE2 HA with Ansible
Goal: Three RKE2 servers and three agents form an HA cluster behind the API VIP
10.10.20.10, with Cilium as the CNI, and you upgrade it via the system-upgrade-controller. · Duration: approx. 120 min · Prerequisite: Day 7
Briefing
- RKE2
v1.36.4+rke2r1gets installed; in Drill 7 a plan takes it tov1.36.5+rke2r1(stable channel, October 2026). - Role
rke2_server: installation,config.yaml, Cilium HelmChartConfig and kube-vip as a manifest. Rolerke2_agentuses the same install tasks. - Join order:
rke2-cp-1initializes etcd,cp-2/cp-3join one at a time via the VIP (port 9345), then the agents. - kube-vip (DaemonSet, ARP, leader election) keeps the VIP on exactly one server. Cilium replaces kube-proxy and can therefore do L2 announcements on Day 9.
- Since v1.36, Traefik is the RKE2 default ingress. ingress-nginx has
been discontinued upstream since March 2026 and is no longer available
for new clusters from v1.37 on. Hence
ingress-controller: traefik.
VIP 10.10.20.10 (kube-vip, ARP) rke2.lab.internal
:6443 API / :9345 supervisor (join)
┌──────────────┬──────────────┬──────────────┐
rke2-cp-1 .11 rke2-cp-2 .12 rke2-cp-3 .13 etcd + CP
└──────┬───────┴──────┬───────┴──────┬───────┘
rke2-wk-1 .21 rke2-wk-2 .22 rke2-gpu-1 .31 Agents
Drill 1: Project, inventory, variables
The inventory comes from Day 7. Add configuration and shared variables:
cd ~/ki-lab/code/infra-rke2
mkdir -p group_vars upgrade roles/rke2_server/{defaults,tasks,templates,handlers} \
roles/rke2_agent/{tasks,templates,handlers}
cat > ansible.cfg <<'EOF'
[defaults]
inventory = inventory.yaml
roles_path = roles
host_key_checking = False
interpreter_python = auto_silent
forks = 10
[ssh_connection]
pipelining = True
EOF
cat > group_vars/all.yml <<'EOF'
---
rke2_version: v1.36.4+rke2r1 # Day 8 Drill 7: upgrade to v1.36.5
# Lab value! For real: export RKE2_TOKEN=$(openssl rand -hex 32)
rke2_token: >-
{{ lookup('ansible.builtin.env', 'RKE2_TOKEN')
| default('rke2-lab-token-1234', true) }}
rke2_vip: 10.10.20.10
rke2_vip_hostname: rke2.lab.internal
rke2_vip_interface: eth0
rke2_first_server: "{{ groups['rke2_servers'][0] }}"
# Servers only for control-plane pods (CriticalAddonsOnly)
rke2_server_taint: true
kube_vip_version: v1.2.4
kubeconfig_local: "~/ki-lab/rke2.yaml"
kubeconfig_context: rke2-lab
EOF
export RKE2_TOKEN=$(openssl rand -hex 32)
echo "export RKE2_TOKEN=$RKE2_TOKEN" > ~/ki-lab/rke2-token.env
chmod 600 ~/ki-lab/rke2-token.env
ansible-inventory --graphExpected:
@all:
|--@ungrouped:
|--@gpu_nodes:
| |--rke2-gpu-1
|--@rke2_agents:
| |--rke2-gpu-1
| |--rke2-wk-1
| |--rke2-wk-2
|--@rke2_servers:
| |--rke2-cp-1
| |--rke2-cp-2
| |--rke2-cp-3
Why a token from the environment? The cluster token is the join
secret and additionally protects the etcd bootstrap data.
rke2-lab-token-1234 is only the lab fallback in case
RKE2_TOKEN is missing.
Save your progress? Sign up for free
Drill 2: Role rke2_server – installation
cat > roles/rke2_server/defaults/main.yml <<'EOF'
---
rke2_server_install_type: server
rke2_server_manifest_dir: /var/lib/rancher/rke2/server/manifests
rke2_server_kubectl: >-
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml
rke2_server_cilium_qps: 30
rke2_server_cilium_burst: 50
EOF
cat > roles/rke2_server/tasks/install.yml <<'EOF'
---
- name: Installationsskript laden
ansible.builtin.get_url:
url: https://get.rke2.io
dest: /root/rke2-install.sh
mode: "0700"
- name: RKE2 installieren ({{ rke2_server_install_type }})
ansible.builtin.command: /root/rke2-install.sh
environment:
INSTALL_RKE2_VERSION: "{{ rke2_version }}"
INSTALL_RKE2_TYPE: "{{ rke2_server_install_type }}"
args:
creates: /usr/local/bin/rke2
- name: Konfigurationsverzeichnis anlegen
ansible.builtin.file:
path: /etc/rancher/rke2
state: directory
mode: "0755"
EOFcreates: makes the installation idempotent. Upgrades
don’t run through this script but through the system-upgrade-controller
(Drill 7).
Save your progress? Sign up for free
Drill 3: config.yaml and Cilium
cat > roles/rke2_server/templates/config.yaml.j2 <<'EOF'
# {{ ansible_managed }}
token: {{ rke2_token }}
{% if inventory_hostname != rke2_first_server %}
server: https://{{ rke2_vip }}:9345
{% endif %}
node-ip: {{ ansible_host }}
tls-san:
- {{ rke2_vip }}
- {{ rke2_vip_hostname }}
cni: cilium
disable-kube-proxy: true
ingress-controller: traefik
write-kubeconfig-mode: "0600"
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 12
{% if rke2_server_taint %}
node-taint:
- "CriticalAddonsOnly=true:NoExecute"
{% endif %}
EOF
cat > roles/rke2_server/templates/rke2-cilium-config.yaml.j2 <<'EOF'
# {{ ansible_managed }}
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-cilium
namespace: kube-system
spec:
valuesContent: |-
kubeProxyReplacement: true
k8sServiceHost: "localhost"
k8sServicePort: "6443"
l2announcements:
enabled: true
externalIPs:
enabled: true
k8sClientRateLimit:
qps: {{ rke2_server_cilium_qps }}
burst: {{ rke2_server_cilium_burst }}
devices: {{ rke2_vip_interface }}
operator:
replicas: 1
EOFWhy like this?
tls-san: The API certificate must contain the VIP and hostname; otherwisekubectlviarke2.lab.internalfails.- Only
rke2-cp-1has noserver:– it initializes etcd. All others join via the VIP. node-taint CriticalAddonsOnly: Workloads (vLLM, LiteLLM, Longhorn) stay on the agents; the 8 GB servers carry only etcd and the control plane.- Cilium:
kubeProxyReplacementwithk8sServiceHost: localhost(RKE2 docs). L2 announcements need the kube-proxy replacement and a higher client rate limit, because every node renews leases. You create the pool itself on Day 9.
Save your progress? Sign up for free
Drill 4: kube-vip for the API VIP
kube-vip runs as a DaemonSet on all servers (hostNetwork). The leader
sets 10.10.20.10 on eth0 via ARP. RKE2 applies
everything in server/manifests/ automatically – no
kubectl apply needed, so it already works on the first
server.
cat > roles/rke2_server/templates/kube-vip.yaml.j2 <<'EOF'
# {{ ansible_managed }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-vip
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:kube-vip-role
rules:
- apiGroups: [""]
resources: ["services/status"]
verbs: ["update"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["list", "get", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list", "get", "watch", "update", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["list", "get", "watch", "update", "create"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["list", "get", "watch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-vip-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-vip-role
subjects:
- kind: ServiceAccount
name: kube-vip
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-vip-ds
namespace: kube-system
spec:
selector:
matchLabels:
name: kube-vip-ds
template:
metadata:
labels:
name: kube-vip-ds
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
containers:
- name: kube-vip
image: ghcr.io/kube-vip/kube-vip:{{ kube_vip_version }}
args: ["manager"]
env:
- {name: vip_arp, value: "true"}
- {name: port, value: "6443"}
- {name: vip_interface, value: "{{ rke2_vip_interface }}"}
- {name: vip_subnet, value: "32"}
- {name: cp_enable, value: "true"}
- {name: cp_namespace, value: kube-system}
- {name: svc_enable, value: "false"}
- {name: vip_leaderelection, value: "true"}
- {name: vip_leaseduration, value: "5"}
- {name: vip_renewdeadline, value: "3"}
- {name: vip_retryperiod, value: "1"}
- {name: address, value: "{{ rke2_vip }}"}
securityContext:
capabilities:
add: ["NET_ADMIN", "NET_RAW", "SYS_TIME"]
hostNetwork: true
serviceAccountName: kube-vip
tolerations:
- {effect: NoSchedule, operator: Exists}
- {effect: NoExecute, operator: Exists}
EOFRBAC and environment variables match the manifest generated by
kube-vip manifest daemonset --controlplane --arp --leaderElection
(kube-vip docs). svc_enable: "false", because Cilium takes
over Services on Day 9 – two ARP speakers for the same IPs would be a
mistake.
Tip: Why not a static pod? The static pod route needs an admin kubeconfig on the host, which doesn’t exist yet on joining RKE2 servers before they start. The DaemonSet uses its ServiceAccount.
Save your progress? Sign up for free
Drill 5: Tasks, handlers and the agent role
cat > roles/rke2_server/tasks/main.yml <<'EOF'
---
- name: RKE2 installieren
ansible.builtin.include_tasks: install.yml
- name: Manifest-Verzeichnis anlegen
ansible.builtin.file:
path: "{{ rke2_server_manifest_dir }}"
state: directory
mode: "0755"
- name: Cilium-HelmChartConfig und kube-vip ablegen
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "{{ rke2_server_manifest_dir }}/{{ item }}"
mode: "0600"
loop:
- rke2-cilium-config.yaml
- kube-vip.yaml
- name: Server-config.yaml schreiben
ansible.builtin.template:
src: config.yaml.j2
dest: "/etc/rancher/rke2/config.yaml"
mode: "0600"
notify: Restart rke2-server
- name: Rke2-server starten
ansible.builtin.systemd_service:
name: rke2-server
enabled: true
state: started
- name: Warten bis der Node Ready ist
ansible.builtin.command: >-
{{ rke2_server_kubectl }} wait --for=condition=Ready
node/{{ inventory_hostname }} --timeout=30s
register: rke2_server_node_ready
until: rke2_server_node_ready.rc == 0
retries: 40
delay: 15
changed_when: false
- name: API-Server über die VIP erreichbar
ansible.builtin.wait_for:
host: "{{ rke2_vip }}"
port: 6443
timeout: 300
EOF
cat > roles/rke2_server/handlers/main.yml <<'EOF'
---
- name: Restart rke2-server
ansible.builtin.systemd_service:
name: rke2-server
state: restarted
EOF
cat > roles/rke2_agent/tasks/main.yml <<'EOF'
---
- name: RKE2 installieren (Tasks aus rke2_server)
ansible.builtin.include_role:
name: rke2_server
tasks_from: install.yml
vars:
rke2_server_install_type: agent
- name: Agent-config.yaml schreiben
ansible.builtin.template:
src: config.yaml.j2
dest: /etc/rancher/rke2/config.yaml
mode: "0600"
notify: Restart rke2-agent
- name: Rke2-agent starten
ansible.builtin.systemd_service:
name: rke2-agent
enabled: true
state: started
EOF
cat > roles/rke2_agent/handlers/main.yml <<'EOF'
---
- name: Restart rke2-agent
ansible.builtin.systemd_service:
name: rke2-agent
state: restarted
EOF
cat > roles/rke2_agent/templates/config.yaml.j2 <<'EOF'
# {{ ansible_managed }}
server: https://{{ rke2_vip }}:9345
token: {{ rke2_token }}
node-ip: {{ ansible_host }}
EOFA server only counts as finished once it’s Ready
and the VIP answers. That way cp-2 never
joins against a VIP that nobody holds yet. The agent role needs no
manifests: agents get everything from the server.
Save your progress? Sign up for free
Drill 6: Join order and kubeconfig
cat > site.yml <<'EOF'
---
- name: Ersten RKE2-Server initialisieren
hosts: rke2_servers[0]
become: true
roles:
- rke2_server
- name: Weitere Server einzeln joinen
hosts: rke2_servers[1:]
become: true
serial: 1
roles:
- rke2_server
- name: Agents joinen
hosts: rke2_agents
become: true
serial: 2
roles:
- rke2_agent
- name: Kubeconfig auf den Laptop holen
hosts: rke2_servers[0]
become: true
tasks:
- name: Kubeconfig lesen
ansible.builtin.slurp:
src: /etc/rancher/rke2/rke2.yaml
register: rke2_kubeconfig
- name: Kubeconfig lokal schreiben (VIP-Hostname, Kontext rke2-lab)
become: false
delegate_to: localhost
ansible.builtin.copy:
dest: "{{ kubeconfig_local }}"
mode: "0600"
content: >-
{{ rke2_kubeconfig.content | b64decode
| replace('https://127.0.0.1:6443',
'https://' ~ rke2_vip_hostname ~ ':6443')
| regex_replace('(?m): default$', ': ' ~ kubeconfig_context) }}
EOF
ansible-playbook --syntax-check site.yml
ansible-lintExpected:
playbook: site.yml
Passed: 0 failure(s), 0 warning(s) in 23 files processed of 24 encountered.
Last profile that met the validation criteria was 'production'.
The plays reflect the join order: first rke2_servers[0],
then the remaining servers with serial: 1 (etcd tolerates
only one new member at a time), then the agents. The last play fetches
the kubeconfig, replaces 127.0.0.1 with the VIP and renames
default to rke2-lab.
grep -q rke2.lab.internal /etc/hosts || \
echo "10.10.20.10 rke2.lab.internal" | sudo tee -a /etc/hosts
source ~/ki-lab/rke2-token.env
ansible-playbook site.ymlExpected (abbreviated, not tested live, approx. 15–20 min):
TASK [rke2_server : Warten bis der Node Ready ist] *****************
FAILED - RETRYING: [rke2-cp-1]: Warten bis der Node Ready ist (40 retries left).
ok: [rke2-cp-1]
...
PLAY RECAP *********************************************************
rke2-cp-1 : ok=14 changed=6 unreachable=0 failed=0
rke2-cp-2 : ok=10 changed=5 unreachable=0 failed=0
rke2-cp-3 : ok=10 changed=5 unreachable=0 failed=0
rke2-gpu-1 : ok=6 changed=4 unreachable=0 failed=0
rke2-wk-1 : ok=6 changed=4 unreachable=0 failed=0
rke2-wk-2 : ok=6 changed=4 unreachable=0 failed=0
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl config current-context
kubectl get nodes -o wide | awk '{print $1, $2, $3, $5, $6}' | column -tExpected (abbreviated, not tested live):
rke2-lab
NAME STATUS ROLES VERSION INTERNAL-IP
rke2-cp-1 Ready control-plane,etcd,master v1.36.4+rke2r1 10.10.20.11
rke2-cp-2 Ready control-plane,etcd,master v1.36.4+rke2r1 10.10.20.12
rke2-cp-3 Ready control-plane,etcd,master v1.36.4+rke2r1 10.10.20.13
rke2-gpu-1 Ready <none> v1.36.4+rke2r1 10.10.20.31
rke2-wk-1 Ready <none> v1.36.4+rke2r1 10.10.20.21
rke2-wk-2 Ready <none> v1.36.4+rke2r1 10.10.20.22
Who holds the VIP, and is Cilium running without kube-proxy?
kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath='{.spec.holderIdentity}{"\n"}'
kubectl -n kube-system get ds kube-vip-ds cilium
kubectl -n kube-system get pods -l component=kube-proxy --no-headers | wc -l
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg status | grep -E 'KubeProxyReplacement|Cluster health'Expected (abbreviated, not tested live):
rke2-cp-1
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
kube-vip-ds 3 3 3 3 3
cilium 6 6 6 6 6
0
KubeProxyReplacement: True [eth0 10.10.20.21 (Direct Routing)]
Cluster health: 6/6 reachable (2026-10-09T18:12:03Z)
Save your progress? Sign up for free
Drill 7: Upgrade with the system-upgrade-controller
The controller runs a Job with the image
rancher/rke2-upgrade per node: cordon, swap the binary,
restart the service. Servers always before agents – the
prepare step in the agent plan takes care of that.
SUC=https://github.com/rancher/system-upgrade-controller/releases/download/v0.20.2
kubectl apply -f $SUC/crd.yaml -f $SUC/system-upgrade-controller.yaml
kubectl -n system-upgrade rollout status deploy/system-upgrade-controller
cat > upgrade/plans.yaml <<'EOF'
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: server-plan
namespace: system-upgrade
spec:
concurrency: 1
cordon: true
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: In
values: ["true"]
tolerations:
- key: CriticalAddonsOnly
operator: Exists
serviceAccountName: system-upgrade
upgrade:
image: rancher/rke2-upgrade
version: v1.36.5+rke2r1
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: agent-plan
namespace: system-upgrade
spec:
concurrency: 1
cordon: true
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
prepare:
image: rancher/rke2-upgrade
args: ["prepare", "server-plan"]
serviceAccountName: system-upgrade
upgrade:
image: rancher/rke2-upgrade
version: v1.36.5+rke2r1
EOF
kubectl apply -f upgrade/plans.yamlExpected (abbreviated, not tested live):
deployment "system-upgrade-controller" successfully rolled out
plan.upgrade.cattle.io/server-plan created
plan.upgrade.cattle.io/agent-plan created
version instead of channel: in the lab, you
want to decide when an upgrade runs. With
channel: https://update.rke2.io/v1-release/channels/stable,
the cluster pulls every new stable version by itself. The server plan
needs the tolerations because of the
CriticalAddonsOnly taint.
kubectl -n system-upgrade get plans -o wide
kubectl get nodes -w # stop with Ctrl+C after approx. 15 minExpected (abbreviated, not tested live):
NAME IMAGE CHANNEL VERSION COMPLETE
server-plan rancher/rke2-upgrade v1.36.5+rke2r1 False
agent-plan rancher/rke2-upgrade v1.36.5+rke2r1 False
rke2-cp-1 Ready,SchedulingDisabled control-plane,etcd,master v1.36.4+rke2r1
rke2-cp-1 Ready control-plane,etcd,master v1.36.5+rke2r1
...
Afterwards, set rke2_version in
group_vars/all.yml to v1.36.5+rke2r1 so new
nodes join with the current version right away.
Warning: Don’t skip a minor version (1.36 → 1.38 doesn’t work). Take an etcd snapshot before every upgrade:
sudo rke2 etcd-snapshot save --name pre-upgradeonrke2-cp-1.
Save your progress? Sign up for free
Checkpoint
export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Kontext rke2-lab" '[ "$(kubectl config current-context)" = rke2-lab ]'
check "6 Nodes Ready" '[ "$(kubectl get nodes --no-headers \
| awk "\$2==\"Ready\"" | wc -l)" -eq 6 ]'
check "3 etcd-Mitglieder" '[ "$(kubectl get nodes \
-l node-role.kubernetes.io/etcd=true --no-headers | wc -l)" -eq 3 ]'
check "API ueber VIP" 'curl -sk --max-time 5 https://10.10.20.10:6443/livez'
check "kube-vip 3/3" '[ "$(kubectl -n kube-system get ds kube-vip-ds \
-o jsonpath="{.status.numberReady}")" -eq 3 ]'
check "kein kube-proxy" '[ "$(kubectl -n kube-system get pods \
-l component=kube-proxy --no-headers | wc -l)" -eq 0 ]'
check "IngressClass traefik" 'kubectl get ingressclass traefik'
check "Plans complete" '[ "$(kubectl -n system-upgrade get plans \
-o jsonpath="{.items[*].status.conditions[?(@.type==\"Complete\")].status}")" \
= "True True" ]'Expected:
✔ Kontext rke2-lab
✔ 6 Nodes Ready
✔ 3 etcd-Mitglieder
✔ API ueber VIP
✔ kube-vip 3/3
✔ kein kube-proxy
✔ IngressClass traefik
✔ Plans complete
Challenge
Prove that the VIP is HA: shut down the server that currently holds
the lease plndr-cp-lock, and show that kubectl
via rke2.lab.internal keeps working after a few seconds and
another server takes over the VIP. Start it again afterwards.
Hint: The VMID follows from the name
(rke2-cp-2 → 212); qm shutdown runs on the
Proxmox host.
Solution
H=$(kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath='{.spec.holderIdentity}'); echo "VIP bei $H"
ssh [email protected] "qm shutdown 21${H##*-}"
for i in $(seq 1 12); do
kubectl get --raw=/livez >/dev/null 2>&1 && echo "API ok nach $((i*5))s" && break
sleep 5
done
kubectl -n kube-system get lease plndr-cp-lock -o jsonpath='{.spec.holderIdentity}'
ssh [email protected] "qm start 21${H##*-}"Expected (abbreviated, not tested live):
VIP bei rke2-cp-1
API ok nach 10s
rke2-cp-3
With two of three etcd members, quorum is preserved. If you shut down two servers, the API stops – that’s exactly why there are three.
Save your progress? Sign up for free
Enterprise Note
- CIS:
profile: cisin the server and agentconfig.yamlenables RKE2 hardening (PSArestricted, NetworkPolicies, audit log). Prerequisites according to the docs: create the useretcdand copyrke2-cis-sysctl.confto/etc/sysctl.d/– deliberately off in the lab, because the GPU Operator and Longhorn need exceptions. - Token and secrets encryption: cluster token from Vault/OpenBao;
secrets-encryptionis the default in RKE2 – put key rotation (rke2 secrets-encrypt rotate-keys) into the operations plan. - Additionally send etcd snapshots to S3 (
etcd-s3: true+ bucket), practice restores regularly (Day 15). - kube-vip in ARP mode needs a flat L2 network. Across data centers: BGP mode or an external load balancer (HAProxy/F5) in front of ports 6443 and 9345.
- Air gap: tarballs and images (
rke2-images-*.tar.zst) instead ofget.rke2.io, registry mirror in/etc/rancher/rke2/registries.yaml.