Day 8: Building the RKE2 Cluster on Proxmox
Goal: Six VMs via OpenTofu, on them RKE2 HA with Cilium (kube-proxy replacement, L2) and kube-vip – reachable via the VIP
10.10.20.10. · Duration: approx. 120 min · Prerequisite: Day 7, Proxmox VE 9.x with approx. 18 vCPU, 36 GB RAM, 300 GB disk
Briefing
- Proxmox only provides compute: a cloud-init template (VMID 9000),
six clones via OpenTofu (
bpg/proxmox). - Ansible installs RKE2: three servers (etcd, control plane, taint
CriticalAddonsOnly), three agents for workloads. - Cilium comes as the RKE2 chart
rke2-cilium; aHelmChartConfigsets the values. kube-proxy is off, Cilium takes over (eBPF). - kube-vip holds the API VIP
10.10.20.10via ARP. Cilium will assign the LoadBalancer IPs for Gateways later (Day 9). - RKE2 ingress is off (
ingress-controller: none): since v1.36, Traefik would bring its own Gateway API CRDs (v1.5.1) and collide with Envoy Gateway.
Laptop: tofu + ansible + kubectl
│ API 8006 (token tofu@pve!provider)
┌──────▼──────────────── Proxmox VE 9 · vmbr0 · 10.10.20.0/24 ────────────┐
│ 9000 ubuntu-2404-tpl ──clone──► rke2-cp-1 .11 rke2-wk-1 .21 │
│ rke2-cp-2 .12 rke2-wk-2 .22 │
│ rke2-cp-3 .13 rke2-wk-3 .23 │
│ VIP 10.10.20.10 (kube-vip, ARP) LB pool .100-.120 (Cilium, Day 9) │
└─────────────────────────────────────────────────────────────────────────┘
Warning: Proxmox, multi-node RKE2 and ARP VIPs cannot be reproduced in kind. All output in this chapter comes from the official documentation or static validation (
tofu validate,ansible-lint,helm template) and is marked “(shortened, may differ depending on the version)”.
Drill 1: API Token and Template on the Proxmox Host
The code lives in ~/gw-lab/code/infra-proxmox. The
script pve-prep.sh runs once as root on the Proxmox
host.
scp ~/gw-lab/code/infra-proxmox/pve-prep.sh [email protected]:
ssh [email protected] 'bash ~/pve-prep.sh'The core of the script – a role with minimal privileges for cloning and cloud-init:
pveum role add TofuProv -privs "VM.Allocate VM.Clone VM.Audit VM.PowerMgmt \
VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk \
VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options \
VM.GuestAgent.Audit Datastore.AllocateSpace Datastore.Audit \
SDN.Use SDN.Audit Sys.Audit"
pveum user add tofu@pve --comment "OpenTofu Lab"
pveum aclmod / -user tofu@pve -role TofuProv
pveum user token add tofu@pve provider --privsep 0Expected (shortened, may differ depending on the version):
┌──────────────┬──────────────────────────────────────┐
│ key │ value │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ tofu@pve!provider │
│ info │ {"privsep":"0"} │
│ value │ 3f0c...-....-....-....-............ │
└──────────────┴──────────────────────────────────────┘
The value appears only once. On the laptop you store it
only in the shell environment:
read -rs PVE_SECRET
export TF_VAR_pve_api_token="tofu@pve!provider=${PVE_SECRET}"The second part builds the template: Ubuntu 24.04 cloud image,
qemu-guest-agent installed into it via
virt-customize, disk via import-from.
virt-customize -a "$IMG" --install qemu-guest-agent \
--truncate /etc/machine-id
qm create 9000 --name ubuntu-2404-tpl --ostype l26 \
--cores 2 --memory 2048 --cpu host \
--net0 "virtio,bridge=$BRIDGE" --scsihw virtio-scsi-single \
--agent enabled=1 --serial0 socket --vga serial0
qm set 9000 --scsi0 "$STORAGE:0,import-from=$PWD/$IMG,discard=on,ssd=1"
qm set 9000 --ide2 "$STORAGE:cloudinit" --boot order=scsi0
qm template 9000Expected (shortened, may differ depending on the version):
[ 0.0] Examining the guest ...
[ 14.2] Installing packages: qemu-guest-agent
[ 41.8] Finishing off
transferred 3.5 GiB of 3.5 GiB (100.00%)
scsi0: successfully created disk 'local-lvm:base-9000-disk-0,discard=on,ssd=1'
Why the guest agent? With agent { enabled = true } the
provider waits for the agent and reads the IPs from it. If it is missing
from the image, tofu apply hangs until the timeout.
Save your progress? Sign up for free
Drill 2: Six VMs With OpenTofu
cd ~/gw-lab/code/infra-proxmox
cat main.tfExpected:
resource "proxmox_virtual_environment_vm" "node" {
for_each = var.nodes
name = each.key
node_name = var.pve_node
vm_id = each.value.vm_id
tags = ["rke2", each.value.role]
on_boot = true
clone {
vm_id = var.template_id
full = true
}
agent {
enabled = true # qemu-guest-agent ist im Template (pve-prep.sh)
}
cpu {
cores = each.value.cores
type = "host" # Cilium/eBPF profitiert von allen CPU-Flags
}
memory {
dedicated = each.value.memory
}
disk {
datastore_id = var.datastore
interface = "scsi0"
size = each.value.disk
discard = "on"
ssd = true
}
network_device {
bridge = var.bridge
model = "virtio"
}
initialization {
datastore_id = var.datastore
dns {
domain = "lab.internal"
servers = [var.dns_server]
}
ip_config {
ipv4 {
address = "${each.value.ip}/24"
gateway = var.gateway
}
}
user_account {
username = "ubuntu"
keys = [trimspace(var.ssh_public_key)]
}
}
}
The six nodes are defined as a map in variables.tf
(servers 2 vCPU/4 GB, agents 4 vCPU/8 GB). Your own values (endpoint,
SSH key) go into terraform.tfvars:
cp terraform.tfvars.example terraform.tfvars
sed -i.bak "s|ssh-ed25519 AAAA.*\"|$(cat ~/.ssh/id_ed25519.pub)\"|" \
terraform.tfvars
tofu init
tofu validateExpected (shortened):
- Installing bpg/proxmox v0.116.0...
- Installed bpg/proxmox v0.116.0 (signed, key ID ...)
OpenTofu has been successfully initialized!
Success! The configuration is valid.
tofu plan -out lab.plan | tail -1
tofu apply lab.planExpected (shortened, may differ depending on the version):
Plan: 6 to add, 0 to change, 0 to destroy.
proxmox_virtual_environment_vm.node["rke2-cp-1"]: Creation complete after 1m12s
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
Tip:
proxmox_virtual_environment_vmis the provider’s stable resource. The new namesproxmox_vm/proxmox_cloned_vmare still experimental or young in v0.116 – in the lab you stick with what is proven.
Save your progress? Sign up for free
Drill 3: Inventory and Reachability
OpenTofu knows the IPs and renders the Ansible inventory from
inventory.tftpl:
tofu output -raw inventory > ~/gw-lab/code/infra-rke2/inventory.yml
cd ~/gw-lab/code/infra-rke2
cat inventory.ymlExpected:
all:
vars:
ansible_user: ubuntu
children:
rke2_servers:
hosts:
rke2-cp-1: { ansible_host: 10.10.20.11 }
rke2-cp-2: { ansible_host: 10.10.20.12 }
rke2-cp-3: { ansible_host: 10.10.20.13 }
rke2_agents:
hosts:
rke2-wk-1: { ansible_host: 10.10.20.21 }
rke2-wk-2: { ansible_host: 10.10.20.22 }
rke2-wk-3: { ansible_host: 10.10.20.23 }
ansible-galaxy collection install -r requirements.yml
ansible all -m ansible.builtin.ping -o
ansible all -a 'ip -br -4 addr show eth0' -oExpected (shortened, may differ depending on the version):
rke2-cp-1 | SUCCESS => {"changed": false,"ping": "pong"}
...
rke2-wk-3 | CHANGED | rc=0 | (stdout) eth0 UP 10.10.20.23/24
Why eth0? The network configuration Proxmox delivers via
cloud-init names the NIC eth0. Exactly this name is set in
group_vars/all.yml as rke2_iface – kube-vip
and Cilium need it.
Save your progress? Sign up for free
Drill 4: Understanding the RKE2 Configuration
cat group_vars/all.ymlExpected:
rke2_version: v1.36.5+rke2r1
rke2_vip: 10.10.20.10
rke2_api_fqdn: rke2.lab.internal
rke2_iface: eth0 # Proxmox-Cloud-Init benennt die NIC eth0
# Lab-Wert! Echt: per ansible-vault oder Umgebungsvariable
rke2_token: >-
{{ lookup('ansible.builtin.env', 'RKE2_TOKEN')
| default('rke2-lab-token-1234', true) }}
kube_vip_version: v1.2.4
# Tag 9 setzt das auf "true", sobald die Gateway-API-CRDs da sind
cilium_gateway_class_create: auto
rke2_first_server: "{{ groups['rke2_servers'][0] }}"The server configuration
(roles/rke2_server/templates/config.yaml.j2):
token: {{ rke2_token }}
{% if inventory_hostname != rke2_first_server %}
server: https://{{ rke2_vip }}:9345
{% endif %}
node-ip: {{ ansible_host }}
tls-san:
- {{ rke2_vip }}
- {{ rke2_api_fqdn }}
cni: cilium
disable-kube-proxy: true
ingress-controller: none
node-taint:
- "CriticalAddonsOnly=true:NoExecute"
write-kubeconfig-mode: "0600"
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 8| Key | Effect |
|---|---|
tls-san |
The API certificate is also valid for the VIP and
rke2.lab.internal |
server |
missing only on rke2-cp-1 – it initializes etcd |
disable-kube-proxy |
no kube-proxy pod, Cilium takes over Services |
ingress-controller: none |
no Traefik, no foreign Gateway API CRDs |
Ansible places the Cilium values as a HelmChartConfig in
/var/lib/rancher/rke2/server/manifests/ – RKE2 merges them
into its chart rke2-cilium:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-cilium
namespace: kube-system
spec:
valuesContent: |-
kubeProxyReplacement: true
k8sServiceHost: localhost # RKE2 supervisor LB on every node
k8sServicePort: 6443
devices: {{ rke2_iface }}
l2announcements:
enabled: true
externalIPs:
enabled: true
k8sClientRateLimit: # L2 leases need API budget
qps: 20
burst: 40
gatewayAPI:
enabled: true
gatewayClass:
create: "{{ cilium_gateway_class_create }}"
operator:
replicas: 2
rollOutPods: true
rollOutCiliumPods: trueWhy localhost:6443? Without kube-proxy, there is no
working kubernetes Service at startup. But every RKE2 node
runs a local load balancer on 127.0.0.1:6443 that forwards
to the servers – Cilium talks to that.
Save your progress? Sign up for free
Drill 5: kube-vip for the API VIP
kube-vip runs as a DaemonSet on the servers only and is also placed
as an auto-deploy manifest. Excerpt from
roles/rke2_server/templates/kube-vip.yaml.j2:
containers:
- name: kube-vip
image: ghcr.io/kube-vip/kube-vip:{{ kube_vip_version }}
args: ["manager"]
env:
- { name: vip_arp, value: "true" }
- { name: port, value: "6443" }
- { name: vip_interface, value: "{{ rke2_iface }}" }
- { name: vip_subnet, value: "32" }
- { name: cp_enable, value: "true" }
- { name: cp_namespace, value: kube-system }
- { name: svc_enable, value: "false" }
- { name: vip_leaderelection, value: "true" }
- { name: vip_leaseduration, value: "5" }
- { name: vip_renewdeadline, value: "3" }
- { name: vip_retryperiod, value: "1" }
- { name: address, value: "{{ rke2_vip }}" }The bootstrap sequence: rke2-cp-1 starts without
server:, the manifest creates the DaemonSet, kube-vip wins
the lease plndr-cp-lock and answers ARP for
10.10.20.10. Only then can rke2-cp-2/3 and the
agents join via https://10.10.20.10:9345 – that is why the
role waits for port 9345 on the VIP.
Why svc_enable: "false"? Otherwise kube-vip and Cilium
LB IPAM would compete for the same LoadBalancer Services.
Before the run you check statically:
ansible-playbook --syntax-check site.yml
ansible-lintExpected:
playbook: site.yml
Passed: 0 failure(s), 0 warning(s) in 16 files processed of 17 encountered.
Last profile that met the validation criteria was 'production'.
Save your progress? Sign up for free
Drill 6: Run the Playbook
export RKE2_TOKEN=$(openssl rand -hex 32)
time ansible-playbook site.ymlExpected (shortened, may differ depending on the version):
PLAY [Erster Server (cluster-init)] ****************************************
TASK [rke2_server : Warten, bis der Node Ready ist] ************************
FAILED - RETRYING: [rke2-cp-1]: Warten, bis der Node Ready ist (20 retries left).
ok: [rke2-cp-1]
TASK [rke2_server : Warten, bis die VIP den Supervisor-Port bedient] *******
ok: [rke2-cp-1]
PLAY [Weitere Server, einzeln (etcd-Quorum!)] ******************************
...
PLAY RECAP *****************************************************************
rke2-cp-1 : ok=14 changed=8 unreachable=0 failed=0 skipped=0
rke2-wk-3 : ok=8 changed=5 unreachable=0 failed=0 skipped=0
real 11m42s
Why serial: 1 for the additional servers? If two new
etcd members join at the same time, the quorum can briefly tip over. One
at a time is robust.
The last play fetches rke2.yaml, replaces
127.0.0.1 with the VIP and the name default
with rke2-lab:
export KUBECONFIG=~/.kube/rke2-lab.yaml
kubectl config current-context
kubectl get nodes -o wideExpected (shortened, may differ depending on the version):
rke2-lab
NAME STATUS ROLES AGE VERSION INTERNAL-IP
rke2-cp-1 Ready control-plane,etcd 14m v1.36.5+rke2r1 10.10.20.11
rke2-cp-2 Ready control-plane,etcd 10m v1.36.5+rke2r1 10.10.20.12
rke2-cp-3 Ready control-plane,etcd 8m v1.36.5+rke2r1 10.10.20.13
rke2-wk-1 Ready <none> 5m v1.36.5+rke2r1 10.10.20.21
rke2-wk-2 Ready <none> 5m v1.36.5+rke2r1 10.10.20.22
rke2-wk-3 Ready <none> 5m v1.36.5+rke2r1 10.10.20.23
Save your progress? Sign up for free
Drill 7: Check Cilium and the VIP
kubectl -n kube-system get ds kube-proxy 2>&1 | tail -1
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg status | grep -E 'KubeProxyReplacement|Cluster health'
kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath='{.spec.holderIdentity}{"\n"}'Expected (shortened, may differ depending on the version):
Error from server (NotFound): daemonsets.apps "kube-proxy" not found
KubeProxyReplacement: True [eth0 10.10.20.21 fe80::... (Direct Routing)]
Cluster health: 6/6 reachable (2026-10-09T10:12:31Z)
rke2-cp-1
Gateway API is enabled in Cilium, but the CRDs are still missing. The operator notices this at startup, reports an error and leaves the controller off – Cilium itself keeps running normally:
kubectl -n kube-system logs deploy/cilium-operator | grep -i gateway | head -2
kubectl get gatewayclass 2>&1Expected (shortened, may differ depending on the version):
level=error msg="Required GatewayAPI resources are not found, please refer to docs
for installation instructions" ...
error: the server doesn't have a resource type "gatewayclass"
That is intended: on Day 9 you install the CRDs and restart the operator – the order is the topic there.
Save your progress? Sign up for free
Checkpoint
export KUBECONFIG=~/.kube/rke2-lab.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "API über VIP erreichbar" \
'curl -sk https://10.10.20.10:6443/version | grep -q gitVersion'
check "6 Nodes Ready" \
'[ "$(kubectl get nodes --no-headers | grep -c " Ready ")" -eq 6 ]'
check "3 etcd-Server" \
'[ "$(kubectl get nodes -l node-role.kubernetes.io/etcd=true \
--no-headers | wc -l)" -eq 3 ]'
check "Server getaintet" \
'kubectl get node rke2-cp-1 -o jsonpath="{.spec.taints}" \
| grep -q CriticalAddonsOnly'
check "kein kube-proxy" '! kubectl -n kube-system get ds kube-proxy'
check "Cilium-DaemonSet vollständig" \
'kubectl -n kube-system rollout status ds/cilium --timeout=10s'
check "kube-vip-Lease vergeben" \
'kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath="{.spec.holderIdentity}" | grep -q rke2-cp'
check "kein Traefik" '! kubectl -n kube-system get deploy rke2-traefik'Challenge
Prove that the API VIP is fail-safe: hard-stop the current lease
holder and measure how long kubectl get nodes via the VIP
does not respond. Start the VM again afterwards.
Hint: kube-vip’s lease duration is 5 s,
qm stop simulates a power failure.
Solution
holder=$(kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath='{.spec.holderIdentity}')
vmid=$(ssh [email protected] \
"qm list | awk '\$2==\"$holder\"{print \$1}'")
ssh [email protected] "qm stop $vmid"
start=$(date +%s)
until kubectl get nodes --request-timeout=2s >/dev/null 2>&1; do sleep 1; done
echo "API wieder da nach $(( $(date +%s) - start )) s"
kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath='{.spec.holderIdentity}{"\n"}'
ssh [email protected] "qm start $vmid"Expected (shortened, may differ depending on the version):
API wieder da nach 9 s
rke2-cp-2
With two of three etcd members the quorum is preserved; only the VIP moves.
Save your progress? Sign up for free
Enterprise Note
- Build the template and VMs via a pipeline (Packer or an image
factory), tag the image version; token with
--privsep 1and ACLs only on pool/storage. rke2_tokenfrom Vault/OpenBao oransible-vault, never in the repo; check the RKE2 CIS profile (profile: cis) and Pod Security.- Instead of an ARP VIP, often an external load balancer (F5, HAProxy pair) or BGP – kube-vip ARP needs a flat L2 segment.
- etcd snapshots to S3 (
etcd-s3: true) and practice restores regularly – Day 14 takes a snapshot as a safety net before the drills. - Upgrades with the System Upgrade Controller and
Planresources instead of ad-hoc scripts (see docs.rke2.io, “Automated Upgrades”).