Skip to content
Hands-on IT IT Drills

Day 8: Building the RKE2 Cluster on Proxmox

Goal: Six VMs via OpenTofu, on them RKE2 HA with Cilium (kube-proxy replacement, L2) and kube-vip – reachable via the VIP 10.10.20.10. · Duration: approx. 120 min · Prerequisite: Day 7, Proxmox VE 9.x with approx. 18 vCPU, 36 GB RAM, 300 GB disk

Briefing

  • Proxmox only provides compute: a cloud-init template (VMID 9000), six clones via OpenTofu (bpg/proxmox).
  • Ansible installs RKE2: three servers (etcd, control plane, taint CriticalAddonsOnly), three agents for workloads.
  • Cilium comes as the RKE2 chart rke2-cilium; a HelmChartConfig sets the values. kube-proxy is off, Cilium takes over (eBPF).
  • kube-vip holds the API VIP 10.10.20.10 via ARP. Cilium will assign the LoadBalancer IPs for Gateways later (Day 9).
  • RKE2 ingress is off (ingress-controller: none): since v1.36, Traefik would bring its own Gateway API CRDs (v1.5.1) and collide with Envoy Gateway.
 Laptop: tofu + ansible + kubectl
        │ API 8006 (token tofu@pve!provider)
 ┌──────▼──────────────── Proxmox VE 9 · vmbr0 · 10.10.20.0/24 ────────────┐
 │ 9000 ubuntu-2404-tpl ──clone──► rke2-cp-1 .11  rke2-wk-1 .21           │
 │                                 rke2-cp-2 .12  rke2-wk-2 .22           │
 │                                 rke2-cp-3 .13  rke2-wk-3 .23           │
 │ VIP 10.10.20.10 (kube-vip, ARP)   LB pool .100-.120 (Cilium, Day 9)    │
 └─────────────────────────────────────────────────────────────────────────┘

Warning: Proxmox, multi-node RKE2 and ARP VIPs cannot be reproduced in kind. All output in this chapter comes from the official documentation or static validation (tofu validate, ansible-lint, helm template) and is marked “(shortened, may differ depending on the version)”.

Drill 1: API Token and Template on the Proxmox Host

The code lives in ~/gw-lab/code/infra-proxmox. The script pve-prep.sh runs once as root on the Proxmox host.

scp ~/gw-lab/code/infra-proxmox/pve-prep.sh [email protected]:
ssh [email protected] 'bash ~/pve-prep.sh'

The core of the script – a role with minimal privileges for cloning and cloud-init:

pveum role add TofuProv -privs "VM.Allocate VM.Clone VM.Audit VM.PowerMgmt \
VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk \
VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options \
VM.GuestAgent.Audit Datastore.AllocateSpace Datastore.Audit \
SDN.Use SDN.Audit Sys.Audit"
pveum user add tofu@pve --comment "OpenTofu Lab"
pveum aclmod / -user tofu@pve -role TofuProv
pveum user token add tofu@pve provider --privsep 0

Expected (shortened, may differ depending on the version):

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ tofu@pve!provider                    │
│ info         │ {"privsep":"0"}                      │
│ value        │ 3f0c...-....-....-....-............  │
└──────────────┴──────────────────────────────────────┘

The value appears only once. On the laptop you store it only in the shell environment:

read -rs PVE_SECRET
export TF_VAR_pve_api_token="tofu@pve!provider=${PVE_SECRET}"

The second part builds the template: Ubuntu 24.04 cloud image, qemu-guest-agent installed into it via virt-customize, disk via import-from.

virt-customize -a "$IMG" --install qemu-guest-agent \
  --truncate /etc/machine-id
qm create 9000 --name ubuntu-2404-tpl --ostype l26 \
  --cores 2 --memory 2048 --cpu host \
  --net0 "virtio,bridge=$BRIDGE" --scsihw virtio-scsi-single \
  --agent enabled=1 --serial0 socket --vga serial0
qm set 9000 --scsi0 "$STORAGE:0,import-from=$PWD/$IMG,discard=on,ssd=1"
qm set 9000 --ide2 "$STORAGE:cloudinit" --boot order=scsi0
qm template 9000

Expected (shortened, may differ depending on the version):

[   0.0] Examining the guest ...
[  14.2] Installing packages: qemu-guest-agent
[  41.8] Finishing off
transferred 3.5 GiB of 3.5 GiB (100.00%)
scsi0: successfully created disk 'local-lvm:base-9000-disk-0,discard=on,ssd=1'

Why the guest agent? With agent { enabled = true } the provider waits for the agent and reads the IPs from it. If it is missing from the image, tofu apply hangs until the timeout.

Save your progress? Sign up for free

Drill 2: Six VMs With OpenTofu

cd ~/gw-lab/code/infra-proxmox
cat main.tf

Expected:

resource "proxmox_virtual_environment_vm" "node" {
  for_each = var.nodes

  name      = each.key
  node_name = var.pve_node
  vm_id     = each.value.vm_id
  tags      = ["rke2", each.value.role]
  on_boot   = true

  clone {
    vm_id = var.template_id
    full  = true
  }

  agent {
    enabled = true # qemu-guest-agent ist im Template (pve-prep.sh)
  }

  cpu {
    cores = each.value.cores
    type  = "host" # Cilium/eBPF profitiert von allen CPU-Flags
  }

  memory {
    dedicated = each.value.memory
  }

  disk {
    datastore_id = var.datastore
    interface    = "scsi0"
    size         = each.value.disk
    discard      = "on"
    ssd          = true
  }

  network_device {
    bridge = var.bridge
    model  = "virtio"
  }

  initialization {
    datastore_id = var.datastore
    dns {
      domain  = "lab.internal"
      servers = [var.dns_server]
    }
    ip_config {
      ipv4 {
        address = "${each.value.ip}/24"
        gateway = var.gateway
      }
    }
    user_account {
      username = "ubuntu"
      keys     = [trimspace(var.ssh_public_key)]
    }
  }
}

The six nodes are defined as a map in variables.tf (servers 2 vCPU/4 GB, agents 4 vCPU/8 GB). Your own values (endpoint, SSH key) go into terraform.tfvars:

cp terraform.tfvars.example terraform.tfvars
sed -i.bak "s|ssh-ed25519 AAAA.*\"|$(cat ~/.ssh/id_ed25519.pub)\"|" \
  terraform.tfvars
tofu init
tofu validate

Expected (shortened):

- Installing bpg/proxmox v0.116.0...
- Installed bpg/proxmox v0.116.0 (signed, key ID ...)
OpenTofu has been successfully initialized!
Success! The configuration is valid.
tofu plan -out lab.plan | tail -1
tofu apply lab.plan

Expected (shortened, may differ depending on the version):

Plan: 6 to add, 0 to change, 0 to destroy.
proxmox_virtual_environment_vm.node["rke2-cp-1"]: Creation complete after 1m12s
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Tip: proxmox_virtual_environment_vm is the provider’s stable resource. The new names proxmox_vm/proxmox_cloned_vm are still experimental or young in v0.116 – in the lab you stick with what is proven.

Save your progress? Sign up for free

Drill 3: Inventory and Reachability

OpenTofu knows the IPs and renders the Ansible inventory from inventory.tftpl:

tofu output -raw inventory > ~/gw-lab/code/infra-rke2/inventory.yml
cd ~/gw-lab/code/infra-rke2
cat inventory.yml

Expected:

all:
  vars:
    ansible_user: ubuntu
  children:
    rke2_servers:
      hosts:
        rke2-cp-1: { ansible_host: 10.10.20.11 }
        rke2-cp-2: { ansible_host: 10.10.20.12 }
        rke2-cp-3: { ansible_host: 10.10.20.13 }
    rke2_agents:
      hosts:
        rke2-wk-1: { ansible_host: 10.10.20.21 }
        rke2-wk-2: { ansible_host: 10.10.20.22 }
        rke2-wk-3: { ansible_host: 10.10.20.23 }
ansible-galaxy collection install -r requirements.yml
ansible all -m ansible.builtin.ping -o
ansible all -a 'ip -br -4 addr show eth0' -o

Expected (shortened, may differ depending on the version):

rke2-cp-1 | SUCCESS => {"changed": false,"ping": "pong"}
...
rke2-wk-3 | CHANGED | rc=0 | (stdout) eth0  UP  10.10.20.23/24

Why eth0? The network configuration Proxmox delivers via cloud-init names the NIC eth0. Exactly this name is set in group_vars/all.yml as rke2_iface – kube-vip and Cilium need it.

Save your progress? Sign up for free

Drill 4: Understanding the RKE2 Configuration

cat group_vars/all.yml

Expected:

rke2_version: v1.36.5+rke2r1
rke2_vip: 10.10.20.10
rke2_api_fqdn: rke2.lab.internal
rke2_iface: eth0             # Proxmox-Cloud-Init benennt die NIC eth0
# Lab-Wert! Echt: per ansible-vault oder Umgebungsvariable
rke2_token: >-
  {{ lookup('ansible.builtin.env', 'RKE2_TOKEN')
     | default('rke2-lab-token-1234', true) }}
kube_vip_version: v1.2.4
# Tag 9 setzt das auf "true", sobald die Gateway-API-CRDs da sind
cilium_gateway_class_create: auto
rke2_first_server: "{{ groups['rke2_servers'][0] }}"

The server configuration (roles/rke2_server/templates/config.yaml.j2):

token: {{ rke2_token }}
{% if inventory_hostname != rke2_first_server %}
server: https://{{ rke2_vip }}:9345
{% endif %}
node-ip: {{ ansible_host }}
tls-san:
  - {{ rke2_vip }}
  - {{ rke2_api_fqdn }}
cni: cilium
disable-kube-proxy: true
ingress-controller: none
node-taint:
  - "CriticalAddonsOnly=true:NoExecute"
write-kubeconfig-mode: "0600"
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 8
Key Effect
tls-san The API certificate is also valid for the VIP and rke2.lab.internal
server missing only on rke2-cp-1 – it initializes etcd
disable-kube-proxy no kube-proxy pod, Cilium takes over Services
ingress-controller: none no Traefik, no foreign Gateway API CRDs

Ansible places the Cilium values as a HelmChartConfig in /var/lib/rancher/rke2/server/manifests/ – RKE2 merges them into its chart rke2-cilium:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    kubeProxyReplacement: true
    k8sServiceHost: localhost        # RKE2 supervisor LB on every node
    k8sServicePort: 6443
    devices: {{ rke2_iface }}
    l2announcements:
      enabled: true
    externalIPs:
      enabled: true
    k8sClientRateLimit:              # L2 leases need API budget
      qps: 20
      burst: 40
    gatewayAPI:
      enabled: true
      gatewayClass:
        create: "{{ cilium_gateway_class_create }}"
    operator:
      replicas: 2
      rollOutPods: true
    rollOutCiliumPods: true

Why localhost:6443? Without kube-proxy, there is no working kubernetes Service at startup. But every RKE2 node runs a local load balancer on 127.0.0.1:6443 that forwards to the servers – Cilium talks to that.

Save your progress? Sign up for free

Drill 5: kube-vip for the API VIP

kube-vip runs as a DaemonSet on the servers only and is also placed as an auto-deploy manifest. Excerpt from roles/rke2_server/templates/kube-vip.yaml.j2:

      containers:
        - name: kube-vip
          image: ghcr.io/kube-vip/kube-vip:{{ kube_vip_version }}
          args: ["manager"]
          env:
            - { name: vip_arp, value: "true" }
            - { name: port, value: "6443" }
            - { name: vip_interface, value: "{{ rke2_iface }}" }
            - { name: vip_subnet, value: "32" }
            - { name: cp_enable, value: "true" }
            - { name: cp_namespace, value: kube-system }
            - { name: svc_enable, value: "false" }
            - { name: vip_leaderelection, value: "true" }
            - { name: vip_leaseduration, value: "5" }
            - { name: vip_renewdeadline, value: "3" }
            - { name: vip_retryperiod, value: "1" }
            - { name: address, value: "{{ rke2_vip }}" }

The bootstrap sequence: rke2-cp-1 starts without server:, the manifest creates the DaemonSet, kube-vip wins the lease plndr-cp-lock and answers ARP for 10.10.20.10. Only then can rke2-cp-2/3 and the agents join via https://10.10.20.10:9345 – that is why the role waits for port 9345 on the VIP.

Why svc_enable: "false"? Otherwise kube-vip and Cilium LB IPAM would compete for the same LoadBalancer Services.

Before the run you check statically:

ansible-playbook --syntax-check site.yml
ansible-lint

Expected:

playbook: site.yml
Passed: 0 failure(s), 0 warning(s) in 16 files processed of 17 encountered.
Last profile that met the validation criteria was 'production'.

Save your progress? Sign up for free

Drill 6: Run the Playbook

export RKE2_TOKEN=$(openssl rand -hex 32)
time ansible-playbook site.yml

Expected (shortened, may differ depending on the version):

PLAY [Erster Server (cluster-init)] ****************************************
TASK [rke2_server : Warten, bis der Node Ready ist] ************************
FAILED - RETRYING: [rke2-cp-1]: Warten, bis der Node Ready ist (20 retries left).
ok: [rke2-cp-1]
TASK [rke2_server : Warten, bis die VIP den Supervisor-Port bedient] *******
ok: [rke2-cp-1]
PLAY [Weitere Server, einzeln (etcd-Quorum!)] ******************************
...
PLAY RECAP *****************************************************************
rke2-cp-1 : ok=14  changed=8  unreachable=0  failed=0  skipped=0
rke2-wk-3 : ok=8   changed=5  unreachable=0  failed=0  skipped=0
real    11m42s

Why serial: 1 for the additional servers? If two new etcd members join at the same time, the quorum can briefly tip over. One at a time is robust.

The last play fetches rke2.yaml, replaces 127.0.0.1 with the VIP and the name default with rke2-lab:

export KUBECONFIG=~/.kube/rke2-lab.yaml
kubectl config current-context
kubectl get nodes -o wide

Expected (shortened, may differ depending on the version):

rke2-lab
NAME        STATUS   ROLES                AGE   VERSION          INTERNAL-IP
rke2-cp-1   Ready    control-plane,etcd   14m   v1.36.5+rke2r1   10.10.20.11
rke2-cp-2   Ready    control-plane,etcd   10m   v1.36.5+rke2r1   10.10.20.12
rke2-cp-3   Ready    control-plane,etcd   8m    v1.36.5+rke2r1   10.10.20.13
rke2-wk-1   Ready    <none>               5m    v1.36.5+rke2r1   10.10.20.21
rke2-wk-2   Ready    <none>               5m    v1.36.5+rke2r1   10.10.20.22
rke2-wk-3   Ready    <none>               5m    v1.36.5+rke2r1   10.10.20.23

Save your progress? Sign up for free

Drill 7: Check Cilium and the VIP

kubectl -n kube-system get ds kube-proxy 2>&1 | tail -1
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status | grep -E 'KubeProxyReplacement|Cluster health'
kubectl -n kube-system get lease plndr-cp-lock \
  -o jsonpath='{.spec.holderIdentity}{"\n"}'

Expected (shortened, may differ depending on the version):

Error from server (NotFound): daemonsets.apps "kube-proxy" not found
KubeProxyReplacement:    True   [eth0   10.10.20.21 fe80::... (Direct Routing)]
Cluster health:          6/6 reachable   (2026-10-09T10:12:31Z)
rke2-cp-1

Gateway API is enabled in Cilium, but the CRDs are still missing. The operator notices this at startup, reports an error and leaves the controller off – Cilium itself keeps running normally:

kubectl -n kube-system logs deploy/cilium-operator | grep -i gateway | head -2
kubectl get gatewayclass 2>&1

Expected (shortened, may differ depending on the version):

level=error msg="Required GatewayAPI resources are not found, please refer to docs
  for installation instructions" ...
error: the server doesn't have a resource type "gatewayclass"

That is intended: on Day 9 you install the CRDs and restart the operator – the order is the topic there.

Save your progress? Sign up for free

Checkpoint

export KUBECONFIG=~/.kube/rke2-lab.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "API über VIP erreichbar" \
  'curl -sk https://10.10.20.10:6443/version | grep -q gitVersion'
check "6 Nodes Ready" \
  '[ "$(kubectl get nodes --no-headers | grep -c " Ready ")" -eq 6 ]'
check "3 etcd-Server" \
  '[ "$(kubectl get nodes -l node-role.kubernetes.io/etcd=true \
     --no-headers | wc -l)" -eq 3 ]'
check "Server getaintet" \
  'kubectl get node rke2-cp-1 -o jsonpath="{.spec.taints}" \
     | grep -q CriticalAddonsOnly'
check "kein kube-proxy" '! kubectl -n kube-system get ds kube-proxy'
check "Cilium-DaemonSet vollständig" \
  'kubectl -n kube-system rollout status ds/cilium --timeout=10s'
check "kube-vip-Lease vergeben" \
  'kubectl -n kube-system get lease plndr-cp-lock \
     -o jsonpath="{.spec.holderIdentity}" | grep -q rke2-cp'
check "kein Traefik" '! kubectl -n kube-system get deploy rke2-traefik'

Challenge

Prove that the API VIP is fail-safe: hard-stop the current lease holder and measure how long kubectl get nodes via the VIP does not respond. Start the VM again afterwards.

Hint: kube-vip’s lease duration is 5 s, qm stop simulates a power failure.

Solution

holder=$(kubectl -n kube-system get lease plndr-cp-lock \
  -o jsonpath='{.spec.holderIdentity}')
vmid=$(ssh [email protected] \
  "qm list | awk '\$2==\"$holder\"{print \$1}'")
ssh [email protected] "qm stop $vmid"
start=$(date +%s)
until kubectl get nodes --request-timeout=2s >/dev/null 2>&1; do sleep 1; done
echo "API wieder da nach $(( $(date +%s) - start )) s"
kubectl -n kube-system get lease plndr-cp-lock \
  -o jsonpath='{.spec.holderIdentity}{"\n"}'
ssh [email protected] "qm start $vmid"

Expected (shortened, may differ depending on the version):

API wieder da nach 9 s
rke2-cp-2

With two of three etcd members the quorum is preserved; only the VIP moves.

Save your progress? Sign up for free

Enterprise Note

  • Build the template and VMs via a pipeline (Packer or an image factory), tag the image version; token with --privsep 1 and ACLs only on pool/storage.
  • rke2_token from Vault/OpenBao or ansible-vault, never in the repo; check the RKE2 CIS profile (profile: cis) and Pod Security.
  • Instead of an ARP VIP, often an external load balancer (F5, HAProxy pair) or BGP – kube-vip ARP needs a flat L2 segment.
  • etcd snapshots to S3 (etcd-s3: true) and practice restores regularly – Day 14 takes a snapshot as a safety net before the drills.
  • Upgrades with the System Upgrade Controller and Plan resources instead of ad-hoc scripts (see docs.rke2.io, “Automated Upgrades”).