Skip to content
Hands-on IT IT Drills

Day 7: Six VMs with OpenTofu

Goal: A single tofu apply clones the six RKE2 VMs from template 9000 with fixed IPs, an SSH key and a passed-through GPU, and writes the Ansible inventory. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • Provider bpg/proxmox (v0.116), resource proxmox_virtual_environment_vm with a clone block. According to the docs, the new proxmox_vm/proxmox_cloned_vm are experimental and support neither cloud-init nor passthrough.
  • A map nodes plus for_each creates all six VMs. The sizes come from sizes per class server, agent, gpu.
  • cloud-init: IP/DNS via ip_config; user, packages and hostname via a dedicated user-data file (snippet) per VM.
  • Only rke2-gpu-1 gets q35 + OVMF + hostpci with mapping gpu0.
  • tofu output -raw ansible_inventory provides the inventory for Day 8.
nodes{6} --for_each--> user-data snippet  (local:snippets/<name>-user.yaml)
                   \--> VM clone of 9000   (VMID 211..231, 10.10.20.x/24)
                          rke2-gpu-1: q35, ovmf, efi_disk, hostpci0=gpu0

Drill 1: Project and provider

mkdir -p ~/ki-lab/code/infra-proxmox/cloud-init
cd ~/ki-lab/code/infra-proxmox
cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.8"

  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~> 0.116"
    }
  }
}
EOF
cat > provider.tf <<'EOF'
# Credentials only from the environment:
#   PROXMOX_VE_ENDPOINT  = https://10.10.20.5:8006/
#   PROXMOX_VE_API_TOKEN = tofu@pve!provider=<secret>
provider "proxmox" {
  insecure = var.pve_insecure

  # Snippet upload (cloud-init user-data) only works via SSH
  ssh {
    agent    = true
    username = "tofu"
  }
}
EOF
tofu init

Expected:

Initializing provider plugins...
- Finding bpg/proxmox versions matching "~> 0.116"...
- Installing bpg/proxmox v0.116.0...
- Installed bpg/proxmox v0.116.0 (signed, key ID 0B3405B36193A495)
...
OpenTofu has been successfully initialized!

Endpoint and token aren’t in the code. The provider reads PROXMOX_VE_ENDPOINT and PROXMOX_VE_API_TOKEN from ~/ki-lab/pve.env (Day 6). The ssh block uses your ssh-agent and the user tofu – only for the snippet upload.

Save your progress? Sign up for free

Drill 2: Variables – six nodes, three sizes

cat > variables.tf <<'EOF'
variable "pve_node" {
  description = "Proxmox-Node mit Template und den CPU-VMs"
  type        = string
  default     = "pve1"
}

variable "gpu_pve_node" {
  description = "Proxmox-Node mit der GPU, null = pve_node"
  type        = string
  default     = null
}

variable "pve_insecure" {
  description = "Selbstsigniertes PVE-Zertifikat akzeptieren (nur Lab)"
  type        = bool
  default     = true
}

variable "template_id" {
  type    = number
  default = 9000
}

variable "vm_datastore" {
  type    = string
  default = "local-lvm"
}

variable "snippet_datastore" {
  type    = string
  default = "local"
}

variable "bridge" {
  type    = string
  default = "vmbr0"
}

variable "vlan_id" {
  description = "VLAN-Tag für das K8s-Netz, null = untagged"
  type        = number
  default     = null
}

variable "gateway" {
  type    = string
  default = "10.10.20.1"
}

variable "dns_servers" {
  type    = list(string)
  default = ["10.10.20.1"]
}

variable "domain" {
  type    = string
  default = "lab.internal"
}

variable "ssh_public_key_file" {
  type    = string
  default = "~/.ssh/id_ed25519.pub"
}

variable "gpu_mapping" {
  description = "PCI-Resource-Mapping aus Tag 6"
  type        = string
  default     = "gpu0"
}

variable "nodes" {
  description = "Die sechs Cluster-VMs: VMID, IP, Klasse"
  type = map(object({
    vmid  = number
    ip    = string
    class = string # server | agent | gpu
  }))
  default = {
    "rke2-cp-1"  = { vmid = 211, ip = "10.10.20.11", class = "server" }
    "rke2-cp-2"  = { vmid = 212, ip = "10.10.20.12", class = "server" }
    "rke2-cp-3"  = { vmid = 213, ip = "10.10.20.13", class = "server" }
    "rke2-wk-1"  = { vmid = 221, ip = "10.10.20.21", class = "agent" }
    "rke2-wk-2"  = { vmid = 222, ip = "10.10.20.22", class = "agent" }
    "rke2-gpu-1" = { vmid = 231, ip = "10.10.20.31", class = "gpu" }
  }
}

variable "sizes" {
  description = "Ressourcen je Klasse (memory in MiB, disk in GiB)"
  type = map(object({
    cores  = number
    memory = number
    disk   = number
  }))
  default = {
    server = { cores = 4, memory = 8192, disk = 60 }
    agent  = { cores = 8, memory = 16384, disk = 100 }
    gpu    = { cores = 8, memory = 32768, disk = 200 }
  }
}
EOF

Why class instead of sizes per node? A third worker is then a single line, and rules like “the GPU VM gets q35” depend on class == "gpu" instead of on a name.

Tip: If the GPU is in a second Proxmox host (Day 0), set gpu_pve_node = "pve2". The clone is then created on pve1 and migrated – with 200 GB and no shared storage, that takes a few minutes.

Save your progress? Sign up for free

Drill 3: cloud-init user-data

cat > cloud-init/user-data.yaml.tftpl <<'EOF'
#cloud-config
hostname: ${hostname}
fqdn: ${hostname}.${domain}
manage_etc_hosts: true
timezone: Europe/Berlin
users:
  - name: ubuntu
    groups: [sudo]
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: true
    ssh_authorized_keys:
      - ${ssh_key}
package_update: true
packages:
  - qemu-guest-agent
  - open-iscsi
  - nfs-common
  - curl
runcmd:
  - systemctl enable --now qemu-guest-agent iscsid
EOF

qemu-guest-agent reports the IP back to Proxmox (the provider waits for it). Longhorn needs open-iscsi and nfs-common on Day 9. Swap is off in the cloud image anyway, so RKE2 is happy.

Warning: user_data_file_id replaces the user-data generated by Proxmox completely – that’s why hostname, user and SSH key are here and not in user_account. Proxmox still generates the network (ip_config) and DNS itself.

Save your progress? Sign up for free

Drill 4: The VMs with for_each

cat > main.tf <<'EOF'
locals {
  ssh_key  = trimspace(file(pathexpand(var.ssh_public_key_file)))
  gpu_host = coalesce(var.gpu_pve_node, var.pve_node)
  host = {
    for k, v in var.nodes : k => v.class == "gpu" ? local.gpu_host : var.pve_node
  }
}

# One user-data file per VM (it contains the hostname)
resource "proxmox_virtual_environment_file" "user_data" {
  for_each     = var.nodes
  content_type = "snippets"
  datastore_id = var.snippet_datastore
  node_name    = local.host[each.key]

  source_raw {
    file_name = "${each.key}-user.yaml"
    data = templatefile("${path.module}/cloud-init/user-data.yaml.tftpl", {
      hostname = each.key
      domain   = var.domain
      ssh_key  = local.ssh_key
    })
  }
}

resource "proxmox_virtual_environment_vm" "node" {
  for_each  = var.nodes
  name      = each.key
  vm_id     = each.value.vmid
  node_name = local.host[each.key]
  tags      = ["rke2", each.value.class]
  on_boot   = true

  clone {
    vm_id     = var.template_id
    node_name = var.pve_node # the template lives here
    full      = true
    retries   = 3
  }

  # GPU VM: q35 + UEFI so that the card appears as a PCIe device
  machine = each.value.class == "gpu" ? "q35" : "pc"
  bios    = each.value.class == "gpu" ? "ovmf" : "seabios"

  dynamic "efi_disk" {
    for_each = each.value.class == "gpu" ? [1] : []
    content {
      datastore_id = var.vm_datastore
      type         = "4m"
    }
  }

  cpu {
    type  = "host"
    cores = var.sizes[each.value.class].cores
  }

  memory {
    dedicated = var.sizes[each.value.class].memory
    floating  = 0 # no ballooning (mandatory with passthrough)
  }

  agent {
    enabled = true
  }

  scsi_hardware = "virtio-scsi-single" # prerequisite for iothread

  disk {
    datastore_id = var.vm_datastore
    interface    = "scsi0"
    size         = var.sizes[each.value.class].disk
    discard      = "on"
    iothread     = true
  }

  network_device {
    bridge  = var.bridge
    vlan_id = var.vlan_id
  }

  dynamic "hostpci" {
    for_each = each.value.class == "gpu" ? [var.gpu_mapping] : []
    content {
      device  = "hostpci0"
      mapping = hostpci.value
      pcie    = true
      rombar  = true
    }
  }

  initialization {
    datastore_id = var.vm_datastore
    dns {
      domain  = var.domain
      servers = var.dns_servers
    }
    ip_config {
      ipv4 {
        address = "${each.value.ip}/24"
        gateway = var.gateway
      }
    }
    user_data_file_id = proxmox_virtual_environment_file.user_data[each.key].id
  }
}
EOF
tofu fmt -check && tofu validate

Expected:

Success! The configuration is valid.

Why like this?

  • clone.full = true: independent disks; the template can be rebuilt later.
  • disk.size enlarges the template’s 3.5 GB disk. cloud-init (growpart) grows the file system on first boot.
  • memory.floating = 0 disables ballooning. With passthrough, RAM must be fully and statically assigned anyway (DMA), and it’s better for etcd too.
  • cpu.type = "host": vLLM and PyTorch want to see AVX2/AVX-512; x86-64-v2-AES (the Proxmox default) isn’t enough for that.

Save your progress? Sign up for free

Drill 5: Check the GPU VM in the plan

A terraform.tfvars for your deviations (node name, storage, VLAN):

cat > terraform.tfvars.example <<'EOF'
pve_node            = "pve1"
vm_datastore        = "local-lvm"
snippet_datastore   = "local"
bridge              = "vmbr0"
# vlan_id           = 20
ssh_public_key_file = "~/.ssh/id_ed25519.pub"
gpu_mapping         = "gpu0"
EOF
cp -n terraform.tfvars.example terraform.tfvars

Then the plan:

source ~/ki-lab/pve.env
ssh-add -l >/dev/null || ssh-add ~/.ssh/id_ed25519
tofu plan -out=tag07.plan | tail -1
tofu show -no-color tag07.plan \
  | sed -n '/node\["rke2-gpu-1"\] will be created/,/^$/p' \
  | grep -E ' (bios|machine|name|vm_id|mapping|pcie|dedicated|floating) '

Expected (generated against a dummy endpoint, order may differ):

Plan: 12 to add, 0 to change, 0 to destroy.
      + bios                                 = "ovmf"
      + machine                              = "q35"
      + name                                 = "rke2-gpu-1"
      + vm_id                                = 231
          + mapping = "gpu0"
          + pcie    = true
          + dedicated      = 32768
          + floating       = 0

Twelve resources: six snippets, six VMs. Why q35 + OVMF only here? pcie = true exists only with q35, and modern NVIDIA cards need a UEFI guest for large BARs (16–48 GB VRAM). The other VMs stay on SeaBIOS – fewer moving parts.

Save your progress? Sign up for free

Drill 6: Apply

tofu apply tag07.plan

Expected (abbreviated, not tested live, duration depends on storage):

proxmox_virtual_environment_file.user_data["rke2-cp-1"]: Creation complete after 1s
...
proxmox_virtual_environment_vm.node["rke2-gpu-1"]: Creation complete after 3m12s
Apply complete! Resources: 12 added, 0 changed, 0 destroyed.

Cross-check on the Proxmox host:

qm list | grep rke2
qm config 231 | grep -E '^(bios|machine|hostpci0|efidisk0|memory|balloon):'

Expected (abbreviated, may differ depending on version):

       211 rke2-cp-1            running    8192              60.00 4321
       ...
       231 rke2-gpu-1           running    32768            200.00 4377
balloon: 0
bios: ovmf
efidisk0: local-lvm:vm-231-disk-1,efitype=4m,size=4M
hostpci0: mapping=gpu0,pcie=1
machine: q35
memory: 32768

Warning: If rke2-gpu-1 starts with “IOMMU group not viable” or “device is in use”, something from Day 6 isn’t right (Drill 6/7). dmesg | grep vfio on the host shows the reason.

Save your progress? Sign up for free

Drill 7: Outputs and Ansible inventory

cat > outputs.tf <<'EOF'
output "nodes" {
  description = "Name -> IP aller VMs"
  value       = { for k, v in var.nodes : k => v.ip }
}

# Ansible inventory for Day 8:
#   tofu output -raw ansible_inventory > ../infra-rke2/inventory.yaml
output "ansible_inventory" {
  value = yamlencode({
    all = {
      vars = { ansible_user = "ubuntu" }
      children = {
        rke2_servers = {
          hosts = {
            for k, v in var.nodes : k => { ansible_host = v.ip }
            if v.class == "server"
          }
        }
        rke2_agents = {
          hosts = {
            for k, v in var.nodes : k => { ansible_host = v.ip }
            if v.class != "server"
          }
        }
        gpu_nodes = {
          hosts = {
            for k, v in var.nodes : k => { ansible_host = v.ip }
            if v.class == "gpu"
          }
        }
      }
    }
  })
}
EOF
tofu apply -auto-approve | grep '^Apply'
mkdir -p ../infra-rke2
tofu output -raw ansible_inventory > ../infra-rke2/inventory.yaml
cat ../infra-rke2/inventory.yaml

Expected:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
"all":
  "children":
    "gpu_nodes":
      "hosts":
        "rke2-gpu-1":
          "ansible_host": "10.10.20.31"
    "rke2_agents":
      "hosts":
        "rke2-gpu-1":
          "ansible_host": "10.10.20.31"
        "rke2-wk-1":
          "ansible_host": "10.10.20.21"
        "rke2-wk-2":
          "ansible_host": "10.10.20.22"
    "rke2_servers":
      "hosts":
        "rke2-cp-1":
          "ansible_host": "10.10.20.11"
        "rke2-cp-2":
          "ansible_host": "10.10.20.12"
        "rke2-cp-3":
          "ansible_host": "10.10.20.13"
  "vars":
    "ansible_user": "ubuntu"

yamlencode puts all keys in quotes – Ansible doesn’t care. Now the first contact:

cd ../infra-rke2
ANSIBLE_HOST_KEY_CHECKING=False \
  ansible all -i inventory.yaml -m ansible.builtin.ping -o
ssh [email protected] 'ip -br link; lspci -nn | grep -i nvidia'

Expected (abbreviated, not tested live):

rke2-cp-1 | SUCCESS => {"changed": false,"ping": "pong"}
...
rke2-gpu-1 | SUCCESS => {"changed": false,"ping": "pong"}
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0             UP             bc:24:11:5e:0a:31 <BROADCAST,MULTICAST,UP,LOWER_UP>
01:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8] (rev a1)

Why is the interface called eth0? The cloud-init network configuration generated by Proxmox renames the NIC via a MAC match. kube-vip and Cilium (Days 8/9) rely on that. If yours is named differently, adjust rke2_vip_interface.

Save your progress? Sign up for free

Checkpoint

cd ~/ki-lab/code/infra-proxmox
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "6 VMs im State" \
  '[ "$(tofu state list | grep -c proxmox_virtual_environment_vm)" -eq 6 ]'
check "Inventory hat 3 Server" \
  '[ "$(grep -c "rke2-cp-" ../infra-rke2/inventory.yaml)" -eq 3 ]'
for ip in 11 12 13 21 22 31; do
  check "SSH 10.10.20.$ip" "ssh -o BatchMode=yes -o ConnectTimeout=5 \
    [email protected].$ip true"
done
check "GPU in rke2-gpu-1 sichtbar" \
  'ssh [email protected] "lspci -nn | grep -q 10de:"'
check "guest agent auf cp-1" \
  'ssh [email protected] "systemctl is-active qemu-guest-agent"'

Expected:

✔ 6 VMs im State
✔ Inventory hat 3 Server
✔ SSH 10.10.20.11
✔ SSH 10.10.20.12
✔ SSH 10.10.20.13
✔ SSH 10.10.20.21
✔ SSH 10.10.20.22
✔ SSH 10.10.20.31
✔ GPU in rke2-gpu-1 sichtbar
✔ guest agent auf cp-1

Challenge

Replace rke2-wk-2 specifically with a fresh VM without touching the other five, and show in the plan beforehand that exactly two resources are recreated.

Hint: tofu plan -replace=ADDRESS can be given multiple times; the snippet resource belongs to it.

Solution

cd ~/ki-lab/code/infra-proxmox
tofu plan \
  -replace='proxmox_virtual_environment_vm.node["rke2-wk-2"]' \
  -replace='proxmox_virtual_environment_file.user_data["rke2-wk-2"]' \
  -out=replace.plan | tail -1
tofu apply replace.plan
ssh-keygen -R 10.10.20.22

Expected (abbreviated):

Plan: 2 to add, 0 to change, 2 to destroy.

If the cluster from Day 8 already exists, first run kubectl drain rke2-wk-2 --ignore-daemonsets --delete-emptydir-data and afterwards kubectl delete node rke2-wk-2, then run the rke2_agent role again for this host only (--limit rke2-wk-2).

Save your progress? Sign up for free

Enterprise Note

  • State belongs in a remote backend with locking and encryption (OpenTofu encryption block, S3/MinIO or GitLab HTTP backend), not on your laptop.
  • Snippets via SSH are a second access path with its own attack surface. Alternative: generate the cloud-init ISO externally or prebake images with Packer; then the API is enough.
  • insecure = true in the lab only. In production, Proxmox gets a certificate from the internal CA, and the provider verifies it.
  • Don’t hard-code IPs: use IPAM (NetBox, phpIPAM) as a data source and create DNS records in the same run.
  • A separate workspace/folder and separate tokens per environment. prevent_destroy on control-plane VMs prevents a wrong apply from deleting etcd.