Day 7: Six VMs with OpenTofu
Goal: A single
tofu applyclones the six RKE2 VMs from template 9000 with fixed IPs, an SSH key and a passed-through GPU, and writes the Ansible inventory. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- Provider
bpg/proxmox(v0.116), resourceproxmox_virtual_environment_vmwith acloneblock. According to the docs, the newproxmox_vm/proxmox_cloned_vmare experimental and support neither cloud-init nor passthrough. - A map
nodesplusfor_eachcreates all six VMs. The sizes come fromsizesper classserver,agent,gpu. - cloud-init: IP/DNS via
ip_config; user, packages and hostname via a dedicated user-data file (snippet) per VM. - Only
rke2-gpu-1getsq35+ OVMF +hostpciwith mappinggpu0. tofu output -raw ansible_inventoryprovides the inventory for Day 8.
nodes{6} --for_each--> user-data snippet (local:snippets/<name>-user.yaml)
\--> VM clone of 9000 (VMID 211..231, 10.10.20.x/24)
rke2-gpu-1: q35, ovmf, efi_disk, hostpci0=gpu0
Drill 1: Project and provider
mkdir -p ~/ki-lab/code/infra-proxmox/cloud-init
cd ~/ki-lab/code/infra-proxmox
cat > versions.tf <<'EOF'
terraform {
required_version = ">= 1.8"
required_providers {
proxmox = {
source = "bpg/proxmox"
version = "~> 0.116"
}
}
}
EOF
cat > provider.tf <<'EOF'
# Credentials only from the environment:
# PROXMOX_VE_ENDPOINT = https://10.10.20.5:8006/
# PROXMOX_VE_API_TOKEN = tofu@pve!provider=<secret>
provider "proxmox" {
insecure = var.pve_insecure
# Snippet upload (cloud-init user-data) only works via SSH
ssh {
agent = true
username = "tofu"
}
}
EOF
tofu initExpected:
Initializing provider plugins...
- Finding bpg/proxmox versions matching "~> 0.116"...
- Installing bpg/proxmox v0.116.0...
- Installed bpg/proxmox v0.116.0 (signed, key ID 0B3405B36193A495)
...
OpenTofu has been successfully initialized!
Endpoint and token aren’t in the code. The provider reads
PROXMOX_VE_ENDPOINT and PROXMOX_VE_API_TOKEN
from ~/ki-lab/pve.env (Day 6). The ssh block
uses your ssh-agent and the user tofu – only for the
snippet upload.
Save your progress? Sign up for free
Drill 2: Variables – six nodes, three sizes
cat > variables.tf <<'EOF'
variable "pve_node" {
description = "Proxmox-Node mit Template und den CPU-VMs"
type = string
default = "pve1"
}
variable "gpu_pve_node" {
description = "Proxmox-Node mit der GPU, null = pve_node"
type = string
default = null
}
variable "pve_insecure" {
description = "Selbstsigniertes PVE-Zertifikat akzeptieren (nur Lab)"
type = bool
default = true
}
variable "template_id" {
type = number
default = 9000
}
variable "vm_datastore" {
type = string
default = "local-lvm"
}
variable "snippet_datastore" {
type = string
default = "local"
}
variable "bridge" {
type = string
default = "vmbr0"
}
variable "vlan_id" {
description = "VLAN-Tag für das K8s-Netz, null = untagged"
type = number
default = null
}
variable "gateway" {
type = string
default = "10.10.20.1"
}
variable "dns_servers" {
type = list(string)
default = ["10.10.20.1"]
}
variable "domain" {
type = string
default = "lab.internal"
}
variable "ssh_public_key_file" {
type = string
default = "~/.ssh/id_ed25519.pub"
}
variable "gpu_mapping" {
description = "PCI-Resource-Mapping aus Tag 6"
type = string
default = "gpu0"
}
variable "nodes" {
description = "Die sechs Cluster-VMs: VMID, IP, Klasse"
type = map(object({
vmid = number
ip = string
class = string # server | agent | gpu
}))
default = {
"rke2-cp-1" = { vmid = 211, ip = "10.10.20.11", class = "server" }
"rke2-cp-2" = { vmid = 212, ip = "10.10.20.12", class = "server" }
"rke2-cp-3" = { vmid = 213, ip = "10.10.20.13", class = "server" }
"rke2-wk-1" = { vmid = 221, ip = "10.10.20.21", class = "agent" }
"rke2-wk-2" = { vmid = 222, ip = "10.10.20.22", class = "agent" }
"rke2-gpu-1" = { vmid = 231, ip = "10.10.20.31", class = "gpu" }
}
}
variable "sizes" {
description = "Ressourcen je Klasse (memory in MiB, disk in GiB)"
type = map(object({
cores = number
memory = number
disk = number
}))
default = {
server = { cores = 4, memory = 8192, disk = 60 }
agent = { cores = 8, memory = 16384, disk = 100 }
gpu = { cores = 8, memory = 32768, disk = 200 }
}
}
EOFWhy class instead of sizes per node? A third worker is
then a single line, and rules like “the GPU VM gets q35” depend on
class == "gpu" instead of on a name.
Tip: If the GPU is in a second Proxmox host (Day 0), set
gpu_pve_node = "pve2". The clone is then created onpve1and migrated – with 200 GB and no shared storage, that takes a few minutes.
Save your progress? Sign up for free
Drill 3: cloud-init user-data
cat > cloud-init/user-data.yaml.tftpl <<'EOF'
#cloud-config
hostname: ${hostname}
fqdn: ${hostname}.${domain}
manage_etc_hosts: true
timezone: Europe/Berlin
users:
- name: ubuntu
groups: [sudo]
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
lock_passwd: true
ssh_authorized_keys:
- ${ssh_key}
package_update: true
packages:
- qemu-guest-agent
- open-iscsi
- nfs-common
- curl
runcmd:
- systemctl enable --now qemu-guest-agent iscsid
EOFqemu-guest-agent reports the IP back to Proxmox (the
provider waits for it). Longhorn needs open-iscsi and
nfs-common on Day 9. Swap is off in the cloud image anyway,
so RKE2 is happy.
Warning:
user_data_file_idreplaces the user-data generated by Proxmox completely – that’s why hostname, user and SSH key are here and not inuser_account. Proxmox still generates the network (ip_config) and DNS itself.
Save your progress? Sign up for free
Drill 4: The VMs with for_each
cat > main.tf <<'EOF'
locals {
ssh_key = trimspace(file(pathexpand(var.ssh_public_key_file)))
gpu_host = coalesce(var.gpu_pve_node, var.pve_node)
host = {
for k, v in var.nodes : k => v.class == "gpu" ? local.gpu_host : var.pve_node
}
}
# One user-data file per VM (it contains the hostname)
resource "proxmox_virtual_environment_file" "user_data" {
for_each = var.nodes
content_type = "snippets"
datastore_id = var.snippet_datastore
node_name = local.host[each.key]
source_raw {
file_name = "${each.key}-user.yaml"
data = templatefile("${path.module}/cloud-init/user-data.yaml.tftpl", {
hostname = each.key
domain = var.domain
ssh_key = local.ssh_key
})
}
}
resource "proxmox_virtual_environment_vm" "node" {
for_each = var.nodes
name = each.key
vm_id = each.value.vmid
node_name = local.host[each.key]
tags = ["rke2", each.value.class]
on_boot = true
clone {
vm_id = var.template_id
node_name = var.pve_node # the template lives here
full = true
retries = 3
}
# GPU VM: q35 + UEFI so that the card appears as a PCIe device
machine = each.value.class == "gpu" ? "q35" : "pc"
bios = each.value.class == "gpu" ? "ovmf" : "seabios"
dynamic "efi_disk" {
for_each = each.value.class == "gpu" ? [1] : []
content {
datastore_id = var.vm_datastore
type = "4m"
}
}
cpu {
type = "host"
cores = var.sizes[each.value.class].cores
}
memory {
dedicated = var.sizes[each.value.class].memory
floating = 0 # no ballooning (mandatory with passthrough)
}
agent {
enabled = true
}
scsi_hardware = "virtio-scsi-single" # prerequisite for iothread
disk {
datastore_id = var.vm_datastore
interface = "scsi0"
size = var.sizes[each.value.class].disk
discard = "on"
iothread = true
}
network_device {
bridge = var.bridge
vlan_id = var.vlan_id
}
dynamic "hostpci" {
for_each = each.value.class == "gpu" ? [var.gpu_mapping] : []
content {
device = "hostpci0"
mapping = hostpci.value
pcie = true
rombar = true
}
}
initialization {
datastore_id = var.vm_datastore
dns {
domain = var.domain
servers = var.dns_servers
}
ip_config {
ipv4 {
address = "${each.value.ip}/24"
gateway = var.gateway
}
}
user_data_file_id = proxmox_virtual_environment_file.user_data[each.key].id
}
}
EOF
tofu fmt -check && tofu validateExpected:
Success! The configuration is valid.
Why like this?
clone.full = true: independent disks; the template can be rebuilt later.disk.sizeenlarges the template’s 3.5 GB disk. cloud-init (growpart) grows the file system on first boot.memory.floating = 0disables ballooning. With passthrough, RAM must be fully and statically assigned anyway (DMA), and it’s better for etcd too.cpu.type = "host": vLLM and PyTorch want to see AVX2/AVX-512;x86-64-v2-AES(the Proxmox default) isn’t enough for that.
Save your progress? Sign up for free
Drill 5: Check the GPU VM in the plan
A terraform.tfvars for your deviations (node name,
storage, VLAN):
cat > terraform.tfvars.example <<'EOF'
pve_node = "pve1"
vm_datastore = "local-lvm"
snippet_datastore = "local"
bridge = "vmbr0"
# vlan_id = 20
ssh_public_key_file = "~/.ssh/id_ed25519.pub"
gpu_mapping = "gpu0"
EOF
cp -n terraform.tfvars.example terraform.tfvarsThen the plan:
source ~/ki-lab/pve.env
ssh-add -l >/dev/null || ssh-add ~/.ssh/id_ed25519
tofu plan -out=tag07.plan | tail -1
tofu show -no-color tag07.plan \
| sed -n '/node\["rke2-gpu-1"\] will be created/,/^$/p' \
| grep -E ' (bios|machine|name|vm_id|mapping|pcie|dedicated|floating) 'Expected (generated against a dummy endpoint, order may differ):
Plan: 12 to add, 0 to change, 0 to destroy.
+ bios = "ovmf"
+ machine = "q35"
+ name = "rke2-gpu-1"
+ vm_id = 231
+ mapping = "gpu0"
+ pcie = true
+ dedicated = 32768
+ floating = 0
Twelve resources: six snippets, six VMs. Why q35 + OVMF only here?
pcie = true exists only with q35, and modern NVIDIA cards
need a UEFI guest for large BARs (16–48 GB VRAM). The other VMs stay on
SeaBIOS – fewer moving parts.
Save your progress? Sign up for free
Drill 6: Apply
tofu apply tag07.planExpected (abbreviated, not tested live, duration depends on storage):
proxmox_virtual_environment_file.user_data["rke2-cp-1"]: Creation complete after 1s
...
proxmox_virtual_environment_vm.node["rke2-gpu-1"]: Creation complete after 3m12s
Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
Cross-check on the Proxmox host:
qm list | grep rke2
qm config 231 | grep -E '^(bios|machine|hostpci0|efidisk0|memory|balloon):'Expected (abbreviated, may differ depending on version):
211 rke2-cp-1 running 8192 60.00 4321
...
231 rke2-gpu-1 running 32768 200.00 4377
balloon: 0
bios: ovmf
efidisk0: local-lvm:vm-231-disk-1,efitype=4m,size=4M
hostpci0: mapping=gpu0,pcie=1
machine: q35
memory: 32768
Warning: If
rke2-gpu-1starts with “IOMMU group not viable” or “device is in use”, something from Day 6 isn’t right (Drill 6/7).dmesg | grep vfioon the host shows the reason.
Save your progress? Sign up for free
Drill 7: Outputs and Ansible inventory
cat > outputs.tf <<'EOF'
output "nodes" {
description = "Name -> IP aller VMs"
value = { for k, v in var.nodes : k => v.ip }
}
# Ansible inventory for Day 8:
# tofu output -raw ansible_inventory > ../infra-rke2/inventory.yaml
output "ansible_inventory" {
value = yamlencode({
all = {
vars = { ansible_user = "ubuntu" }
children = {
rke2_servers = {
hosts = {
for k, v in var.nodes : k => { ansible_host = v.ip }
if v.class == "server"
}
}
rke2_agents = {
hosts = {
for k, v in var.nodes : k => { ansible_host = v.ip }
if v.class != "server"
}
}
gpu_nodes = {
hosts = {
for k, v in var.nodes : k => { ansible_host = v.ip }
if v.class == "gpu"
}
}
}
}
})
}
EOF
tofu apply -auto-approve | grep '^Apply'
mkdir -p ../infra-rke2
tofu output -raw ansible_inventory > ../infra-rke2/inventory.yaml
cat ../infra-rke2/inventory.yamlExpected:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
"all":
"children":
"gpu_nodes":
"hosts":
"rke2-gpu-1":
"ansible_host": "10.10.20.31"
"rke2_agents":
"hosts":
"rke2-gpu-1":
"ansible_host": "10.10.20.31"
"rke2-wk-1":
"ansible_host": "10.10.20.21"
"rke2-wk-2":
"ansible_host": "10.10.20.22"
"rke2_servers":
"hosts":
"rke2-cp-1":
"ansible_host": "10.10.20.11"
"rke2-cp-2":
"ansible_host": "10.10.20.12"
"rke2-cp-3":
"ansible_host": "10.10.20.13"
"vars":
"ansible_user": "ubuntu"
yamlencode puts all keys in quotes – Ansible doesn’t
care. Now the first contact:
cd ../infra-rke2
ANSIBLE_HOST_KEY_CHECKING=False \
ansible all -i inventory.yaml -m ansible.builtin.ping -o
ssh [email protected] 'ip -br link; lspci -nn | grep -i nvidia'Expected (abbreviated, not tested live):
rke2-cp-1 | SUCCESS => {"changed": false,"ping": "pong"}
...
rke2-gpu-1 | SUCCESS => {"changed": false,"ping": "pong"}
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0 UP bc:24:11:5e:0a:31 <BROADCAST,MULTICAST,UP,LOWER_UP>
01:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8] (rev a1)
Why is the interface called eth0? The cloud-init network
configuration generated by Proxmox renames the NIC via a MAC match.
kube-vip and Cilium (Days 8/9) rely on that. If yours is named
differently, adjust rke2_vip_interface.
Save your progress? Sign up for free
Checkpoint
cd ~/ki-lab/code/infra-proxmox
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "6 VMs im State" \
'[ "$(tofu state list | grep -c proxmox_virtual_environment_vm)" -eq 6 ]'
check "Inventory hat 3 Server" \
'[ "$(grep -c "rke2-cp-" ../infra-rke2/inventory.yaml)" -eq 3 ]'
for ip in 11 12 13 21 22 31; do
check "SSH 10.10.20.$ip" "ssh -o BatchMode=yes -o ConnectTimeout=5 \
[email protected].$ip true"
done
check "GPU in rke2-gpu-1 sichtbar" \
'ssh [email protected] "lspci -nn | grep -q 10de:"'
check "guest agent auf cp-1" \
'ssh [email protected] "systemctl is-active qemu-guest-agent"'Expected:
✔ 6 VMs im State
✔ Inventory hat 3 Server
✔ SSH 10.10.20.11
✔ SSH 10.10.20.12
✔ SSH 10.10.20.13
✔ SSH 10.10.20.21
✔ SSH 10.10.20.22
✔ SSH 10.10.20.31
✔ GPU in rke2-gpu-1 sichtbar
✔ guest agent auf cp-1
Challenge
Replace rke2-wk-2 specifically with a fresh VM without
touching the other five, and show in the plan beforehand that exactly
two resources are recreated.
Hint: tofu plan -replace=ADDRESS can be
given multiple times; the snippet resource belongs to it.
Solution
cd ~/ki-lab/code/infra-proxmox
tofu plan \
-replace='proxmox_virtual_environment_vm.node["rke2-wk-2"]' \
-replace='proxmox_virtual_environment_file.user_data["rke2-wk-2"]' \
-out=replace.plan | tail -1
tofu apply replace.plan
ssh-keygen -R 10.10.20.22Expected (abbreviated):
Plan: 2 to add, 0 to change, 2 to destroy.
If the cluster from Day 8 already exists, first run
kubectl drain rke2-wk-2 --ignore-daemonsets --delete-emptydir-data
and afterwards kubectl delete node rke2-wk-2, then run the
rke2_agent role again for this host only
(--limit rke2-wk-2).
Save your progress? Sign up for free
Enterprise Note
- State belongs in a remote backend with locking and encryption
(OpenTofu
encryptionblock, S3/MinIO or GitLab HTTP backend), not on your laptop. - Snippets via SSH are a second access path with its own attack surface. Alternative: generate the cloud-init ISO externally or prebake images with Packer; then the API is enough.
insecure = truein the lab only. In production, Proxmox gets a certificate from the internal CA, and the provider verifies it.- Don’t hard-code IPs: use IPAM (NetBox, phpIPAM) as a data source and create DNS records in the same run.
- A separate workspace/folder and separate tokens per environment.
prevent_destroyon control-plane VMs prevents a wrongapplyfrom deleting etcd.