Skip to content
Hands-on IT IT Drills

Day 6: Preparing Proxmox

Goal: Proxmox has an API token for OpenTofu, an Ubuntu template 9000, and a GPU reserved for a VM via vfio. · Duration: approx. 90 min · Prerequisite: Day 0

Briefing

  • OpenTofu talks to Proxmox via an API token (tofu@pve!provider) with its own role TofuProv, never as root@pam.
  • The bpg/proxmox provider uploads cloud-init snippets via SSH. For that there’s the system user tofu with exactly two sudo rules.
  • All six VMs are created as clones of template 9000 (ubuntu-2404-tpl).
  • GPU passthrough needs IOMMU, vfio modules and a host kernel that keeps its hands off the card. The resource mapping gpu0 decouples the VM from the PCI address.
Laptop (tofu, ssh-agent)
   |  HTTPS :8006  API token tofu@pve!provider
   |  SSH   :22    user tofu (only tee -> snippets/)
   v
pve1  vmbr0 ── 10.10.20.0/24 ── VMs 211-231
      local:snippets   local-lvm: template 9000
      GPU 0000:01:00.0 -> vfio-pci -> mapping gpu0 -> rke2-gpu-1

Warning: The Proxmox and GPU commands in this chapter have been checked against the official docs (pve.proxmox.com for PVE 9, registry.terraform.io/providers/bpg/proxmox v0.116), but not tested live. Outputs are abbreviated and may differ depending on version. The scripts are under code/infra-proxmox/host/.

Drill 1: Check the network – bridge and VLAN

All VMs are attached to vmbr0. If the Kubernetes network lives in its own VLAN, the bridge must be VLAN-aware.

ip -br addr show vmbr0
grep -A8 '^iface vmbr0' /etc/network/interfaces

Expected (abbreviated, may differ depending on version):

vmbr0            UP             10.10.20.5/24
iface vmbr0 inet static
        address 10.10.20.5/24
        gateway 10.10.20.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

For a separate VLAN (e.g., 20), extend the bridge like this and later set vlan_id = 20 in OpenTofu:

        bridge-vlan-aware yes
        bridge-vids 2-4094
ifreload -a
bridge vlan show dev eno1 | head -3

Why? Without bridge-vlan-aware, the bridge silently discards the VMs’ tagged frames. For the lab, untagged on vmbr0 is enough if 10.10.20.0/24 is the management network.

Tip: Reserve 10.10.20.10 (API VIP) and 10.10.20.100–120 (LoadBalancer pool) in your router’s DHCP server; otherwise it will hand them out twice.

Save your progress? Sign up for free

Drill 2: Role, user and API token

The role contains only what cloning, configuring, starting and using mappings require. SDN.Audit/SDN.Use are necessary; otherwise the token can’t see the bridges.

cat > /root/01-tofu-user.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

PRIVS="Datastore.Allocate Datastore.AllocateSpace Datastore.Audit"
PRIVS+=" VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.CPU"
PRIVS+=" VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType"
PRIVS+=" VM.Config.Memory VM.Config.Network VM.Config.Options"
PRIVS+=" VM.PowerMgmt VM.Migrate VM.Console VM.GuestAgent.Audit"
PRIVS+=" Sys.Audit Sys.Console SDN.Audit SDN.Use"
PRIVS+=" Mapping.Audit Mapping.Use"

pveum role add TofuProv -privs "${PRIVS// /,}"
pveum user add tofu@pve --comment "OpenTofu bpg/proxmox"
pveum aclmod / -user tofu@pve -role TofuProv
pveum user token add tofu@pve provider --privsep 0
EOF
bash /root/01-tofu-user.sh

Expected (abbreviated, may differ depending on version):

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ tofu@pve!provider                    │
│ info         │ {"privsep":"0"}                      │
│ value        │ 3f1c...-....-....-....-............  │
└──────────────┴──────────────────────────────────────┘

The value appears only once. --privsep 0 means the token inherits exactly the permissions of tofu@pve, so you maintain ACLs in only one place.

Security: Never write the token into *.tf or terraform.tfvars. It belongs in a file with mode 600 outside of Git.

On the laptop:

cat > ~/ki-lab/pve.env <<'EOF'
export PROXMOX_VE_ENDPOINT='https://10.10.20.5:8006/'
export PROXMOX_VE_API_TOKEN='tofu@pve!provider=HIER-DEN-VALUE-EINFUEGEN'
EOF
chmod 600 ~/ki-lab/pve.env
source ~/ki-lab/pve.env
curl -sk -H "Authorization: PVEAPIToken=${PROXMOX_VE_API_TOKEN}" \
  "${PROXMOX_VE_ENDPOINT}api2/json/version" | jq -r .data.version

Expected (may differ depending on version):

9.1.4

Save your progress? Sign up for free

Drill 3: SSH user and snippets storage

bpg/proxmox uploads snippets (cloud-init user-data) via SSH, because the Proxmox API has no upload for them. The user tofu may run only two precisely described commands via sudo.

cat > /root/02-tofu-ssh.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
apt-get install -y sudo
id tofu >/dev/null 2>&1 || useradd -m -s /bin/bash tofu
install -d -m 700 -o tofu -g tofu /home/tofu/.ssh
cat > /etc/sudoers.d/tofu <<'SUDO'
tofu ALL=(root) NOPASSWD: /usr/sbin/pvesm apiinfo
tofu ALL=(root) NOPASSWD: /usr/bin/tee /var/lib/vz/snippets/[a-zA-Z0-9_][a-zA-Z0-9_.-]*
SUDO
chmod 440 /etc/sudoers.d/tofu
visudo -cf /etc/sudoers.d/tofu
pvesm set local --content iso,vztmpl,backup,snippets
EOF
bash /root/02-tofu-ssh.sh

Expected (abbreviated):

/etc/sudoers.d/tofu: parsed OK

Security: Never allow NOPASSWD: /usr/sbin/qm or /var/lib/vz/*. qm set --hookscript and * with ../ are both root-equivalent – the provider docs say so as well.

Store your laptop key and test the path:

# on the laptop
ssh [email protected] 'cat >> /home/tofu/.ssh/authorized_keys' \
  < ~/.ssh/id_ed25519.pub
ssh [email protected] 'chown tofu: /home/tofu/.ssh/authorized_keys'
ssh-add ~/.ssh/id_ed25519
ssh [email protected] 'sudo -n /usr/sbin/pvesm apiinfo; pvesm status \
  --content snippets'

Expected (abbreviated, may differ depending on version):

APIVER 13
APIAGE 4
Name         Type     Status           Total            Used       Available        %
local         dir     active       98497780        12345678        81089632   12.53%

Save your progress? Sign up for free

Drill 4: Cloud-init template 9000

The official Ubuntu 24.04 cloud image is imported directly as a disk. virtio-scsi-single later allows an iothread per disk.

cat > /root/03-template.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
VMID=9000
STORAGE=local-lvm
IMG=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
URL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

[ -f "$IMG" ] || wget -q -O "$IMG" "$URL"
qm create "$VMID" --name ubuntu-2404-tpl --ostype l26 \
  --memory 2048 --cores 2 --cpu host --agent enabled=1 \
  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single
qm set "$VMID" --scsi0 "$STORAGE:0,import-from=$IMG,discard=on,iothread=1"
qm set "$VMID" --ide2 "$STORAGE:cloudinit"
qm set "$VMID" --boot order=scsi0
qm set "$VMID" --serial0 socket --vga serial0
qm template "$VMID"
EOF
bash /root/03-template.sh
qm config 9000 | grep -E '^(name|scsi0|ide2|boot|scsihw|template|agent):'

Expected (abbreviated, may differ depending on version):

agent: enabled=1
boot: order=scsi0
ide2: local-lvm:vm-9000-cloudinit,media=cdrom
name: ubuntu-2404-tpl
scsi0: local-lvm:base-9000-disk-0,discard=on,iothread=1,size=3584M
scsihw: virtio-scsi-single
template: 1

Why --serial0 socket --vga serial0? Ubuntu cloud images write their console to the serial port. Without it, the Proxmox console stays black. The disk is only 3.5 GB – OpenTofu enlarges it when cloning (Day 7).

Tip: The image doesn’t contain qemu-guest-agent. The user-data installs it on Day 7. Until then Proxmox reports “guest agent not running” – that’s normal.

Save your progress? Sign up for free

Drill 5: Enable and check IOMMU

From here on, only on the GPU host. First in the BIOS/UEFI: enable VT-d (Intel) or AMD-Vi/IOMMU, plus “Above 4G Decoding” and – if available – “Resizable BAR” for large GPUs. On AMD, and on Intel with kernel ≥ 6.8, IOMMU is then already on in the kernel. Proxmox 9 ships a newer kernel, so you no longer need intel_iommu=on. Optionally, iommu=pt gives a bit of performance.

Which bootloader?

uname -r
proxmox-boot-tool status 2>/dev/null | tail -2 || echo "GRUB ohne proxmox-boot-tool"
cat /proc/cmdline

Expected (abbreviated, may differ depending on version):

6.17.2-1-pve
E3A1-56F2 is configured with: uefi (versions: 6.17.2-1-pve)
BOOT_IMAGE=/boot/vmlinuz-6.17.2-1-pve root=/dev/mapper/pve-root ro quiet

Set iommu=pt – GRUB:

sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT="quiet\)"/\1 iommu=pt"/' \
  /etc/default/grub
update-grub

Or systemd-boot (ZFS root with UEFI):

sed -i 's/$/ iommu=pt/' /etc/kernel/cmdline
proxmox-boot-tool refresh

After the reboot (Drill 6 includes the reboot):

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi | head -5

Expected (abbreviated, text depends on CPU and kernel):

DMAR: IOMMU enabled
DMAR-IR: Enabled IRQ remapping in x2apic mode

If “remapping” is missing, the platform isn’t suitable for passthrough or the BIOS option is off.

Save your progress? Sign up for free

Drill 6: vfio instead of nouveau

The host must not drive the GPU itself. You load the three vfio modules, blacklist nouveau and bind the card directly to vfio-pci via its vendor:device ID.

lspci -nn | grep -i nvidia

Expected (example NVIDIA L4, your IDs will differ):

01:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8] (rev a1)
cat > /root/04-gpu-host.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
IDS=${1:?Vendor:Device-ID(s), kommasepariert}

cat > /etc/modules-load.d/vfio.conf <<'MOD'
vfio
vfio_iommu_type1
vfio_pci
MOD
cat > /etc/modprobe.d/ki-lab-gpu.conf <<CONF
blacklist nouveau
blacklist nvidiafb
options vfio-pci ids=${IDS}
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
CONF
update-initramfs -u -k all
EOF
bash /root/04-gpu-host.sh 10de:27b8
reboot

If the card has an audio function (01:00.1, typical for RTX cards), specify both IDs: 10de:2684,10de:22ba.

After the reboot:

lsmod | grep -E '^vfio'
lspci -nnk -s 01:00.0 | grep 'driver in use'

Expected (abbreviated, may differ depending on version):

vfio_pci               16384  0
vfio_iommu_type1       45056  0
vfio                   61440  2 vfio_iommu_type1,vfio_pci
        Kernel driver in use: vfio-pci

Why softdep? If an NVIDIA driver happens to be installed on the host, it would otherwise load before vfio-pci and grab the card.

Save your progress? Sign up for free

Drill 7: Check the IOMMU group and create a resource mapping

The GPU must sit alone in its IOMMU group (or only with its own functions or its root port).

NODE=$(hostname)
pvesh get /nodes/$NODE/hardware/pci --pci-class-blacklist "" \
  --output-format json \
  | jq -r '.[] | select(.vendor=="0x10de")
           | "\(.id) group=\(.iommugroup) \(.device_name)"'

Expected (abbreviated, may differ depending on version):

0000:01:00.0 group=14 AD104GL [L4]
G=14
ls /sys/kernel/iommu_groups/$G/devices/

Expected:

0000:01:00.0

Now the cluster-wide mapping gpu0. The VM will later reference only this name, not the PCI address:

pvesh create /cluster/mapping/pci --id gpu0 \
  --description "KI-Lab GPU fuer rke2-gpu-1" \
  --map node=$NODE,path=0000:01:00.0,id=10de:27b8
pvesh get /cluster/mapping/pci --output-format json | jq -r '.[].id'

Expected:

gpu0

Why a mapping? With Mapping.Use, the token tofu@pve may assign the GPU without being root@pam. Proxmox allows a direct hostpci0: 0000:01:00.0 assignment via the API only for root – that’s why bpg/proxmox requires username and password for hostpci.id, while the token is enough for hostpci.mapping.

Tip: Cards with an audio function (01:00.1) are easiest to set up in the GUI under Datacenter → Resource Mappings, where you select both functions. You check the result again with pvesh get /cluster/mapping/pci.

Save your progress? Sign up for free

Checkpoint

On the Proxmox host (GPU part on the GPU host):

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Rolle TofuProv" 'pveum role list --output-format json \
  | jq -e ".[] | select(.roleid==\"TofuProv\")"'
check "Token tofu@pve!provider" 'pveum user token list tofu@pve \
  --output-format json | jq -e ".[] | select(.tokenid==\"provider\")"'
check "Snippets auf local" 'pvesm status --content snippets | grep -q "^local "'
check "sudoers gueltig" 'visudo -cf /etc/sudoers.d/tofu'
check "Template 9000" '[ "$(qm config 9000 | sed -n "s/^template: //p")" = 1 ]'
check "IOMMU aktiv" 'dmesg | grep -qE "DMAR: IOMMU enabled|AMD-Vi"'
check "vfio-pci gebunden" 'lspci -nnk | grep -q "driver in use: vfio-pci"'
check "Mapping gpu0" 'pvesh get /cluster/mapping/pci --output-format json \
  | jq -e ".[] | select(.id==\"gpu0\")"'

Expected:

✔ Rolle TofuProv
✔ Token tofu@pve!provider
✔ Snippets auf local
✔ sudoers gueltig
✔ Template 9000
✔ IOMMU aktiv
✔ vfio-pci gebunden
✔ Mapping gpu0

Challenge

Prove that the role really is narrow: create a second, privilege-separated token tofu@pve!readonly that has only PVEAuditor, and show that it can read VMs but cannot clone template 9000. Delete it afterwards.

Hint: With --privsep 1, only ACLs attached directly to the token apply (-token instead of -user).

Solution

pveum user token add tofu@pve readonly --privsep 1 --output-format json \
  | jq -r .value > /root/ro.secret
pveum aclmod / -token 'tofu@pve!readonly' -role PVEAuditor
H="Authorization: PVEAPIToken=tofu@pve!readonly=$(cat /root/ro.secret)"
curl -sk -H "$H" https://localhost:8006/api2/json/nodes/$(hostname)/qemu \
  | jq -r '.data[].name'
curl -sk -o /dev/null -w '%{http_code}\n' -H "$H" -X POST \
  -d newid=9999 https://localhost:8006/api2/json/nodes/$(hostname)/qemu/9000/clone
pveum user token remove tofu@pve readonly && rm /root/ro.secret

Expected (abbreviated, may differ depending on version):

ubuntu-2404-tpl
403

Save your progress? Sign up for free

Enterprise Note

  • One token per environment and pipeline, with an expiration date (--expire) and ACLs on a pool (/pool/k8s-prod) instead of /. The token secret comes from Vault/OpenBao, not from files.
  • Proxmox clusters with several nodes: put the template on shared storage (Ceph/NFS) or keep one template per node; otherwise every clone migrates over the network. The sudo rule for tofu must exist on every node.
  • Don’t take images straight from the internet: version cloud images, verify checksums (SHA256SUMS), mirror them in the internal artifact repo, rebuild the template regularly (Packer or a pipeline).
  • GPU hosts: document firmware/BIOS profiles, check ECC and persistence. Multiple VMs per GPU require vGPU (NVIDIA license) or MIG instead of passthrough.
  • VLANs and firewall: separate the Kubernetes network from Proxmox management; make the host’s ports 8006/22 reachable only from the admin network.