Day 6: Preparing Proxmox
Goal: Proxmox has an API token for OpenTofu, an Ubuntu template 9000, and a GPU reserved for a VM via vfio. · Duration: approx. 90 min · Prerequisite: Day 0
Briefing
- OpenTofu talks to Proxmox via an API token
(
tofu@pve!provider) with its own roleTofuProv, never asroot@pam. - The
bpg/proxmoxprovider uploads cloud-init snippets via SSH. For that there’s the system usertofuwith exactly two sudo rules. - All six VMs are created as clones of template
9000(ubuntu-2404-tpl). - GPU passthrough needs IOMMU, vfio modules and a host kernel that
keeps its hands off the card. The resource mapping
gpu0decouples the VM from the PCI address.
Laptop (tofu, ssh-agent)
| HTTPS :8006 API token tofu@pve!provider
| SSH :22 user tofu (only tee -> snippets/)
v
pve1 vmbr0 ── 10.10.20.0/24 ── VMs 211-231
local:snippets local-lvm: template 9000
GPU 0000:01:00.0 -> vfio-pci -> mapping gpu0 -> rke2-gpu-1
Warning: The Proxmox and GPU commands in this chapter have been checked against the official docs (pve.proxmox.com for PVE 9, registry.terraform.io/providers/bpg/proxmox v0.116), but not tested live. Outputs are abbreviated and may differ depending on version. The scripts are under
code/infra-proxmox/host/.
Drill 1: Check the network – bridge and VLAN
All VMs are attached to vmbr0. If the Kubernetes network
lives in its own VLAN, the bridge must be VLAN-aware.
ip -br addr show vmbr0
grep -A8 '^iface vmbr0' /etc/network/interfacesExpected (abbreviated, may differ depending on version):
vmbr0 UP 10.10.20.5/24
iface vmbr0 inet static
address 10.10.20.5/24
gateway 10.10.20.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
For a separate VLAN (e.g., 20), extend the bridge like this and later
set vlan_id = 20 in OpenTofu:
bridge-vlan-aware yes
bridge-vids 2-4094
ifreload -a
bridge vlan show dev eno1 | head -3Why? Without bridge-vlan-aware, the bridge silently
discards the VMs’ tagged frames. For the lab, untagged on
vmbr0 is enough if 10.10.20.0/24 is the
management network.
Tip: Reserve
10.10.20.10(API VIP) and10.10.20.100–120(LoadBalancer pool) in your router’s DHCP server; otherwise it will hand them out twice.
Save your progress? Sign up for free
Drill 2: Role, user and API token
The role contains only what cloning, configuring, starting and using
mappings require. SDN.Audit/SDN.Use are
necessary; otherwise the token can’t see the bridges.
cat > /root/01-tofu-user.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
PRIVS="Datastore.Allocate Datastore.AllocateSpace Datastore.Audit"
PRIVS+=" VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.CPU"
PRIVS+=" VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType"
PRIVS+=" VM.Config.Memory VM.Config.Network VM.Config.Options"
PRIVS+=" VM.PowerMgmt VM.Migrate VM.Console VM.GuestAgent.Audit"
PRIVS+=" Sys.Audit Sys.Console SDN.Audit SDN.Use"
PRIVS+=" Mapping.Audit Mapping.Use"
pveum role add TofuProv -privs "${PRIVS// /,}"
pveum user add tofu@pve --comment "OpenTofu bpg/proxmox"
pveum aclmod / -user tofu@pve -role TofuProv
pveum user token add tofu@pve provider --privsep 0
EOF
bash /root/01-tofu-user.shExpected (abbreviated, may differ depending on version):
┌──────────────┬──────────────────────────────────────┐
│ key │ value │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ tofu@pve!provider │
│ info │ {"privsep":"0"} │
│ value │ 3f1c...-....-....-....-............ │
└──────────────┴──────────────────────────────────────┘
The value appears only once. --privsep 0
means the token inherits exactly the permissions of
tofu@pve, so you maintain ACLs in only one place.
Security: Never write the token into
*.tforterraform.tfvars. It belongs in a file with mode 600 outside of Git.
On the laptop:
cat > ~/ki-lab/pve.env <<'EOF'
export PROXMOX_VE_ENDPOINT='https://10.10.20.5:8006/'
export PROXMOX_VE_API_TOKEN='tofu@pve!provider=HIER-DEN-VALUE-EINFUEGEN'
EOF
chmod 600 ~/ki-lab/pve.env
source ~/ki-lab/pve.env
curl -sk -H "Authorization: PVEAPIToken=${PROXMOX_VE_API_TOKEN}" \
"${PROXMOX_VE_ENDPOINT}api2/json/version" | jq -r .data.versionExpected (may differ depending on version):
9.1.4
Save your progress? Sign up for free
Drill 3: SSH user and snippets storage
bpg/proxmox uploads snippets (cloud-init
user-data) via SSH, because the Proxmox API has no upload for them. The
user tofu may run only two precisely described commands via
sudo.
cat > /root/02-tofu-ssh.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
apt-get install -y sudo
id tofu >/dev/null 2>&1 || useradd -m -s /bin/bash tofu
install -d -m 700 -o tofu -g tofu /home/tofu/.ssh
cat > /etc/sudoers.d/tofu <<'SUDO'
tofu ALL=(root) NOPASSWD: /usr/sbin/pvesm apiinfo
tofu ALL=(root) NOPASSWD: /usr/bin/tee /var/lib/vz/snippets/[a-zA-Z0-9_][a-zA-Z0-9_.-]*
SUDO
chmod 440 /etc/sudoers.d/tofu
visudo -cf /etc/sudoers.d/tofu
pvesm set local --content iso,vztmpl,backup,snippets
EOF
bash /root/02-tofu-ssh.shExpected (abbreviated):
/etc/sudoers.d/tofu: parsed OK
Security: Never allow
NOPASSWD: /usr/sbin/qmor/var/lib/vz/*.qm set --hookscriptand*with../are both root-equivalent – the provider docs say so as well.
Store your laptop key and test the path:
# on the laptop
ssh [email protected] 'cat >> /home/tofu/.ssh/authorized_keys' \
< ~/.ssh/id_ed25519.pub
ssh [email protected] 'chown tofu: /home/tofu/.ssh/authorized_keys'
ssh-add ~/.ssh/id_ed25519
ssh [email protected] 'sudo -n /usr/sbin/pvesm apiinfo; pvesm status \
--content snippets'Expected (abbreviated, may differ depending on version):
APIVER 13
APIAGE 4
Name Type Status Total Used Available %
local dir active 98497780 12345678 81089632 12.53%
Save your progress? Sign up for free
Drill 4: Cloud-init template 9000
The official Ubuntu 24.04 cloud image is imported directly as a disk.
virtio-scsi-single later allows an iothread
per disk.
cat > /root/03-template.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
VMID=9000
STORAGE=local-lvm
IMG=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
URL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
[ -f "$IMG" ] || wget -q -O "$IMG" "$URL"
qm create "$VMID" --name ubuntu-2404-tpl --ostype l26 \
--memory 2048 --cores 2 --cpu host --agent enabled=1 \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single
qm set "$VMID" --scsi0 "$STORAGE:0,import-from=$IMG,discard=on,iothread=1"
qm set "$VMID" --ide2 "$STORAGE:cloudinit"
qm set "$VMID" --boot order=scsi0
qm set "$VMID" --serial0 socket --vga serial0
qm template "$VMID"
EOF
bash /root/03-template.sh
qm config 9000 | grep -E '^(name|scsi0|ide2|boot|scsihw|template|agent):'Expected (abbreviated, may differ depending on version):
agent: enabled=1
boot: order=scsi0
ide2: local-lvm:vm-9000-cloudinit,media=cdrom
name: ubuntu-2404-tpl
scsi0: local-lvm:base-9000-disk-0,discard=on,iothread=1,size=3584M
scsihw: virtio-scsi-single
template: 1
Why --serial0 socket --vga serial0? Ubuntu cloud images
write their console to the serial port. Without it, the Proxmox console
stays black. The disk is only 3.5 GB – OpenTofu enlarges it when cloning
(Day 7).
Tip: The image doesn’t contain
qemu-guest-agent. The user-data installs it on Day 7. Until then Proxmox reports “guest agent not running” – that’s normal.
Save your progress? Sign up for free
Drill 5: Enable and check IOMMU
From here on, only on the GPU host. First in the BIOS/UEFI: enable
VT-d (Intel) or AMD-Vi/IOMMU, plus
“Above 4G Decoding” and – if available – “Resizable BAR” for large GPUs.
On AMD, and on Intel with kernel ≥ 6.8, IOMMU is then already on in the
kernel. Proxmox 9 ships a newer kernel, so you no longer need
intel_iommu=on. Optionally, iommu=pt gives a
bit of performance.
Which bootloader?
uname -r
proxmox-boot-tool status 2>/dev/null | tail -2 || echo "GRUB ohne proxmox-boot-tool"
cat /proc/cmdlineExpected (abbreviated, may differ depending on version):
6.17.2-1-pve
E3A1-56F2 is configured with: uefi (versions: 6.17.2-1-pve)
BOOT_IMAGE=/boot/vmlinuz-6.17.2-1-pve root=/dev/mapper/pve-root ro quiet
Set iommu=pt – GRUB:
sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT="quiet\)"/\1 iommu=pt"/' \
/etc/default/grub
update-grubOr systemd-boot (ZFS root with UEFI):
sed -i 's/$/ iommu=pt/' /etc/kernel/cmdline
proxmox-boot-tool refreshAfter the reboot (Drill 6 includes the reboot):
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi | head -5Expected (abbreviated, text depends on CPU and kernel):
DMAR: IOMMU enabled
DMAR-IR: Enabled IRQ remapping in x2apic mode
If “remapping” is missing, the platform isn’t suitable for passthrough or the BIOS option is off.
Save your progress? Sign up for free
Drill 6: vfio instead of nouveau
The host must not drive the GPU itself. You load the three vfio
modules, blacklist nouveau and bind the card directly to
vfio-pci via its vendor:device ID.
lspci -nn | grep -i nvidiaExpected (example NVIDIA L4, your IDs will differ):
01:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8] (rev a1)
cat > /root/04-gpu-host.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
IDS=${1:?Vendor:Device-ID(s), kommasepariert}
cat > /etc/modules-load.d/vfio.conf <<'MOD'
vfio
vfio_iommu_type1
vfio_pci
MOD
cat > /etc/modprobe.d/ki-lab-gpu.conf <<CONF
blacklist nouveau
blacklist nvidiafb
options vfio-pci ids=${IDS}
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
CONF
update-initramfs -u -k all
EOF
bash /root/04-gpu-host.sh 10de:27b8
rebootIf the card has an audio function (01:00.1, typical for
RTX cards), specify both IDs: 10de:2684,10de:22ba.
After the reboot:
lsmod | grep -E '^vfio'
lspci -nnk -s 01:00.0 | grep 'driver in use'Expected (abbreviated, may differ depending on version):
vfio_pci 16384 0
vfio_iommu_type1 45056 0
vfio 61440 2 vfio_iommu_type1,vfio_pci
Kernel driver in use: vfio-pci
Why softdep? If an NVIDIA driver happens to be installed
on the host, it would otherwise load before vfio-pci and
grab the card.
Save your progress? Sign up for free
Drill 7: Check the IOMMU group and create a resource mapping
The GPU must sit alone in its IOMMU group (or only with its own functions or its root port).
NODE=$(hostname)
pvesh get /nodes/$NODE/hardware/pci --pci-class-blacklist "" \
--output-format json \
| jq -r '.[] | select(.vendor=="0x10de")
| "\(.id) group=\(.iommugroup) \(.device_name)"'Expected (abbreviated, may differ depending on version):
0000:01:00.0 group=14 AD104GL [L4]
G=14
ls /sys/kernel/iommu_groups/$G/devices/Expected:
0000:01:00.0
Now the cluster-wide mapping gpu0. The VM will later
reference only this name, not the PCI address:
pvesh create /cluster/mapping/pci --id gpu0 \
--description "KI-Lab GPU fuer rke2-gpu-1" \
--map node=$NODE,path=0000:01:00.0,id=10de:27b8
pvesh get /cluster/mapping/pci --output-format json | jq -r '.[].id'Expected:
gpu0
Why a mapping? With Mapping.Use, the token
tofu@pve may assign the GPU without being
root@pam. Proxmox allows a direct
hostpci0: 0000:01:00.0 assignment via the API only for root
– that’s why bpg/proxmox requires username and password for
hostpci.id, while the token is enough for
hostpci.mapping.
Tip: Cards with an audio function (
01:00.1) are easiest to set up in the GUI under Datacenter → Resource Mappings, where you select both functions. You check the result again withpvesh get /cluster/mapping/pci.
Save your progress? Sign up for free
Checkpoint
On the Proxmox host (GPU part on the GPU host):
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Rolle TofuProv" 'pveum role list --output-format json \
| jq -e ".[] | select(.roleid==\"TofuProv\")"'
check "Token tofu@pve!provider" 'pveum user token list tofu@pve \
--output-format json | jq -e ".[] | select(.tokenid==\"provider\")"'
check "Snippets auf local" 'pvesm status --content snippets | grep -q "^local "'
check "sudoers gueltig" 'visudo -cf /etc/sudoers.d/tofu'
check "Template 9000" '[ "$(qm config 9000 | sed -n "s/^template: //p")" = 1 ]'
check "IOMMU aktiv" 'dmesg | grep -qE "DMAR: IOMMU enabled|AMD-Vi"'
check "vfio-pci gebunden" 'lspci -nnk | grep -q "driver in use: vfio-pci"'
check "Mapping gpu0" 'pvesh get /cluster/mapping/pci --output-format json \
| jq -e ".[] | select(.id==\"gpu0\")"'Expected:
✔ Rolle TofuProv
✔ Token tofu@pve!provider
✔ Snippets auf local
✔ sudoers gueltig
✔ Template 9000
✔ IOMMU aktiv
✔ vfio-pci gebunden
✔ Mapping gpu0
Challenge
Prove that the role really is narrow: create a second,
privilege-separated token tofu@pve!readonly that has only
PVEAuditor, and show that it can read VMs but cannot clone
template 9000. Delete it afterwards.
Hint: With --privsep 1, only ACLs
attached directly to the token apply (-token instead of
-user).
Solution
pveum user token add tofu@pve readonly --privsep 1 --output-format json \
| jq -r .value > /root/ro.secret
pveum aclmod / -token 'tofu@pve!readonly' -role PVEAuditor
H="Authorization: PVEAPIToken=tofu@pve!readonly=$(cat /root/ro.secret)"
curl -sk -H "$H" https://localhost:8006/api2/json/nodes/$(hostname)/qemu \
| jq -r '.data[].name'
curl -sk -o /dev/null -w '%{http_code}\n' -H "$H" -X POST \
-d newid=9999 https://localhost:8006/api2/json/nodes/$(hostname)/qemu/9000/clone
pveum user token remove tofu@pve readonly && rm /root/ro.secretExpected (abbreviated, may differ depending on version):
ubuntu-2404-tpl
403
Save your progress? Sign up for free
Enterprise Note
- One token per environment and pipeline, with an expiration date
(
--expire) and ACLs on a pool (/pool/k8s-prod) instead of/. The token secret comes from Vault/OpenBao, not from files. - Proxmox clusters with several nodes: put the template on shared
storage (Ceph/NFS) or keep one template per node; otherwise every clone
migrates over the network. The sudo rule for
tofumust exist on every node. - Don’t take images straight from the internet: version cloud images,
verify checksums (
SHA256SUMS), mirror them in the internal artifact repo, rebuild the template regularly (Packer or a pipeline). - GPU hosts: document firmware/BIOS profiles, check ECC and persistence. Multiple VMs per GPU require vGPU (NVIDIA license) or MIG instead of passthrough.
- VLANs and firewall: separate the Kubernetes network from Proxmox management; make the host’s ports 8006/22 reachable only from the admin network.