Day 12: Infrastructure as Code
Goal: Tenant onboarding runs as an OpenTofu module against kind, and the LLM VM is set up idempotently with an Ansible role – both with drift detection. · Duration: approx. 150 min · Prerequisite: Days 5, 6 and 7
Briefing
- OpenTofu manages Kubernetes objects (kubernetes provider) and Helm
releases (helm provider); a
tenantmodule replaces the YAML bundle from Day 6. - Existing objects are adopted with an
importblock – not deleted and recreated. - Ansible sets up the VM: Ollama with systemd limits, node_exporter,
ufw. Second run =
changed=0. - Drift:
tofu plan -detailed-exitcode(exit 2) oransible-playbook --check --diff.
tofu-tenant/ ansible-llm-vm/
├── main.tf for_each tenants ├── site.yml
├── platform.tf helm_release ├── inventory/{hosts,docker-test}.ini
├── terraform.tfvars └── roles/llm_vm/{tasks,handlers,
└── modules/tenant/ ns, quota, … defaults,templates}
Drill 1: Reading the tenant
module
mkdir -p ~/sre-lab/tag12 && cd ~/sre-lab/tag12
cp -R ~/sre-lab/code/tofu-tenant ~/sre-lab/code/ansible-llm-vm .
cd tofu-tenant
grep -h '^resource' modules/tenant/*.tf
cat main.tf terraform.tfvarsExpected:
resource "kubernetes_namespace_v1" "this" {
resource "kubernetes_default_service_account_v1" "this" {
resource "kubernetes_role_binding_v1" "admin" {
resource "kubernetes_resource_quota_v1" "this" {
resource "kubernetes_limit_range_v1" "this" {
resource "kubernetes_network_policy_v1" "default_deny" {
resource "kubernetes_network_policy_v1" "allow_dns" {
resource "kubernetes_network_policy_v1" "same_namespace" {
resource "kubernetes_network_policy_v1" "to_gateway" {
module "tenant" {
source = "./modules/tenant"
for_each = var.tenants
name = each.key
team_group = each.value.team_group
gpu_quota = each.value.gpu_quota
}
output "namespaces" {
value = sort([for t in module.tenant : t.namespace])
}
tenants = {
acme = { team_group = "team-acme", gpu_quota = 1 }
globex = { team_group = "team-globex" }
}
The module variables validate input before anything reaches the cluster:
sed -n '/variable "gpu_quota"/,/^}/p' modules/tenant/variables.tfExpected:
variable "gpu_quota" {
description = "Maximal angeforderte GPUs (requests.nvidia.com/gpu)"
type = number
default = 0
validation {
condition = var.gpu_quota >= 0 && floor(var.gpu_quota) == var.gpu_quota
error_message = "gpu_quota muss eine ganze Zahl >= 0 sein."
}
}
Why _v1 resources? The unversioned names
(kubernetes_namespace) have been deprecated since provider
2.x. Why for_each instead of count? With
count, removing acme shifts the indices of all
following tenants – OpenTofu would destroy and recreate
globex.
Save your progress? Sign up for free
Drill 2: init and importing what exists
The namespaces from Day 6 and the metrics-server from Day 7 already
exist. An apply would fail with AlreadyExists
– so import them:
tofu init
cat > imports.tf <<'EOF'
# Bestand aus Tag 6/7 übernehmen; nach erfolgreichem apply löschen
import {
for_each = var.tenants
to = module.tenant[each.key].kubernetes_namespace_v1.this
id = "tenant-${each.key}"
}
import {
for_each = var.tenants
to = module.tenant[each.key].kubernetes_default_service_account_v1.this
id = "tenant-${each.key}/default"
}
import {
for_each = var.tenants
to = module.tenant[each.key].kubernetes_role_binding_v1.admin
id = "tenant-${each.key}/tenant-admin"
}
import {
for_each = var.tenants
to = module.tenant[each.key].kubernetes_resource_quota_v1.this
id = "tenant-${each.key}/tenant-quota"
}
import {
for_each = var.tenants
to = module.tenant[each.key].kubernetes_limit_range_v1.this
id = "tenant-${each.key}/tenant-defaults"
}
EOF
for p in default_deny:default-deny-all allow_dns:allow-dns \
same_namespace:allow-same-namespace to_gateway:allow-egress-llm-gateway; do
cat >> imports.tf <<EOF
import {
for_each = var.tenants
to = module.tenant[each.key].kubernetes_network_policy_v1.${p%%:*}
id = "tenant-\${each.key}/${p##*:}"
}
EOF
done
cat >> imports.tf <<'EOF'
import {
to = helm_release.metrics_server
id = "kube-system/metrics-server"
}
EOF
tofu fmt imports.tf
tofu plan -no-color -out=tfplan | grep -E '^ # .*updated|^Plan:'Expected (shortened):
- Installed hashicorp/kubernetes v3.3.0 (signed, key ID 0C0AF313E5FD9F80)
- Installed hashicorp/helm v3.3.0 (signed, key ID 0C0AF313E5FD9F80)
OpenTofu has been successfully initialized!
# helm_release.metrics_server will be updated in-place
# module.tenant["acme"].kubernetes_namespace_v1.this will be updated in-place
# module.tenant["acme"].kubernetes_role_binding_v1.admin will be updated in-place
# module.tenant["globex"].kubernetes_namespace_v1.this will be updated in-place
# module.tenant["globex"].kubernetes_role_binding_v1.admin will be updated in-place
Plan: 19 to import, 0 to add, 5 to change, 0 to destroy.
Read every ~ update before the apply: for the
helm_release, OpenTofu only takes the repository and the
set values into the state; for the RoleBinding, the
provider adds namespace = "default" to the group subject –
meaningless for RBAC. For the namespaces, the plan shows the PSA labels
as +, although they have been set since Day 6: the provider
(3.3.0) apparently does not read labels with kubernetes.io
in the key into the state on import. The apply changes nothing in the
cluster, and afterwards the plan is empty.
tofu apply -no-color tfplan | grep 'Apply complete'
rm imports.tf tfplan
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"Expected:
Apply complete! Resources: 19 imported, 0 added, 5 changed, 0 destroyed.
Exit-Code: 0
Warning: Import blocks with
for_each = var.tenantsalso apply to every new tenant. Ifimports.tfstays around, the onboarding in Drill 4 fails withMissing Resource Identity After Read, because OpenTofu tries to import a namespace that does not exist.
Save your progress? Sign up for free
Drill 3: Understanding state
tofu state list | head -4
tofu state list | wc -l
tofu state show 'module.tenant["acme"].kubernetes_resource_quota_v1.this' \
| grep -E '^#|gpu'Expected:
helm_release.metrics_server
module.tenant["acme"].kubernetes_default_service_account_v1.this
module.tenant["acme"].kubernetes_limit_range_v1.this
module.tenant["acme"].kubernetes_namespace_v1.this
19
# module.tenant["acme"].kubernetes_resource_quota_v1.this:
"requests.nvidia.com/gpu" = "1"
Here the state lives locally in terraform.tfstate –
including secrets, if resources contain any. In a team it belongs in a
remote backend with locking and encryption (Enterprise Note).
Security: Never commit
terraform.tfstate; the.gitignorein the code folder covers this.
Save your progress? Sign up for free
Drill 4: Detecting and healing drift
Someone deletes the default-deny policy “just for a quick debugging session”:
kubectl -n tenant-acme delete networkpolicy default-deny-all
tofu plan -no-color -detailed-exitcode > drift.txt; echo "Exit-Code: $?"
grep -E '^ # |^Plan:' drift.txt
tofu apply -auto-approve -no-color | grep 'Apply complete'Expected:
networkpolicy.networking.k8s.io "default-deny-all" deleted from tenant-acme namespace
Exit-Code: 2
# module.tenant["acme"].kubernetes_network_policy_v1.default_deny will be created
Plan: 1 to add, 0 to change, 0 to destroy.
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Exit codes of -detailed-exitcode: 0 no
change, 1 error, 2 drift. A nightly CI job
running this command is the simplest drift alarm.
Save your progress? Sign up for free
Drill 5: Onboarding and offboarding by variable
cat > initech.auto.tfvars <<'EOF'
tenants = {
acme = { team_group = "team-acme", gpu_quota = 1 }
globex = { team_group = "team-globex" }
initech = { team_group = "team-initech", gpu_quota = 2 }
}
EOF
tofu apply -auto-approve -no-color | grep -E '^Plan:|Apply complete'
kubectl -n tenant-initech get quota tenant-quota \
-o jsonpath='{.spec.hard.requests\.nvidia\.com/gpu}'; echo
rm initech.auto.tfvars
tofu plan -no-color | grep '^Plan:'
tofu apply -auto-approve -no-color | grep 'Apply complete'
kubectl get ns tenant-initechExpected:
Plan: 9 to add, 0 to change, 0 to destroy.
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
2
Plan: 0 to add, 0 to change, 9 to destroy.
Apply complete! Resources: 0 added, 0 changed, 9 destroyed.
Error from server (NotFound): namespaces "tenant-initech" not found
Why *.auto.tfvars? OpenTofu loads them automatically
after terraform.tfvars – later files override earlier ones.
In real setups there is one file per tenant (or a YAML file read in with
yamldecode()), and onboarding is a merge request.
Warning: Offboarding deletes the namespace with all its data (PVCs!). In production, use
lifecycle { prevent_destroy = true }on the namespace and a deliberate decommissioning process with a backup.
Save your progress? Sign up for free
Drill 6: An Ansible test target without a VM
Without llm-vm, an Ubuntu container with systemd serves
as the target; a stub replaces the Ollama installation (about 1.5 GB
download):
mkdir -p ~/sre-lab/tag12/vmimg && cd ~/sre-lab/tag12/vmimg
cat > Dockerfile <<'EOF'
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
systemd systemd-sysv python3 python3-apt sudo iptables \
&& rm -rf /var/lib/apt/lists/*
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]
EOF
docker build -q -t sre-lab/llm-vm:24.04 .
docker run -d --name llm-vm-test --hostname llm-vm-test --privileged \
--cgroupns=host -v /sys/fs/cgroup:/sys/fs/cgroup:rw \
--tmpfs /run --tmpfs /run/lock sre-lab/llm-vm:24.04
sleep 5
docker exec llm-vm-test systemctl is-system-running
docker exec llm-vm-test bash -c 'apt-get update -qq
printf "#!/bin/sh\ncase \"\$1\" in serve) exec sleep infinity;; *) exit 0;; esac\n" \
> /usr/local/bin/ollama && chmod +x /usr/local/bin/ollama'Expected (shortened):
sha256:87217f98…
f636322a34fe…
running
Warning: If the container reports
Failed to allocate manager object: Too many open files, the inotify instances of the Docker VM are exhausted (typical with kind). Workaround until the next VM restart:docker run --rm --privileged ubuntu:24.04 sysctl -w fs.inotify.max_user_instances=512.
Inventory for the test target – the Docker connection plugin needs no SSH:
cd ~/sre-lab/tag12/ansible-llm-vm
cat inventory/docker-test.ini
ansible-galaxy collection install -r requirements.yml
ansible-playbook site.yml --syntax-check
ansible-lintExpected (shortened):
[llm_vms]
llm-vm-test ansible_connection=community.docker.docker
[llm_vms:vars]
llm_vm_gateway_cidr=172.18.0.0/16
llm_vm_ollama_skip_install=true
playbook: site.yml
Passed: 0 failure(s), 0 warning(s) in 9 files processed of 12 encountered.
Last profile that met the validation criteria was 'production'.
Why are all variables called llm_vm_…? ansible-lint
(var-naming[no-role-prefix]) requires the role name as a
prefix – otherwise gateway_cidr will collide with another
role sooner or later.
Save your progress? Sign up for free
Drill 7: Applying the role, checking idempotency
The core pieces of the role – a drop-in with limits, and the firewall:
cat roles/llm_vm/templates/override.conf.j2
grep -A5 'Ollama nur vom Gateway' roles/llm_vm/tasks/main.ymlExpected:
# {{ ansible_managed }}
[Service]
Environment="OLLAMA_HOST={{ llm_vm_ollama_listen }}"
Environment="OLLAMA_MODELS={{ llm_vm_ollama_models_dir }}"
Environment="OLLAMA_KEEP_ALIVE={{ llm_vm_ollama_keep_alive }}"
MemoryMax={{ llm_vm_ollama_memory_max }}
CPUQuota={{ llm_vm_ollama_cpu_quota }}
LimitNOFILE={{ llm_vm_ollama_nofile }}
- name: UFW – Ollama nur vom Gateway
community.general.ufw:
rule: allow
port: "11434"
proto: tcp
from_ip: "{{ llm_vm_gateway_cidr }}"
First check mode on the fresh host, then for real, then once more:
run() { ansible-playbook -i inventory/docker-test.ini site.yml "$@" \
| grep -E '^fatal|^llm-vm-test +:' | cut -c1-80; }
run --check
run
runExpected (shortened):
fatal: [llm-vm-test]: FAILED! => {"changed": false, "msg": "Could not find the
llm-vm-test : ok=8 changed=7 unreachable=0 failed=1
llm-vm-test : ok=16 changed=14 unreachable=0 failed=0
llm-vm-test : ok=15 changed=0 unreachable=0 failed=0
Why does check mode fail? In a dry run the unit is never written –
the systemd task finds no service. Check mode is a drift tool for hosts
that are already set up, not a substitute for a test
run. changed=0 in the third run is the proof of
idempotency; ok drops from 16 to 15 because no handler runs
anymore.
docker exec llm-vm-test bash -c 'systemctl is-active ollama prometheus-node-exporter
systemctl show ollama -p MemoryMax -p LimitNOFILE; ufw status | grep ALLOW'Expected:
active
active
MemoryMax=6442450944
LimitNOFILE=65536
22/tcp ALLOW Anywhere
11434/tcp ALLOW 172.18.0.0/16
9100/tcp ALLOW 172.18.0.0/16
Save your progress? Sign up for free
Drill 8: Drift on the VM with
--check --diff
docker exec llm-vm-test sed -i 's/MemoryMax=6G/MemoryMax=12G/' \
/etc/systemd/system/ollama.service.d/override.conf
ansible-playbook -i inventory/docker-test.ini site.yml --check --diff \
| grep -E '^[-+@]|^changed|HANDLER|^llm-vm-test +:' | cut -c1-80
runExpected (shortened):
--- before: /etc/systemd/system/ollama.service.d/override.conf
+++ after: …/override.conf.j2
@@ -3,6 +3,6 @@
-MemoryMax=12G
+MemoryMax=6G
changed: [llm-vm-test]
RUNNING HANDLER [llm_vm : Restart ollama] ****************************************
changed: [llm-vm-test]
llm-vm-test : ok=16 changed=2 unreachable=0 failed=0
llm-vm-test : ok=16 changed=2 unreachable=0 failed=0
The dry run shows the deviation including the handler but changes
nothing; only the real run (run) resets it to 6G and
restarts Ollama.
For the real VM, just swap the inventory
(inventory/hosts.ini, IP from
multipass info llm-vm) and run
ansible-playbook site.yml – then the role actually
downloads Ollama 0.40.2 and ministral-3:3b.
And the VM itself? OpenTofu creates it, e.g. at Hetzner Cloud –
checked only with tofu validate:
cd ~/sre-lab/tag12/tofu-tenant/examples/llm-vm-hcloud
grep -E '^resource|server_type|source_ips|^output' main.tf
tofu init -backend=false >/dev/null && tofu validateExpected:
resource "hcloud_ssh_key" "admin" {
resource "hcloud_firewall" "llm_vm" {
source_ips = ["0.0.0.0/0", "::/0"]
source_ips = [var.gateway_cidr]
resource "hcloud_server" "llm_vm" {
server_type = "ccx33" # 8 dedizierte vCPU, 32 GB RAM
output "ansible_inventory" {
Success! The configuration is valid.
The chain in production: tofu apply (VM + cloud
firewall) → output as the Ansible inventory →
ansible-playbook site.yml. For on-prem you swap the
provider (dmacvicar/libvirt, vSphere, Proxmox); for AWS,
aws_instance with a GPU type (g6.xlarge) and a
security group instead of the hcloud firewall.
Save your progress? Sign up for free
Checkpoint
cd ~/sre-lab/tag12/tofu-tenant
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "tofu validate" 'tofu validate'
check "kein Drift (Exit 0)" 'tofu plan -detailed-exitcode'
check "19 Objekte im State" '[ "$(tofu state list | wc -l)" -eq 19 ]'
check "imports.tf entfernt" '[ ! -f imports.tf ]'
check "State nicht im Git" 'grep -q tfstate ~/sre-lab/code/tofu-tenant/.gitignore'
cd ~/sre-lab/tag12/ansible-llm-vm
check "ansible-lint sauber" 'ansible-lint'
check "Idempotent (changed=0)" \
'ansible-playbook -i inventory/docker-test.ini site.yml | grep -q "changed=0 "'
check "Ollama-Limit aktiv" \
'docker exec llm-vm-test systemctl show ollama -p MemoryMax | grep -q 6442450944'Challenge
A tenant may get at most 4 GPUs – more only with its own node pool.
Enforce this in the root module so that tofu plan already
fails, and prove it with a tenant hooli and 8 GPUs, without
changing terraform.tfvars.
Hint: A validation block in
variable "tenants" can check all map entries with
alltrue([for …]).
Solution
cd ~/sre-lab/tag12/tofu-tenant
cat > variables.tf <<'EOF'
variable "kube_context" {
type = string
default = "kind-sre-lab"
}
variable "tenants" {
description = "Mandanten: Name => Einstellungen"
type = map(object({
team_group = string
gpu_quota = optional(number, 0)
}))
validation {
condition = alltrue([for t in values(var.tenants) : t.gpu_quota <= 4])
error_message = "Maximal 4 GPUs pro Mandant – mehr nur mit eigenem Node-Pool."
}
}
EOF
tofu plan -var 'tenants={hooli={team_group="team-hooli",gpu_quota=8}}' 2>&1 \
| grep -E 'Error|Maximal'
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"Expected:
│ Error: Invalid value for variable
│ Maximal 4 GPUs pro Mandant – mehr nur mit eigenem Node-Pool.
Exit-Code: 0
Clean up (optional; this also deletes the tenant namespaces from Day 6 and the metrics-server):
cd ~/sre-lab/tag12/tofu-tenant
tofu destroy -auto-approve -no-color | grep 'Destroy complete'
docker rm -f llm-vm-testSave your progress? Sign up for free
Interview Note
- Typical question: “Terraform or OpenTofu?” – OpenTofu is the
MPL-licensed fork after Terraform’s switch to the BSL in 2023,
compatible at the CLI and provider level; differences include state
encryption and
for_eachinimportblocks in OpenTofu. What counts for customers: license, registry, support contract. - Crisp: “For me, onboarding is a merge request: tenant in tfvars,
plan in CI, review, apply – and a nightly
plan -detailed-exitcodereports drift.” - Ansible vs. Tofu: Tofu for declarative APIs (cloud, Kubernetes),
Ansible for the state inside the machine (packages, systemd,
firewall). You prove idempotency with
changed=0, not with hope. - Bridge from your own practice: you have already managed OpenBao policies, auth mounts and namespaces with Terraform/OpenTofu – including importing legacy resources and drift caused by click changes in the UI.
- Counter-question: How does Mistral provision customer deployments – its own modules per cloud, Helm + GitOps, or an operator?
Enterprise Note
- Remote state with locking (S3 + DynamoDB/S3 lockfile, GCS, Azure Blob, Postgres) and OpenTofu state encryption; access only for the CI identity.
- Pin provider and module versions, commit
.terraform.lock.hcl, modules from an internal registry/Git with tags. - In the long run, Kubernetes objects via GitOps (Argo CD/Flux) instead of Tofu when teams deploy themselves; Tofu for clusters, networks, IAM and the onboarding skeleton.
- Ansible at customers on-prem: execution environments (containers),
inventory from the CMDB/Tofu output,
--check --diffas change evidence for the CAB. - Air gap: the Ollama binary and models from an internal mirror
instead of
ollama.com/install.sh(Day 14).