Skip to content
Hands-on IT IT Drills

Day 12: Infrastructure as Code

Goal: Tenant onboarding runs as an OpenTofu module against kind, and the LLM VM is set up idempotently with an Ansible role – both with drift detection. · Duration: approx. 150 min · Prerequisite: Days 5, 6 and 7

Briefing

  • OpenTofu manages Kubernetes objects (kubernetes provider) and Helm releases (helm provider); a tenant module replaces the YAML bundle from Day 6.
  • Existing objects are adopted with an import block – not deleted and recreated.
  • Ansible sets up the VM: Ollama with systemd limits, node_exporter, ufw. Second run = changed=0.
  • Drift: tofu plan -detailed-exitcode (exit 2) or ansible-playbook --check --diff.
tofu-tenant/                         ansible-llm-vm/
├── main.tf        for_each tenants  ├── site.yml
├── platform.tf    helm_release      ├── inventory/{hosts,docker-test}.ini
├── terraform.tfvars                 └── roles/llm_vm/{tasks,handlers,
└── modules/tenant/  ns, quota, …        defaults,templates}

Drill 1: Reading the tenant module

mkdir -p ~/sre-lab/tag12 && cd ~/sre-lab/tag12
cp -R ~/sre-lab/code/tofu-tenant ~/sre-lab/code/ansible-llm-vm .
cd tofu-tenant
grep -h '^resource' modules/tenant/*.tf
cat main.tf terraform.tfvars

Expected:

resource "kubernetes_namespace_v1" "this" {
resource "kubernetes_default_service_account_v1" "this" {
resource "kubernetes_role_binding_v1" "admin" {
resource "kubernetes_resource_quota_v1" "this" {
resource "kubernetes_limit_range_v1" "this" {
resource "kubernetes_network_policy_v1" "default_deny" {
resource "kubernetes_network_policy_v1" "allow_dns" {
resource "kubernetes_network_policy_v1" "same_namespace" {
resource "kubernetes_network_policy_v1" "to_gateway" {
module "tenant" {
  source   = "./modules/tenant"
  for_each = var.tenants

  name       = each.key
  team_group = each.value.team_group
  gpu_quota  = each.value.gpu_quota
}

output "namespaces" {
  value = sort([for t in module.tenant : t.namespace])
}
tenants = {
  acme   = { team_group = "team-acme", gpu_quota = 1 }
  globex = { team_group = "team-globex" }
}

The module variables validate input before anything reaches the cluster:

sed -n '/variable "gpu_quota"/,/^}/p' modules/tenant/variables.tf

Expected:

variable "gpu_quota" {
  description = "Maximal angeforderte GPUs (requests.nvidia.com/gpu)"
  type        = number
  default     = 0
  validation {
    condition     = var.gpu_quota >= 0 && floor(var.gpu_quota) == var.gpu_quota
    error_message = "gpu_quota muss eine ganze Zahl >= 0 sein."
  }
}

Why _v1 resources? The unversioned names (kubernetes_namespace) have been deprecated since provider 2.x. Why for_each instead of count? With count, removing acme shifts the indices of all following tenants – OpenTofu would destroy and recreate globex.

Save your progress? Sign up for free

Drill 2: init and importing what exists

The namespaces from Day 6 and the metrics-server from Day 7 already exist. An apply would fail with AlreadyExists – so import them:

tofu init
cat > imports.tf <<'EOF'
# Bestand aus Tag 6/7 übernehmen; nach erfolgreichem apply löschen
import {
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_namespace_v1.this
  id       = "tenant-${each.key}"
}
import {
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_default_service_account_v1.this
  id       = "tenant-${each.key}/default"
}
import {
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_role_binding_v1.admin
  id       = "tenant-${each.key}/tenant-admin"
}
import {
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_resource_quota_v1.this
  id       = "tenant-${each.key}/tenant-quota"
}
import {
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_limit_range_v1.this
  id       = "tenant-${each.key}/tenant-defaults"
}
EOF
for p in default_deny:default-deny-all allow_dns:allow-dns \
         same_namespace:allow-same-namespace to_gateway:allow-egress-llm-gateway; do
cat >> imports.tf <<EOF
import {
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_network_policy_v1.${p%%:*}
  id       = "tenant-\${each.key}/${p##*:}"
}
EOF
done
cat >> imports.tf <<'EOF'
import {
  to = helm_release.metrics_server
  id = "kube-system/metrics-server"
}
EOF
tofu fmt imports.tf
tofu plan -no-color -out=tfplan | grep -E '^  # .*updated|^Plan:'

Expected (shortened):

- Installed hashicorp/kubernetes v3.3.0 (signed, key ID 0C0AF313E5FD9F80)
- Installed hashicorp/helm v3.3.0 (signed, key ID 0C0AF313E5FD9F80)
OpenTofu has been successfully initialized!
  # helm_release.metrics_server will be updated in-place
  # module.tenant["acme"].kubernetes_namespace_v1.this will be updated in-place
  # module.tenant["acme"].kubernetes_role_binding_v1.admin will be updated in-place
  # module.tenant["globex"].kubernetes_namespace_v1.this will be updated in-place
  # module.tenant["globex"].kubernetes_role_binding_v1.admin will be updated in-place
Plan: 19 to import, 0 to add, 5 to change, 0 to destroy.

Read every ~ update before the apply: for the helm_release, OpenTofu only takes the repository and the set values into the state; for the RoleBinding, the provider adds namespace = "default" to the group subject – meaningless for RBAC. For the namespaces, the plan shows the PSA labels as +, although they have been set since Day 6: the provider (3.3.0) apparently does not read labels with kubernetes.io in the key into the state on import. The apply changes nothing in the cluster, and afterwards the plan is empty.

tofu apply -no-color tfplan | grep 'Apply complete'
rm imports.tf tfplan
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"

Expected:

Apply complete! Resources: 19 imported, 0 added, 5 changed, 0 destroyed.
Exit-Code: 0

Warning: Import blocks with for_each = var.tenants also apply to every new tenant. If imports.tf stays around, the onboarding in Drill 4 fails with Missing Resource Identity After Read, because OpenTofu tries to import a namespace that does not exist.

Save your progress? Sign up for free

Drill 3: Understanding state

tofu state list | head -4
tofu state list | wc -l
tofu state show 'module.tenant["acme"].kubernetes_resource_quota_v1.this' \
  | grep -E '^#|gpu'

Expected:

helm_release.metrics_server
module.tenant["acme"].kubernetes_default_service_account_v1.this
module.tenant["acme"].kubernetes_limit_range_v1.this
module.tenant["acme"].kubernetes_namespace_v1.this
      19
# module.tenant["acme"].kubernetes_resource_quota_v1.this:
            "requests.nvidia.com/gpu" = "1"

Here the state lives locally in terraform.tfstate – including secrets, if resources contain any. In a team it belongs in a remote backend with locking and encryption (Enterprise Note).

Security: Never commit terraform.tfstate; the .gitignore in the code folder covers this.

Save your progress? Sign up for free

Drill 4: Detecting and healing drift

Someone deletes the default-deny policy “just for a quick debugging session”:

kubectl -n tenant-acme delete networkpolicy default-deny-all
tofu plan -no-color -detailed-exitcode > drift.txt; echo "Exit-Code: $?"
grep -E '^  # |^Plan:' drift.txt
tofu apply -auto-approve -no-color | grep 'Apply complete'

Expected:

networkpolicy.networking.k8s.io "default-deny-all" deleted from tenant-acme namespace
Exit-Code: 2
  # module.tenant["acme"].kubernetes_network_policy_v1.default_deny will be created
Plan: 1 to add, 0 to change, 0 to destroy.
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Exit codes of -detailed-exitcode: 0 no change, 1 error, 2 drift. A nightly CI job running this command is the simplest drift alarm.

Save your progress? Sign up for free

Drill 5: Onboarding and offboarding by variable

cat > initech.auto.tfvars <<'EOF'
tenants = {
  acme    = { team_group = "team-acme", gpu_quota = 1 }
  globex  = { team_group = "team-globex" }
  initech = { team_group = "team-initech", gpu_quota = 2 }
}
EOF
tofu apply -auto-approve -no-color | grep -E '^Plan:|Apply complete'
kubectl -n tenant-initech get quota tenant-quota \
  -o jsonpath='{.spec.hard.requests\.nvidia\.com/gpu}'; echo
rm initech.auto.tfvars
tofu plan -no-color | grep '^Plan:'
tofu apply -auto-approve -no-color | grep 'Apply complete'
kubectl get ns tenant-initech

Expected:

Plan: 9 to add, 0 to change, 0 to destroy.
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
2
Plan: 0 to add, 0 to change, 9 to destroy.
Apply complete! Resources: 0 added, 0 changed, 9 destroyed.
Error from server (NotFound): namespaces "tenant-initech" not found

Why *.auto.tfvars? OpenTofu loads them automatically after terraform.tfvars – later files override earlier ones. In real setups there is one file per tenant (or a YAML file read in with yamldecode()), and onboarding is a merge request.

Warning: Offboarding deletes the namespace with all its data (PVCs!). In production, use lifecycle { prevent_destroy = true } on the namespace and a deliberate decommissioning process with a backup.

Save your progress? Sign up for free

Drill 6: An Ansible test target without a VM

Without llm-vm, an Ubuntu container with systemd serves as the target; a stub replaces the Ollama installation (about 1.5 GB download):

mkdir -p ~/sre-lab/tag12/vmimg && cd ~/sre-lab/tag12/vmimg
cat > Dockerfile <<'EOF'
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
      systemd systemd-sysv python3 python3-apt sudo iptables \
    && rm -rf /var/lib/apt/lists/*
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]
EOF
docker build -q -t sre-lab/llm-vm:24.04 .
docker run -d --name llm-vm-test --hostname llm-vm-test --privileged \
  --cgroupns=host -v /sys/fs/cgroup:/sys/fs/cgroup:rw \
  --tmpfs /run --tmpfs /run/lock sre-lab/llm-vm:24.04
sleep 5
docker exec llm-vm-test systemctl is-system-running
docker exec llm-vm-test bash -c 'apt-get update -qq
  printf "#!/bin/sh\ncase \"\$1\" in serve) exec sleep infinity;; *) exit 0;; esac\n" \
    > /usr/local/bin/ollama && chmod +x /usr/local/bin/ollama'

Expected (shortened):

sha256:87217f98…
f636322a34fe…
running

Warning: If the container reports Failed to allocate manager object: Too many open files, the inotify instances of the Docker VM are exhausted (typical with kind). Workaround until the next VM restart: docker run --rm --privileged ubuntu:24.04 sysctl -w fs.inotify.max_user_instances=512.

Inventory for the test target – the Docker connection plugin needs no SSH:

cd ~/sre-lab/tag12/ansible-llm-vm
cat inventory/docker-test.ini
ansible-galaxy collection install -r requirements.yml
ansible-playbook site.yml --syntax-check
ansible-lint

Expected (shortened):

[llm_vms]
llm-vm-test ansible_connection=community.docker.docker

[llm_vms:vars]
llm_vm_gateway_cidr=172.18.0.0/16
llm_vm_ollama_skip_install=true

playbook: site.yml
Passed: 0 failure(s), 0 warning(s) in 9 files processed of 12 encountered.
Last profile that met the validation criteria was 'production'.

Why are all variables called llm_vm_…? ansible-lint (var-naming[no-role-prefix]) requires the role name as a prefix – otherwise gateway_cidr will collide with another role sooner or later.

Save your progress? Sign up for free

Drill 7: Applying the role, checking idempotency

The core pieces of the role – a drop-in with limits, and the firewall:

cat roles/llm_vm/templates/override.conf.j2
grep -A5 'Ollama nur vom Gateway' roles/llm_vm/tasks/main.yml

Expected:

# {{ ansible_managed }}
[Service]
Environment="OLLAMA_HOST={{ llm_vm_ollama_listen }}"
Environment="OLLAMA_MODELS={{ llm_vm_ollama_models_dir }}"
Environment="OLLAMA_KEEP_ALIVE={{ llm_vm_ollama_keep_alive }}"
MemoryMax={{ llm_vm_ollama_memory_max }}
CPUQuota={{ llm_vm_ollama_cpu_quota }}
LimitNOFILE={{ llm_vm_ollama_nofile }}
- name: UFW – Ollama nur vom Gateway
  community.general.ufw:
    rule: allow
    port: "11434"
    proto: tcp
    from_ip: "{{ llm_vm_gateway_cidr }}"

First check mode on the fresh host, then for real, then once more:

run() { ansible-playbook -i inventory/docker-test.ini site.yml "$@" \
  | grep -E '^fatal|^llm-vm-test +:' | cut -c1-80; }
run --check
run
run

Expected (shortened):

fatal: [llm-vm-test]: FAILED! => {"changed": false, "msg": "Could not find the
llm-vm-test                : ok=8    changed=7    unreachable=0    failed=1
llm-vm-test                : ok=16   changed=14   unreachable=0    failed=0
llm-vm-test                : ok=15   changed=0    unreachable=0    failed=0

Why does check mode fail? In a dry run the unit is never written – the systemd task finds no service. Check mode is a drift tool for hosts that are already set up, not a substitute for a test run. changed=0 in the third run is the proof of idempotency; ok drops from 16 to 15 because no handler runs anymore.

docker exec llm-vm-test bash -c 'systemctl is-active ollama prometheus-node-exporter
  systemctl show ollama -p MemoryMax -p LimitNOFILE; ufw status | grep ALLOW'

Expected:

active
active
MemoryMax=6442450944
LimitNOFILE=65536
22/tcp                     ALLOW       Anywhere
11434/tcp                  ALLOW       172.18.0.0/16
9100/tcp                   ALLOW       172.18.0.0/16

Save your progress? Sign up for free

Drill 8: Drift on the VM with --check --diff

docker exec llm-vm-test sed -i 's/MemoryMax=6G/MemoryMax=12G/' \
  /etc/systemd/system/ollama.service.d/override.conf
ansible-playbook -i inventory/docker-test.ini site.yml --check --diff \
  | grep -E '^[-+@]|^changed|HANDLER|^llm-vm-test +:' | cut -c1-80
run

Expected (shortened):

--- before: /etc/systemd/system/ollama.service.d/override.conf
+++ after: …/override.conf.j2
@@ -3,6 +3,6 @@
-MemoryMax=12G
+MemoryMax=6G
changed: [llm-vm-test]
RUNNING HANDLER [llm_vm : Restart ollama] ****************************************
changed: [llm-vm-test]
llm-vm-test                : ok=16   changed=2    unreachable=0    failed=0
llm-vm-test                : ok=16   changed=2    unreachable=0    failed=0

The dry run shows the deviation including the handler but changes nothing; only the real run (run) resets it to 6G and restarts Ollama.

For the real VM, just swap the inventory (inventory/hosts.ini, IP from multipass info llm-vm) and run ansible-playbook site.yml – then the role actually downloads Ollama 0.40.2 and ministral-3:3b.

And the VM itself? OpenTofu creates it, e.g. at Hetzner Cloud – checked only with tofu validate:

cd ~/sre-lab/tag12/tofu-tenant/examples/llm-vm-hcloud
grep -E '^resource|server_type|source_ips|^output' main.tf
tofu init -backend=false >/dev/null && tofu validate

Expected:

resource "hcloud_ssh_key" "admin" {
resource "hcloud_firewall" "llm_vm" {
    source_ips = ["0.0.0.0/0", "::/0"]
    source_ips = [var.gateway_cidr]
resource "hcloud_server" "llm_vm" {
  server_type  = "ccx33" # 8 dedizierte vCPU, 32 GB RAM
output "ansible_inventory" {
Success! The configuration is valid.

The chain in production: tofu apply (VM + cloud firewall) → output as the Ansible inventory → ansible-playbook site.yml. For on-prem you swap the provider (dmacvicar/libvirt, vSphere, Proxmox); for AWS, aws_instance with a GPU type (g6.xlarge) and a security group instead of the hcloud firewall.

Save your progress? Sign up for free

Checkpoint

cd ~/sre-lab/tag12/tofu-tenant
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "tofu validate" 'tofu validate'
check "kein Drift (Exit 0)" 'tofu plan -detailed-exitcode'
check "19 Objekte im State" '[ "$(tofu state list | wc -l)" -eq 19 ]'
check "imports.tf entfernt" '[ ! -f imports.tf ]'
check "State nicht im Git" 'grep -q tfstate ~/sre-lab/code/tofu-tenant/.gitignore'
cd ~/sre-lab/tag12/ansible-llm-vm
check "ansible-lint sauber" 'ansible-lint'
check "Idempotent (changed=0)" \
  'ansible-playbook -i inventory/docker-test.ini site.yml | grep -q "changed=0 "'
check "Ollama-Limit aktiv" \
  'docker exec llm-vm-test systemctl show ollama -p MemoryMax | grep -q 6442450944'

Challenge

A tenant may get at most 4 GPUs – more only with its own node pool. Enforce this in the root module so that tofu plan already fails, and prove it with a tenant hooli and 8 GPUs, without changing terraform.tfvars.

Hint: A validation block in variable "tenants" can check all map entries with alltrue([for …]).

Solution

cd ~/sre-lab/tag12/tofu-tenant
cat > variables.tf <<'EOF'
variable "kube_context" {
  type    = string
  default = "kind-sre-lab"
}

variable "tenants" {
  description = "Mandanten: Name => Einstellungen"
  type = map(object({
    team_group = string
    gpu_quota  = optional(number, 0)
  }))
  validation {
    condition     = alltrue([for t in values(var.tenants) : t.gpu_quota <= 4])
    error_message = "Maximal 4 GPUs pro Mandant – mehr nur mit eigenem Node-Pool."
  }
}
EOF
tofu plan -var 'tenants={hooli={team_group="team-hooli",gpu_quota=8}}' 2>&1 \
  | grep -E 'Error|Maximal'
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"

Expected:

│ Error: Invalid value for variable
│ Maximal 4 GPUs pro Mandant – mehr nur mit eigenem Node-Pool.
Exit-Code: 0

Clean up (optional; this also deletes the tenant namespaces from Day 6 and the metrics-server):

cd ~/sre-lab/tag12/tofu-tenant
tofu destroy -auto-approve -no-color | grep 'Destroy complete'
docker rm -f llm-vm-test

Save your progress? Sign up for free

Interview Note

  • Typical question: “Terraform or OpenTofu?” – OpenTofu is the MPL-licensed fork after Terraform’s switch to the BSL in 2023, compatible at the CLI and provider level; differences include state encryption and for_each in import blocks in OpenTofu. What counts for customers: license, registry, support contract.
  • Crisp: “For me, onboarding is a merge request: tenant in tfvars, plan in CI, review, apply – and a nightly plan -detailed-exitcode reports drift.”
  • Ansible vs. Tofu: Tofu for declarative APIs (cloud, Kubernetes), Ansible for the state inside the machine (packages, systemd, firewall). You prove idempotency with changed=0, not with hope.
  • Bridge from your own practice: you have already managed OpenBao policies, auth mounts and namespaces with Terraform/OpenTofu – including importing legacy resources and drift caused by click changes in the UI.
  • Counter-question: How does Mistral provision customer deployments – its own modules per cloud, Helm + GitOps, or an operator?

Enterprise Note

  • Remote state with locking (S3 + DynamoDB/S3 lockfile, GCS, Azure Blob, Postgres) and OpenTofu state encryption; access only for the CI identity.
  • Pin provider and module versions, commit .terraform.lock.hcl, modules from an internal registry/Git with tags.
  • In the long run, Kubernetes objects via GitOps (Argo CD/Flux) instead of Tofu when teams deploy themselves; Tofu for clusters, networks, IAM and the onboarding skeleton.
  • Ansible at customers on-prem: execution environments (containers), inventory from the CMDB/Tofu output, --check --diff as change evidence for the CAB.
  • Air gap: the Ollama binary and models from an internal mirror instead of ollama.com/install.sh (Day 14).