Skip to content
Hands-on IT IT Drills

Day 13: Quality and Tests

Goal: Safeguard the notes module so that formatting errors, dead variables, wrong inputs, and risky plans get noticed before anyone types apply. · Duration: approx. 90 min · Prerequisite: Day 5, Day 7

Briefing

Five checks, from fast and cheap to slow and thorough:

Tool checks needs
tofu fmt -check formatting nothing
tofu validate syntax, types, references tofu init
tflint style, dead variables, missing types ruleset
tofu test behavior: plan or real creation provider, possibly a target
Trivy, Conftest security, custom rules code or plan as JSON
  • tflint knows rules that validate doesn’t check – such as variables without a type or variables nobody uses.
  • tofu test runs *.tftest.hcl files. Each run block does a plan or an apply and checks conditions. After the test, OpenTofu cleans everything up again.
  • Trivy finds known misconfigurations in Terraform, Kubernetes, and Helm files. Conftest checks any JSON against custom rules in Rego – here the plan.

Drill 1: Installing the Tools

All three tools are single programs. tflint and Conftest go into ~/.local/bin, Trivy comes as a Debian package:

mkdir -p ~/.local/bin && cd /tmp
GH=https://github.com
curl -sSLO $GH/terraform-linters/tflint/releases/download/v0.65.0/tflint_linux_amd64.zip
unzip -o -q tflint_linux_amd64.zip -d ~/.local/bin && rm tflint_linux_amd64.zip
curl -sSLO $GH/aquasecurity/trivy/releases/download/v0.76.1/trivy_0.76.1_Linux-64bit.deb
sudo apt-get install -y -q ./trivy_0.76.1_Linux-64bit.deb >/dev/null 2>&1
rm trivy_*.deb
CT=conftest_0.71.1_Linux_x86_64.tar.gz
curl -sSLO $GH/open-policy-agent/conftest/releases/download/v0.71.1/$CT
tar -xzf $CT -C ~/.local/bin conftest && rm $CT
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/tf-lab/setup/env.sh
source ~/tf-lab/setup/env.sh
tflint --version | head -1; trivy --version | head -1; conftest --version | head -1

Expected:

TFLint version 0.65.0
Version: 0.76.1
Conftest: 0.71.1

GH and CT just shorten the long addresses. For a different version, you swap out the version number.

Save your progress? Sign up for free

Drill 2: fmt, validate, tflint

Two typical slip-ups in the module – a misaligned indentation and a variable nobody needs:

cd ~/tf-lab/module/notes-docker
printf '.terraform/\n.terraform.lock.hcl\n' > ../.gitignore
sed -i 's/^  name     = "${var.name}-api/  name = "${var.name}-api/' main.tf
printf '\nvariable "debug" {\n  default = false\n}\n' >> variables.tf
tofu fmt -check -diff -no-color .; echo "fmt: $?"
tofu init -no-color >/dev/null
tofu validate -no-color
tflint --format compact | cut -c30-
tflint >/dev/null; echo "tflint: $?"

Expected (shortened):

main.tf
--- old/main.tf
+++ new/main.tf
...
-  name = "${var.name}-api-${each.key}"
+  name     = "${var.name}-api-${each.key}"
...
fmt: 3
Success! The configuration is valid.
`debug` variable has no type (terraform_typed_variables)
variable "debug" is declared but not used (terraform_unused_declarations)
tflint: 2
  • cut -c30- just cuts off file, line, and severity at the front so the message fits on one line.
  • tofu fmt -check changes nothing and reports back with a non-zero exit code. Without -check it formats the files itself.
  • validate is happy: syntactically everything is correct. Only tflint finds the dead variable.
  • tflint already ships with the ruleset for Terraform code. For AWS, Azure, and Google there are separate rulesets you register in .tflint.hcl.

Revert both:

git checkout main.tf variables.tf
tofu fmt -check . && tflint && echo sauber

Expected:

sauber

Save your progress? Sign up for free

Drill 3: Validating and Testing Inputs

A port below 1024 is a mistake in the lab. A validation in the variable catches it; a test proves it:

cd ~/tf-lab/module/notes-docker
cat > variables.tf <<'EOF'
variable "name" {
  description = "Präfix für alle Namen, z. B. blau"
  type        = string
}

variable "port" {
  description = "Port des Proxys auf 127.0.0.1"
  type        = number

  validation {
    condition     = var.port >= 1024 && var.port <= 65535
    error_message = "port muss zwischen 1024 und 65535 liegen."
  }
}

variable "instanzen" {
  description = "Namen der notes-api-Instanzen"
  type        = set(string)
  default     = ["a"]
}

variable "db_passwort" {
  type      = string
  sensitive = true
}
EOF
git diff --stat

Expected:

 notes-docker/variables.tf | 5 +++++
 1 file changed, 5 insertions(+)

The only new thing is the validation block in port – git diff confirms that nothing else is different.

Now the tests. They live in tests/ next to the module:

mkdir -p tests
cat > tests/plan.tftest.hcl <<'EOF'
variables {
  name        = "test"
  port        = 8199
  db_passwort = "test-pw"
}

run "url_und_eine_instanz" {
  command = plan

  assert {
    condition     = output.url == "http://127.0.0.1:8199"
    error_message = "url stimmt nicht: ${output.url}"
  }

  assert {
    condition     = length(docker_container.api) == 1
    error_message = "Standard sollte genau eine Instanz sein."
  }
}

run "proxy_nur_lokal" {
  command = plan

  assert {
    condition     = docker_container.proxy.ports[0].ip == "127.0.0.1"
    error_message = "Der Proxy darf nur auf 127.0.0.1 lauschen."
  }
}

run "port_unter_1024_abgelehnt" {
  command = plan

  variables {
    port = 80
  }

  expect_failures = [var.port]
}
EOF
tofu test -no-color

Expected:

tests/plan.tftest.hcl... pass
  run "url_und_eine_instanz"... pass
  run "proxy_nur_lokal"... pass
  run "port_unter_1024_abgelehnt"... pass
Success! 3 passed, 0 failed.
  • variables at the top applies to all run blocks; a variables inside a block overrides individual values.
  • command = plan creates nothing. The tests run in seconds and only need the provider.
  • expect_failures reverses the logic: the test passes because the validation of var.port fires. That’s how you check that protective rules really kick in.

Save your progress? Sign up for free

Drill 4: A Test That Really Creates Things

A plan doesn’t prove that notes responds afterwards. A run without command = plan creates real containers. A small helper module then calls /healthz:

mkdir -p tests/http
cat > tests/http/main.tf <<'EOF'
terraform {
  required_providers {
    http = { source = "hashicorp/http", version = "~> 3.5" }
  }
}

variable "url" {
  type = string
}

data "http" "healthz" {
  url = var.url
  retry {
    attempts = 10
  }
}
EOF
cat > tests/apply.tftest.hcl <<'EOF'
variables {
  name        = "tst"
  port        = 8198
  db_passwort = "test-pw"
}

run "anlegen" {
  assert {
    condition     = length(output.container) == 3
    error_message = "Erwartet: db, proxy und eine api."
  }
}

run "healthz" {
  module {
    source = "./tests/http"
  }

  variables {
    url = "${run.anlegen.url}/healthz"
  }

  assert {
    condition     = data.http.healthz.status_code == 200
    error_message = "healthz antwortet nicht mit 200."
  }
}
EOF
tofu init -no-color >/dev/null
tofu test -no-color -filter=tests/apply.tftest.hcl
docker ps -a --format '{{.Names}}' | grep -c '^tst-'

Expected:

tests/apply.tftest.hcl... pass
  run "anlegen"... pass
  run "healthz"... pass
Success! 2 passed, 0 failed.
0
  • run.anlegen.url accesses the outputs of an earlier run.
  • After the last run, OpenTofu destroys everything the test created – that’s why grep counts 0 containers at the end.
  • Apply tests are slow and need a real target. Many teams run them only at night or before a release, and plan tests on every commit.

Record the new module version:

cd ~/tf-lab/module && git add -A
git -c user.name=Lab -c [email protected] commit -qm "Tests"
git tag v1.2.0 && git tag

Expected:

v1.0.0
v1.1.0
v1.2.0

Save your progress? Sign up for free

Drill 5: Trivy as a Security Gate

Trivy checks code for known misconfigurations. It has no rules for the module’s Docker resources, but it does for the Helm chart from Day 7. You check a copy:

mkdir -p ~/tf-lab/tag13 && cd ~/tf-lab/tag13
rm -rf chart && cp -r ~/tf-lab/code/charts/notes chart
trivy config --quiet --severity HIGH,CRITICAL chart \
  | grep -oE '^(Failures: .*|KSV-[0-9]+ \([A-Z]+\))'
trivy config --quiet --severity HIGH,CRITICAL --exit-code 1 chart >/dev/null
echo "Gate: $?"

Expected:

Failures: 3 (HIGH: 3, CRITICAL: 0)
KSV-0014 (HIGH)
KSV-0118 (HIGH)
KSV-0118 (HIGH)
Gate: 1
  • KSV-0014: the container’s file system is writable. KSV-0118: pod and container run with the default security context, so possibly as root.
  • --exit-code 1 turns the report into a gate: in a pipeline, the job aborts.

The fix is a securityContext for the pod and for each container. sed … r inserts a file after the matching line:

cat > pod-sc.yaml <<'EOF'
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        seccompProfile:
          type: RuntimeDefault
EOF
cat > container-sc.yaml <<'EOF'
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
EOF
D=chart/templates/deployment.yaml
sed -i '/^    spec:$/r pod-sc.yaml' $D
sed -i '/^          imagePullPolicy:/r container-sc.yaml' $D
grep -c 'securityContext' $D
trivy config --quiet --severity HIGH,CRITICAL --exit-code 1 chart >/dev/null
echo "Gate: $?"

Expected:

3
Gate: 0

Three matches: one for the pod, one each for the migration and the API container.

Without --severity, Trivy still shows three findings of severity LOW. Which severity triggers a gate is the team’s decision – and the team writes the exceptions, with a justification, into a .trivyignore.

Save your progress? Sign up for free

Drill 6: Custom Rules for the Plan with Conftest

Some rules no tool knows, because they apply only to your team: “No port on 0.0.0.0” and “Nothing gets deleted without approval”. Conftest checks them against the plan as JSON.

mkdir -p ~/tf-lab/policy
cat > ~/tf-lab/policy/plan.rego <<'EOF'
package main

deny contains msg if {
    some rc in input.resource_changes
    "delete" in rc.change.actions
    msg := sprintf("%s löscht – Freigabe nötig", [rc.address])
}

deny contains msg if {
    some rc in input.resource_changes
    rc.type == "docker_container"
    some p in rc.change.after.ports
    p.ip != "127.0.0.1"
    msg := sprintf("%s veröffentlicht Port %d auf %s", [rc.address, p.external, p.ip])
}
EOF
conftest fmt --check ~/tf-lab/policy && echo "Rego formatiert"

Expected:

Rego formatiert
  • input.resource_changes is the list of all planned changes. Each has an address, a type, and change.actions – such as ["create"] or ["delete", "create"] for a replacement.
  • Every deny rule that matches returns a message. One message is enough for Conftest to exit with an error.

First a plan without changes, then one that removes an instance in dev:

cd ~/tf-lab/tag10/envs/dev
tofu plan -no-color -out=ok.plan >/dev/null
tofu show -json ok.plan > ok.json
conftest test -p ~/tf-lab/policy ok.json --no-color
sed -i 's/^instanzen = \["a", "c"\]$/instanzen = ["a"]/' dev.auto.tfvars
tofu plan -no-color -out=weg.plan >/dev/null
tofu show -json weg.plan > weg.json
conftest test -p ~/tf-lab/policy weg.json --no-color; echo "Gate: $?"

Expected:

2 tests, 2 passed, 0 warnings, 0 failures, 0 exceptions
FAIL - weg.json - main - module.notes.docker_container.api["c"] löscht – Freigabe nötig
FAIL - weg.json - main - module.notes.docker_container.proxy löscht – Freigabe nötig
2 tests, 0 passed, 0 warnings, 2 failures, 0 exceptions
Gate: 1

The second message is instructive: the proxy is replaced (its configuration changes), and a replacement contains a delete. So the rule also catches replacements – exactly the kind that almost slipped through on Day 11.

Clean up – the plan files contain sensitive values in plain text:

sed -i 's/^instanzen = \["a"\]$/instanzen = ["a", "c"]/' dev.auto.tfvars
grep -c 'lab-db-pw-123' weg.json
rm ok.plan ok.json weg.plan weg.json
tofu plan -no-color | tail -1

Expected:

1
No changes. Your infrastructure matches the configuration.

Save your progress? Sign up for free

Checkpoint

cd ~/tf-lab/module/notes-docker && tofu test -no-color | tail -1

Expected: three plan tests and two apply tests.

Success! 5 passed, 0 failed.
cd ~/tf-lab/module && git tag --points-at HEAD

Expected:

v1.2.0
trivy config --quiet --severity HIGH,CRITICAL --exit-code 1 ~/tf-lab/tag13/chart \
  >/dev/null; echo $?

Expected:

0

Challenge

Write a plan test zwei_instanzen: with instanzen = ["a", "b"], the output instanzen must have the value 2 and the nginx configuration must contain both instances.

Hint: In the plan, the nginx configuration is under docker_container.proxy.upload. strcontains(text, teil) checks whether one text contains another.

Solution

cd ~/tf-lab/module/notes-docker
cat >> tests/plan.tftest.hcl <<'EOF'

run "zwei_instanzen" {
  command = plan

  variables {
    instanzen = ["a", "b"]
  }

  assert {
    condition     = output.instanzen == 2
    error_message = "Erwartet 2 Instanzen."
  }

  assert {
    condition = alltrue([
      for n in ["test-api-a", "test-api-b"] :
      strcontains(one(docker_container.proxy.upload).content, n)
    ])
    error_message = "nginx kennt nicht beide Instanzen."
  }
}
EOF
tofu test -no-color -filter=tests/plan.tftest.hcl | grep -E 'zwei_instanzen|Success'

Expected:

  run "zwei_instanzen"... pass
Success! 4 passed, 0 failed.

one() fetches the single element from the set of upload blocks. Without one() you’d have to guess an index – sets don’t have one.

Save your progress? Sign up for free

Enterprise Note

  • Checks in order of cost: fmt and tflint in seconds before every commit (pre-commit hook), plan tests and Conftest in every pull request, apply tests at night in a dedicated test environment.
  • Policy as code, centrally: Rego rules live in their own repository with their own reviews. All pipelines pull the same version – like a module.
  • Plan JSON is a secret: tofu show -json contains sensitive values in plain text. In CI it belongs in a protected artifact with short retention, never in a public log.