Day 6: notes on Kubernetes
Goal: Bring notes to k3s with the Kubernetes provider – namespace, Secret, Deployment, Service, and Ingress –, detect drift in the cluster, and deliberately ignore what someone else manages. · Duration: approx. 90 min · Prerequisite: Day 5
Briefing
- The Kubernetes provider talks to the cluster’s API.
Every Kubernetes kind has its own resource:
kubernetes_namespace_v1,kubernetes_deployment_v1… The fields are named as in YAML, just insnake_caseand as blocks. - Unlike
kubectl apply, OpenTofu remembers what it created: if you delete a resource from the code,applydeletes it in the cluster. kubernetes_deployment_v1waits until the rollout has finished – anapplyonly ends once the pods are running.lifecycle { ignore_changes = [ … ] }leaves fields alone that someone else manages, such as the number of replicas with an autoscaler.
tag06/*.tf ─► Provider kubernetes ─► k3s API
Namespace notes ─► Secret notes-api ─► Deployment notes-api ─► Service ─► Ingress
http://notes.<VM_IP>.sslip.io (Traefik)
Drill 1: Provider and Namespace
k3s uses its own container runtime, not Docker. The locally built image has to be transferred once:
docker save notes-api:0.4.0 | sudo k3s ctr images import - 2>&1 | tail -1 | cut -c1-40
sudo k3s ctr images ls -q | grep notes-api
mkdir -p ~/tf-lab/tag06 && cd ~/tf-lab/tag06
cat > versions.tf <<'EOF'
terraform {
required_version = ">= 1.13.0"
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 3.4"
}
}
}
provider "kubernetes" {
config_path = "/etc/rancher/k3s/k3s.yaml"
}
EOF
cat > namespace.tf <<'EOF'
resource "kubernetes_namespace_v1" "notes" {
metadata {
name = "notes"
labels = {
"app.kubernetes.io/managed-by" = "opentofu"
}
}
}
EOF
tofu init -no-color | grep -E 'Installed|initialized'
tofu apply -auto-approve -no-color | grep 'Apply complete'
kubectl get namespace notes --show-labelsExpected (age differs):
unpacking docker.io/library/notes-api:0.4.0
docker.io/library/notes-api:0.4.0
- Installed hashicorp/kubernetes v3.4.0 (signed, key ID 0C0AF313E5FD9F80)
OpenTofu has been successfully initialized!
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
NAME STATUS AGE LABELS
notes Active 5s app.kubernetes.io/managed-by=opentofu,kubernetes.io/metadata.na…
config_path points to the k3s kubeconfig. For a cluster
in the cloud, you’d find address, certificate, and token there – or a
command that fetches them.
Save your progress? Sign up for free
Drill 2: Secret and Deployment
cat > variables.tf <<'EOF'
variable "api_key" {
type = string
sensitive = true
default = "lab-api-key-123"
}
variable "vm_ip" {
description = "IP der VM für den Hostnamen <name>.<IP>.sslip.io"
type = string
}
EOF
cat > notes.tf <<'EOF'
resource "kubernetes_secret_v1" "api" {
metadata {
name = "notes-api"
namespace = kubernetes_namespace_v1.notes.metadata[0].name
}
data = {
NOTES_API_KEY = var.api_key
}
}
resource "kubernetes_deployment_v1" "api" {
metadata {
name = "notes-api"
namespace = kubernetes_namespace_v1.notes.metadata[0].name
labels = { app = "notes-api" }
}
spec {
replicas = 1
selector {
match_labels = { app = "notes-api" }
}
template {
metadata {
labels = { app = "notes-api" }
}
spec {
container {
name = "notes-api"
image = "notes-api:0.4.0"
image_pull_policy = "Never"
port {
container_port = 8080
}
env_from {
secret_ref {
name = kubernetes_secret_v1.api.metadata[0].name
}
}
readiness_probe {
http_get {
path = "/healthz"
port = 8080
}
period_seconds = 2
}
resources {
requests = { cpu = "50m", memory = "32Mi" }
limits = { memory = "64Mi" }
}
}
}
}
}
}
EOF
export TF_VAR_vm_ip=$VM_IP
tofu apply -auto-approve -no-color | grep -E 'Creation complete|Apply complete'
kubectl -n notes get pods -l app=notes-apiExpected (IDs, pod name, and age differ):
kubernetes_secret_v1.api: Creation complete after 0s [id=notes/notes-api]
kubernetes_deployment_v1.api: Creation complete after 4s [id=notes/notes-api]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
NAME READY STATUS RESTARTS AGE
notes-api-576b8c9b56-7bq4x 1/1 Running 0 5s
image_pull_policy = "Never": the image exists only locally in k3s; there is no registry.- Without
NOTES_DB_URL, notes-api stores data in memory – enough for a start. Day 7 brings Postgres. - The namespace reference
kubernetes_namespace_v1.notes.metadata[0].nameensures the right order: first the namespace, then everything in it.
Save your progress? Sign up for free
Drill 3: Service and Ingress
cat > netz.tf <<'EOF'
resource "kubernetes_service_v1" "api" {
metadata {
name = "notes-api"
namespace = kubernetes_namespace_v1.notes.metadata[0].name
}
spec {
selector = kubernetes_deployment_v1.api.spec[0].template[0].metadata[0].labels
port {
port = 80
target_port = 8080
}
}
}
resource "kubernetes_ingress_v1" "api" {
metadata {
name = "notes-api"
namespace = kubernetes_namespace_v1.notes.metadata[0].name
}
spec {
ingress_class_name = "traefik"
rule {
host = "notes.${var.vm_ip}.sslip.io"
http {
path {
path = "/"
path_type = "Prefix"
backend {
service {
name = kubernetes_service_v1.api.metadata[0].name
port {
number = 80
}
}
}
}
}
}
}
}
output "url" {
value = "http://notes.${var.vm_ip}.sslip.io"
}
EOF
tofu apply -auto-approve -no-color | grep 'Apply complete'
U=$(tofu output -raw url)
until curl -sf -o /dev/null "$U/healthz"; do sleep 2; done
curl -s -X POST -H "X-API-Key: $NOTES_API_KEY" -H 'Content-Type: application/json' \
-d '{"text":"aus Kubernetes"}' "$U/notes" | jq -r .text
curl -s "$U/notes" | jq -r '.[].text'Expected:
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
aus Kubernetes
aus Kubernetes
The Service selects the pods via the same labels as the Deployment – read here directly from its spec so the two never drift apart.
Save your progress? Sign up for free
Drill 4: Drift and ignore_changes
kubectl -n notes scale deployment notes-api --replicas=3 >/dev/null
tofu plan -no-color | grep -E 'replicas|Plan:'
tofu apply -auto-approve -no-color | grep 'Apply complete'
kubectl -n notes get deployment notes-api -o jsonpath='{.spec.replicas}'; echoExpected:
~ replicas = "3" -> "1"
Plan: 0 to add, 1 to change, 0 to destroy.
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
1
OpenTofu resets the cluster to the code. But if an autoscaler is supposed to control the replicas, the two fight each other. Then the field belongs to the other one:
printf ' lifecycle {\n ignore_changes = [spec[0].replicas]\n }\n' > ignore.txt
sed -i '/^resource "kubernetes_deployment_v1" "api" {$/r ignore.txt' notes.tf
kubectl -n notes scale deployment notes-api --replicas=2 >/dev/null
tofu plan -no-color | grep -E 'Plan:|No changes'Expected:
No changes. Your infrastructure matches the configuration.
ignore_changes only applies to changes from the outside.
The initial value from the code is still set on first creation.
Save your progress? Sign up for free
Drill 5: What OpenTofu Doesn’t Know
kubectl -n notes create configmap von-hand --from-literal=x=1 >/dev/null
tofu plan -no-color | grep -E 'Plan:|No changes'
kubectl -n notes get configmapExpected (age differs):
No changes. Your infrastructure matches the configuration.
NAME DATA AGE
kube-root-ca.crt 1 10m
von-hand 1 1s
OpenTofu only sees what’s in its state. It doesn’t notice a ConfigMap created by hand in the same namespace – only when the namespace is deleted does it disappear along with it. If you manage a namespace with OpenTofu, don’t create anything in it by hand.
Save your progress? Sign up for free
Checkpoint
curl -s -o /dev/null -w '%{http_code}\n' "$(tofu output -raw url)/healthz"Expected:
200
tofu state listExpected:
kubernetes_deployment_v1.api
kubernetes_ingress_v1.api
kubernetes_namespace_v1.notes
kubernetes_secret_v1.api
kubernetes_service_v1.api
kubectl -n notes get secret notes-api -o jsonpath='{.data.NOTES_API_KEY}' \
| base64 -d; echoExpected:
lab-api-key-123
Challenge
Change the API key to lab-api-key-456. The Deployment
should then start new pods so they read the new key – without you
restarting anything by hand.
Hint: Pods read a Secret only at startup. An
annotation on the pod template with a hash of the Secret’s content
(sha256(...)) forces a rollout as soon as the content
changes.
Solution
zeile=' annotations = { "notes/secret-hash" = sha256(var.api_key) }'
echo "$zeile" > hash.txt
sed -i '/^ template {$/{n;r hash.txt
}' notes.tf
tofu fmt >/dev/null
tofu apply -auto-approve -no-color -var api_key=lab-api-key-456 | grep 'Apply complete'
U=$(tofu output -raw url)
until curl -sf -o /dev/null "$U/healthz"; do sleep 2; done
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "X-API-Key: lab-api-key-456" \
-H 'Content-Type: application/json' -d '{"text":"neuer Key"}' "$U/notes"
tofu apply -auto-approve -no-color | grep 'Apply complete'Expected:
Apply complete! Resources: 0 added, 2 changed, 0 destroyed.
201
Apply complete! Resources: 0 added, 2 changed, 0 destroyed.
The second run without -var puts the default key back –
and rolls out again. If the Secret changes, the hash changes, and
Kubernetes restarts the pods.
Save your progress? Sign up for free
Enterprise Note
- Raw resources or Helm? The Kubernetes provider is a good fit for namespaces, permissions, and baseline setup. Applications usually come as a Helm chart (Day 7) or via GitOps (Argo CD, Flux).
- Separate cluster and content: If you build the cluster itself with OpenTofu, manage its content in a separate configuration. A provider whose credentials only come into existence in the same run leads to dependencies that are hard to resolve.
- Secrets in the state: The value of
kubernetes_secret_v1is stored in the state in plain text – another reason for an encrypted remote state (Day 9).