Skip to content
Hands-on IT IT Drills

Day 6: notes on Kubernetes

Goal: Bring notes to k3s with the Kubernetes provider – namespace, Secret, Deployment, Service, and Ingress –, detect drift in the cluster, and deliberately ignore what someone else manages. · Duration: approx. 90 min · Prerequisite: Day 5

Briefing

  • The Kubernetes provider talks to the cluster’s API. Every Kubernetes kind has its own resource: kubernetes_namespace_v1, kubernetes_deployment_v1 … The fields are named as in YAML, just in snake_case and as blocks.
  • Unlike kubectl apply, OpenTofu remembers what it created: if you delete a resource from the code, apply deletes it in the cluster.
  • kubernetes_deployment_v1 waits until the rollout has finished – an apply only ends once the pods are running.
  • lifecycle { ignore_changes = [ … ] } leaves fields alone that someone else manages, such as the number of replicas with an autoscaler.
 tag06/*.tf ─► Provider kubernetes ─► k3s API
   Namespace notes ─► Secret notes-api ─► Deployment notes-api ─► Service ─► Ingress
                                         http://notes.<VM_IP>.sslip.io (Traefik)

Drill 1: Provider and Namespace

k3s uses its own container runtime, not Docker. The locally built image has to be transferred once:

docker save notes-api:0.4.0 | sudo k3s ctr images import - 2>&1 | tail -1 | cut -c1-40
sudo k3s ctr images ls -q | grep notes-api
mkdir -p ~/tf-lab/tag06 && cd ~/tf-lab/tag06
cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.13.0"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 3.4"
    }
  }
}

provider "kubernetes" {
  config_path = "/etc/rancher/k3s/k3s.yaml"
}
EOF
cat > namespace.tf <<'EOF'
resource "kubernetes_namespace_v1" "notes" {
  metadata {
    name = "notes"
    labels = {
      "app.kubernetes.io/managed-by" = "opentofu"
    }
  }
}
EOF
tofu init -no-color | grep -E 'Installed|initialized'
tofu apply -auto-approve -no-color | grep 'Apply complete'
kubectl get namespace notes --show-labels

Expected (age differs):

unpacking docker.io/library/notes-api:0.4.0
docker.io/library/notes-api:0.4.0
- Installed hashicorp/kubernetes v3.4.0 (signed, key ID 0C0AF313E5FD9F80)
OpenTofu has been successfully initialized!
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
NAME    STATUS   AGE   LABELS
notes   Active   5s    app.kubernetes.io/managed-by=opentofu,kubernetes.io/metadata.na…

config_path points to the k3s kubeconfig. For a cluster in the cloud, you’d find address, certificate, and token there – or a command that fetches them.

Save your progress? Sign up for free

Drill 2: Secret and Deployment

cat > variables.tf <<'EOF'
variable "api_key" {
  type      = string
  sensitive = true
  default   = "lab-api-key-123"
}

variable "vm_ip" {
  description = "IP der VM für den Hostnamen <name>.<IP>.sslip.io"
  type        = string
}
EOF
cat > notes.tf <<'EOF'
resource "kubernetes_secret_v1" "api" {
  metadata {
    name      = "notes-api"
    namespace = kubernetes_namespace_v1.notes.metadata[0].name
  }
  data = {
    NOTES_API_KEY = var.api_key
  }
}

resource "kubernetes_deployment_v1" "api" {
  metadata {
    name      = "notes-api"
    namespace = kubernetes_namespace_v1.notes.metadata[0].name
    labels    = { app = "notes-api" }
  }
  spec {
    replicas = 1
    selector {
      match_labels = { app = "notes-api" }
    }
    template {
      metadata {
        labels = { app = "notes-api" }
      }
      spec {
        container {
          name              = "notes-api"
          image             = "notes-api:0.4.0"
          image_pull_policy = "Never"
          port {
            container_port = 8080
          }
          env_from {
            secret_ref {
              name = kubernetes_secret_v1.api.metadata[0].name
            }
          }
          readiness_probe {
            http_get {
              path = "/healthz"
              port = 8080
            }
            period_seconds = 2
          }
          resources {
            requests = { cpu = "50m", memory = "32Mi" }
            limits   = { memory = "64Mi" }
          }
        }
      }
    }
  }
}
EOF
export TF_VAR_vm_ip=$VM_IP
tofu apply -auto-approve -no-color | grep -E 'Creation complete|Apply complete'
kubectl -n notes get pods -l app=notes-api

Expected (IDs, pod name, and age differ):

kubernetes_secret_v1.api: Creation complete after 0s [id=notes/notes-api]
kubernetes_deployment_v1.api: Creation complete after 4s [id=notes/notes-api]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
NAME                         READY   STATUS    RESTARTS   AGE
notes-api-576b8c9b56-7bq4x   1/1     Running   0          5s
  • image_pull_policy = "Never": the image exists only locally in k3s; there is no registry.
  • Without NOTES_DB_URL, notes-api stores data in memory – enough for a start. Day 7 brings Postgres.
  • The namespace reference kubernetes_namespace_v1.notes.metadata[0].name ensures the right order: first the namespace, then everything in it.

Save your progress? Sign up for free

Drill 3: Service and Ingress

cat > netz.tf <<'EOF'
resource "kubernetes_service_v1" "api" {
  metadata {
    name      = "notes-api"
    namespace = kubernetes_namespace_v1.notes.metadata[0].name
  }
  spec {
    selector = kubernetes_deployment_v1.api.spec[0].template[0].metadata[0].labels
    port {
      port        = 80
      target_port = 8080
    }
  }
}

resource "kubernetes_ingress_v1" "api" {
  metadata {
    name      = "notes-api"
    namespace = kubernetes_namespace_v1.notes.metadata[0].name
  }
  spec {
    ingress_class_name = "traefik"
    rule {
      host = "notes.${var.vm_ip}.sslip.io"
      http {
        path {
          path      = "/"
          path_type = "Prefix"
          backend {
            service {
              name = kubernetes_service_v1.api.metadata[0].name
              port {
                number = 80
              }
            }
          }
        }
      }
    }
  }
}

output "url" {
  value = "http://notes.${var.vm_ip}.sslip.io"
}
EOF
tofu apply -auto-approve -no-color | grep 'Apply complete'
U=$(tofu output -raw url)
until curl -sf -o /dev/null "$U/healthz"; do sleep 2; done
curl -s -X POST -H "X-API-Key: $NOTES_API_KEY" -H 'Content-Type: application/json' \
  -d '{"text":"aus Kubernetes"}' "$U/notes" | jq -r .text
curl -s "$U/notes" | jq -r '.[].text'

Expected:

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
aus Kubernetes
aus Kubernetes

The Service selects the pods via the same labels as the Deployment – read here directly from its spec so the two never drift apart.

Save your progress? Sign up for free

Drill 4: Drift and ignore_changes

kubectl -n notes scale deployment notes-api --replicas=3 >/dev/null
tofu plan -no-color | grep -E 'replicas|Plan:'
tofu apply -auto-approve -no-color | grep 'Apply complete'
kubectl -n notes get deployment notes-api -o jsonpath='{.spec.replicas}'; echo

Expected:

          ~ replicas                  = "3" -> "1"
Plan: 0 to add, 1 to change, 0 to destroy.
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
1

OpenTofu resets the cluster to the code. But if an autoscaler is supposed to control the replicas, the two fight each other. Then the field belongs to the other one:

printf '  lifecycle {\n    ignore_changes = [spec[0].replicas]\n  }\n' > ignore.txt
sed -i '/^resource "kubernetes_deployment_v1" "api" {$/r ignore.txt' notes.tf
kubectl -n notes scale deployment notes-api --replicas=2 >/dev/null
tofu plan -no-color | grep -E 'Plan:|No changes'

Expected:

No changes. Your infrastructure matches the configuration.

ignore_changes only applies to changes from the outside. The initial value from the code is still set on first creation.

Save your progress? Sign up for free

Drill 5: What OpenTofu Doesn’t Know

kubectl -n notes create configmap von-hand --from-literal=x=1 >/dev/null
tofu plan -no-color | grep -E 'Plan:|No changes'
kubectl -n notes get configmap

Expected (age differs):

No changes. Your infrastructure matches the configuration.
NAME               DATA   AGE
kube-root-ca.crt   1      10m
von-hand           1      1s

OpenTofu only sees what’s in its state. It doesn’t notice a ConfigMap created by hand in the same namespace – only when the namespace is deleted does it disappear along with it. If you manage a namespace with OpenTofu, don’t create anything in it by hand.

Save your progress? Sign up for free

Checkpoint

curl -s -o /dev/null -w '%{http_code}\n' "$(tofu output -raw url)/healthz"

Expected:

200
tofu state list

Expected:

kubernetes_deployment_v1.api
kubernetes_ingress_v1.api
kubernetes_namespace_v1.notes
kubernetes_secret_v1.api
kubernetes_service_v1.api
kubectl -n notes get secret notes-api -o jsonpath='{.data.NOTES_API_KEY}' \
  | base64 -d; echo

Expected:

lab-api-key-123

Challenge

Change the API key to lab-api-key-456. The Deployment should then start new pods so they read the new key – without you restarting anything by hand.

Hint: Pods read a Secret only at startup. An annotation on the pod template with a hash of the Secret’s content (sha256(...)) forces a rollout as soon as the content changes.

Solution

zeile='        annotations = { "notes/secret-hash" = sha256(var.api_key) }'
echo "$zeile" > hash.txt
sed -i '/^    template {$/{n;r hash.txt
}' notes.tf
tofu fmt >/dev/null
tofu apply -auto-approve -no-color -var api_key=lab-api-key-456 | grep 'Apply complete'
U=$(tofu output -raw url)
until curl -sf -o /dev/null "$U/healthz"; do sleep 2; done
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "X-API-Key: lab-api-key-456" \
  -H 'Content-Type: application/json' -d '{"text":"neuer Key"}' "$U/notes"
tofu apply -auto-approve -no-color | grep 'Apply complete'

Expected:

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.
201
Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

The second run without -var puts the default key back – and rolls out again. If the Secret changes, the hash changes, and Kubernetes restarts the pods.

Save your progress? Sign up for free

Enterprise Note

  • Raw resources or Helm? The Kubernetes provider is a good fit for namespaces, permissions, and baseline setup. Applications usually come as a Helm chart (Day 7) or via GitOps (Argo CD, Flux).
  • Separate cluster and content: If you build the cluster itself with OpenTofu, manage its content in a separate configuration. A provider whose credentials only come into existence in the same run leads to dependencies that are hard to resolve.
  • Secrets in the state: The value of kubernetes_secret_v1 is stored in the state in plain text – another reason for an encrypted remote state (Day 9).