Skip to content
Hands-on IT IT Drills

Day 4: Kustomization in Depth

Goal: Give notes-api a database and master ordering, waiting, variables and suspending Kustomizations. · Duration: approx. 90 min · Prerequisite: Day 3

Briefing

The Kustomization has more fields than path and prune. Today the most important ones are added:

Field Effect
wait: true Ready only when all created objects are ready
timeout wait this long, then Ready=False (default: same as interval)
dependsOn apply only when another Kustomization is Ready
postBuild.substitute replace variables ${NAME} in the manifests
postBuild.substituteFrom fetch variables from a ConfigMap or a Secret
suspend: true pause reconciliation (flux suspend sets it for you)

The common thread: notes-api gets Postgres. The database is a separate Kustomization notes-db, and notes starts only when it is ready.

 Kustomization notes-db (wait)  ──Ready──►  Kustomization notes (dependsOn, wait)
   Namespace notes                            Secret notes-api
   Secret notes-db, PVC                       Deployment notes (+ migrate)
   Deployment + Service notes-db              Service notes :30080

Drill 1: Postgres

cd ~/flux-lab/fleet
mkdir -p apps/notes-db
cd apps/notes-db
cat > secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-db
  namespace: notes
stringData:
  password: lab-db-pass-123
EOF
cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: notes-db
  namespace: notes
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
EOF
cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-db
  namespace: notes
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: notes-db
  template:
    metadata:
      labels:
        app: notes-db
    spec:
      containers:
        - name: postgres
          image: postgres:18.6-alpine
          env:
            - name: POSTGRES_USER
              value: notes
            - name: POSTGRES_DB
              value: notes
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-db
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "notes", "-d", "notes"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: notes-db
EOF
cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes-db
  namespace: notes
spec:
  selector:
    app: notes-db
  ports:
    - port: 5432
EOF
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - secret.yaml
  - pvc.yaml
  - deployment.yaml
  - service.yaml
EOF
cd ~/flux-lab/fleet
  • strategy: Recreate: two Postgres pods must never write to the same volume at the same time. During an update, Kubernetes stops the old pod first.
  • The password is in plain text in Git, like the API key. Day 8 encrypts both.

The matching Kustomization, with --wait:

flux create kustomization notes-db --source=GitRepository/flux-system \
  --path=./apps/notes-db --prune=true --interval=10m --wait=true \
  --export > clusters/lab/notes-db.yaml
grep -A1 -E 'timeout|wait' clusters/lab/notes-db.yaml

Expected:

  timeout: 2m0s
  wait: true

With --wait, flux create sets a timeout of two minutes on its own.

Save your progress? Sign up for free

Drill 2: A namespace moves

The namespace notes is still in apps/notes. On a fresh cluster, notes-db would come first and find no namespace. So it has to move to notes-db.

Moving an object from one Kustomization to another is dangerous with Flux. The old Kustomization sees that the namespace has disappeared from its folder and removes it via prune. When a namespace goes, everything inside it goes too.

Tried it: A plain move in a single commit deleted the namespace notes in the lab and recreated it. notes-api was gone for a minute, and together with the namespace, the volume and all data disappeared as well. That is why you do it in two steps.

Step 1: The namespace gets an annotation that protects it from prune.

cat > apps/notes/namespace.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: notes
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled
EOF
git add -A
git commit -m "Postgres für notes, Namespace notes nie prunen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes

Expected (shortened):

✔ applied revision main@sha1:a10caaee769338dafa03c28963848786c0f2ea5a
...
✔ applied revision main@sha1:a10caaee769338dafa03c28963848786c0f2ea5a

Only once the annotation is on the namespace in the cluster is it protected.

Step 2: Now the move.

git mv apps/notes/namespace.yaml apps/notes-db/namespace.yaml
sed -i '/namespace.yaml/d' apps/notes/kustomization.yaml
sed -i 's/^resources:/resources:\n  - namespace.yaml/' apps/notes-db/kustomization.yaml
git commit -am "Namespace notes gehört jetzt notes-db"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
kubectl get ns notes -o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}{"\n"}'

Expected (shortened):

✔ applied revision main@sha1:8e28df7feaa7b27c26ebfa4bf242bd9257a15682
...
notes-db

The label kustomize.toolkit.fluxcd.io/name names the Kustomization that an object belongs to. The namespace now belongs to notes-db and has not been recreated:

kubectl -n notes get pods

Expected (shortened): the notes-api pods are older than the last few minutes, and notes-db is running.

Save your progress? Sign up for free

Drill 3: Connecting notes-api to the database

notes-api needs two new environment variables and an init container that creates the database schema before startup (migrate). Replace apps/notes/deployment.yaml:

cat > apps/notes/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  namespace: notes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      initContainers:
        - name: migrate
          image: ghcr.io/thomaszachmann/notes-api:0.5.3
          args: ["migrate"]
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-db
                  key: password
            - name: NOTES_DB_URL
              value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
      containers:
        - name: api
          image: ghcr.io/thomaszachmann/notes-api:0.5.3
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: notes-api
                  key: apiKey
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-db
                  key: password
            - name: NOTES_DB_URL
              value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
EOF

$(DB_PASSWORD) is not a shell variable but Kubernetes syntax: it inserts a previously defined environment variable.

The Kustomization notes gets dependsOn, wait and a timeout:

flux create kustomization notes --source=GitRepository/flux-system \
  --path=./apps/notes --prune=true --interval=5m --wait=true --timeout=2m \
  --depends-on=notes-db --export > clusters/lab/notes.yaml
git commit -am "notes-api mit Postgres"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
flux get kustomizations

Expected (shortened):

NAME        REVISION                SUSPENDED   READY   MESSAGE
flux-system main@sha1:60742105      False       True    Applied revision: main@sha1:60742105
notes       main@sha1:60742105      False       True    Applied revision: main@sha1:60742105
notes-db    main@sha1:60742105      False       True    Applied revision: main@sha1:60742105
podinfo     6.17.0@sha256:3a5e82b1  False       True    Applied revision: 6.17.0@sha256:3a5e82b1
registry    main@sha1:60742105      False       True    Applied revision: main@sha1:60742105

Now a note survives a restart:

curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo Flux"}' localhost:30080/notes
kubectl -n notes rollout restart deploy/notes
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/notes

Expected (shortened, the ID differs):

{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}
...
[{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}]

Save your progress? Sign up for free

Drill 4: What wait gives you

Without wait, a Kustomization reports Ready as soon as the API server has accepted the objects, regardless of whether the pods ever start. Try a version that does not exist:

sed -i 's/notes-api:0.5.3/notes-api:0.9.99/g' apps/notes/deployment.yaml
git commit -am "notes-api 0.9.99"
git push
flux reconcile source git flux-system
flux reconcile kustomization notes

The CLI waits now. After a little over three minutes (two minutes of timeout plus follow-up):

flux get kustomization notes

Expected:

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   main@sha1:60742105  False       False   health check failed after 2m0.122046047s: timeout waiting for: [Deployment/notes/notes status: 'InProgress']
kubectl -n notes get pods
curl -s localhost:30080/version

Expected (shortened):

NAME                        READY   STATUS              RESTARTS   AGE
notes-79c46dfb9c-zkq6d      0/1     Init:ErrImagePull   0          3m20s
notes-86b7bffdbb-4h7pf      1/1     Running             0          3m44s
notes-86b7bffdbb-nh9qc      1/1     Running             0          3m40s
...
0.5.3
  • The old pods keep running; the Kubernetes rolling update takes care of that.
  • Flux did apply the new manifest. REVISION still shows the old commit, because a revision only counts as “applied” once the health checks have passed.
  • READY=False is the signal you will hang alerts on later (Day 11).
git revert --no-edit HEAD
git push
flux reconcile source git flux-system
flux reconcile kustomization notes

Expected (shortened):

[main 39634fa] Revert "notes-api 0.9.99"
...
✔ applied revision main@sha1:39634fad5a6246cad3a6348f4a7d2439b4adba6c

Save your progress? Sign up for free

Drill 5: What dependsOn gives you

Now the database breaks:

sed -i 's/postgres:18.6-alpine/postgres:99-alpine/' apps/notes-db/deployment.yaml
git commit -am "Postgres 99"
git push
flux reconcile source git flux-system
flux get kustomizations notes notes-db

Expected (after a few seconds):

NAME        REVISION            SUSPENDED   READY   MESSAGE
notes       main@sha1:39634fad  False       False   dependency 'flux-system/notes-db' revision is not up to date
notes-db    main@sha1:39634fad  False       Unknown Reconciliation in progress

notes waits until notes-db has applied the new revision healthily. An update of notes-api would not be rolled out now. But dependsOn cannot protect the running database:

kubectl -n notes get pods

Expected (shortened):

NAME                        READY   STATUS             RESTARTS   AGE
notes-78cd6c995-khnw2       0/1     Running            0          105s
notes-78cd6c995-qfd2h       0/1     Running            0          101s
notes-db-784b5556cd-27kqw   0/1     ImagePullBackOff   0          76s

Recreate stopped the old Postgres before the new one was running. Without a database, the readiness probes of notes-api fail, and the API is unreachable. Back:

git revert --no-edit HEAD
git push
flux reconcile source git flux-system
flux reconcile kustomization notes-db
flux reconcile kustomization notes
curl -s localhost:30080/notes

Expected (shortened):

...
[{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}]

The note is still there; it lives on the volume.

Save your progress? Sign up for free

Drill 6: Variables

The notes-api version appears twice in the Deployment, and each cluster should be able to have its own number of replicas. For this, Flux replaces variables of the form ${NAME} after the Kustomize build:

sed -i 's/notes-api:0.5.3/notes-api:${NOTES_VERSION}/g' apps/notes/deployment.yaml
sed -i 's/replicas: 2/replicas: ${NOTES_REPLICAS:=1}/' apps/notes/deployment.yaml
grep -n -E 'image|replicas' apps/notes/deployment.yaml

Expected:

7:  replicas: ${NOTES_REPLICAS:=1}
18:          image: ghcr.io/thomaszachmann/notes-api:${NOTES_VERSION}
30:          image: ghcr.io/thomaszachmann/notes-api:${NOTES_VERSION}

${NOTES_REPLICAS:=1} has a default value, ${NOTES_VERSION} does not. The values come from two sources: a ConfigMap that applies to the whole cluster, and directly from the Kustomization.

cat > clusters/lab/cluster-vars.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-vars
  namespace: flux-system
data:
  CLUSTER_NAME: lab
  NOTES_VERSION: 0.5.3
EOF
cat >> clusters/lab/notes.yaml <<'EOF'
  postBuild:
    substitute:
      NOTES_REPLICAS: "2"
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
EOF

Before you push, you look at the result. flux build builds the manifests locally, with the variables from the cluster:

flux build kustomization notes --path ./apps/notes | grep -E 'replicas|image:'

Expected:

  replicas: 2
        image: ghcr.io/thomaszachmann/notes-api:0.5.3
        image: ghcr.io/thomaszachmann/notes-api:0.5.3
git add -A
git commit -m "notes: Version und Replicas als Variablen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
kubectl -n notes get deploy notes \
  -o jsonpath='{.spec.replicas} {.spec.template.spec.containers[0].image}{"\n"}'

Expected (shortened):

✔ applied revision main@sha1:d8043e69aaae6c67bf7369fb241be0e27680f2f2
2 ghcr.io/thomaszachmann/notes-api:0.5.3

flux diff shows what a change would do, again without pushing. As a test, set three replicas:

sed -i 's/NOTES_REPLICAS: "2"/NOTES_REPLICAS: "3"/' clusters/lab/notes.yaml
flux diff kustomization notes --path ./apps/notes \
  --kustomization-file clusters/lab/notes.yaml

Expected:

► Deployment/notes/notes drifted

metadata.generation
± value change
- 5
+ 6

spec.replicas
± value change
- 2
+ 3

⚠️ identified at least one change, exiting with non-zero exit code
git checkout clusters/lab/notes.yaml

That discards the change again. flux diff is the tool for pull requests: it shows before the merge what would happen in the cluster (Day 13).

Save your progress? Sign up for free

Drill 7: Suspending

During an incident, someone wants to intervene by hand without Flux rolling it back:

flux suspend kustomization notes
kubectl -n notes scale deploy notes --replicas=4
flux reconcile kustomization notes
flux get kustomization notes

Expected:

► suspending kustomization notes in flux-system namespace
✔ kustomization suspended
deployment.apps/notes scaled
✗ resource is suspended
NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   main@sha1:d8043e69  True        True    Applied revision: main@sha1:d8043e69

The four replicas stay. After the incident:

flux resume kustomization notes
kubectl -n notes get deploy notes

Expected (shortened):

✔ applied revision main@sha1:d8043e69aaae6c67bf7369fb241be0e27680f2f2
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   2/2     2            2           11m

resume reconciles immediately and restores the state from Git.

Save your progress? Sign up for free

Checkpoint

flux get kustomizations

Expected: five Kustomizations, all READY True.

flux get kustomization notes

Expected: SUSPENDED False.

curl -s localhost:30080/notes

Expected: your note “Hallo Flux”.

diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag04/apps

Expected: no output.

Challenge

Someone makes a typo in the ConfigMap:

sed -i 's/  NOTES_VERSION:/  NOTES_VERSON:/' clusters/lab/cluster-vars.yaml
git commit -am "Tippfehler"
git push
flux reconcile kustomization flux-system --with-source

What happens to notes-api? Does the image become notes-api: without a version?

Solution

flux get kustomization notes

Expected:

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   main@sha1:d8043e69  False       False   post build failed for 'Deployment.v1.apps/notes': envsubst error: variable substitution failed: variable not set (strict mode): "NOTES_VERSION"

Flux replaces variables in strict mode: if a variable without a default value is missing, the build aborts and nothing is applied. notes-api keeps running unchanged.

git revert --no-edit HEAD
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes

Expected (shortened):

...
✔ applied revision main@sha1:c1eb001d3263eb43ffb4e4ed64d3aaefddced91e

Save your progress? Sign up for free

Enterprise Note

  • Databases do not belong in a Deployment. In operations you use a Postgres operator such as CloudNativePG or a cloud database. A mistake like Drill 5 then does not end the only instance.
  • Protect dangerous objects: namespaces, PVCs and CRDs with data get kustomize.toolkit.fluxcd.io/prune: disabled. A folder accidentally deleted in Git then does not delete the production data.
  • Use variables sparingly: postBuild is handy for a few values per cluster. Differences between environments are better solved with Kustomize overlays (Day 5). There you see every file exactly as it is applied.