Day 4: Kustomization in Depth
Goal: Give notes-api a database and master ordering, waiting, variables and suspending Kustomizations. · Duration: approx. 90 min · Prerequisite: Day 3
Briefing
The Kustomization has more fields than path and
prune. Today the most important ones are added:
| Field | Effect |
|---|---|
wait: true |
Ready only when all created objects are ready |
timeout |
wait this long, then Ready=False (default: same as
interval) |
dependsOn |
apply only when another Kustomization is Ready |
postBuild.substitute |
replace variables ${NAME} in the manifests |
postBuild.substituteFrom |
fetch variables from a ConfigMap or a Secret |
suspend: true |
pause reconciliation (flux suspend sets it for
you) |
The common thread: notes-api gets Postgres. The database is a
separate Kustomization notes-db, and notes
starts only when it is ready.
Kustomization notes-db (wait) ──Ready──► Kustomization notes (dependsOn, wait)
Namespace notes Secret notes-api
Secret notes-db, PVC Deployment notes (+ migrate)
Deployment + Service notes-db Service notes :30080
Drill 1: Postgres
cd ~/flux-lab/fleet
mkdir -p apps/notes-db
cd apps/notes-db
cat > secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: notes-db
namespace: notes
stringData:
password: lab-db-pass-123
EOF
cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: notes-db
namespace: notes
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
EOF
cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes-db
namespace: notes
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: notes-db
template:
metadata:
labels:
app: notes-db
spec:
containers:
- name: postgres
image: postgres:18.6-alpine
env:
- name: POSTGRES_USER
value: notes
- name: POSTGRES_DB
value: notes
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: notes-db
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- containerPort: 5432
readinessProbe:
exec:
command: ["pg_isready", "-U", "notes", "-d", "notes"]
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: notes-db
EOF
cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: notes-db
namespace: notes
spec:
selector:
app: notes-db
ports:
- port: 5432
EOF
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- secret.yaml
- pvc.yaml
- deployment.yaml
- service.yaml
EOF
cd ~/flux-lab/fleetstrategy: Recreate: two Postgres pods must never write to the same volume at the same time. During an update, Kubernetes stops the old pod first.- The password is in plain text in Git, like the API key. Day 8 encrypts both.
The matching Kustomization, with --wait:
flux create kustomization notes-db --source=GitRepository/flux-system \
--path=./apps/notes-db --prune=true --interval=10m --wait=true \
--export > clusters/lab/notes-db.yaml
grep -A1 -E 'timeout|wait' clusters/lab/notes-db.yamlExpected:
timeout: 2m0s
wait: true
With --wait, flux create sets a timeout of
two minutes on its own.
Save your progress? Sign up for free
Drill 2: A namespace moves
The namespace notes is still in apps/notes.
On a fresh cluster, notes-db would come first and find no
namespace. So it has to move to notes-db.
Moving an object from one Kustomization to another is dangerous with
Flux. The old Kustomization sees that the namespace has disappeared from
its folder and removes it via prune. When a namespace goes,
everything inside it goes too.
Tried it: A plain move in a single commit deleted the namespace
notesin the lab and recreated it. notes-api was gone for a minute, and together with the namespace, the volume and all data disappeared as well. That is why you do it in two steps.
Step 1: The namespace gets an annotation that
protects it from prune.
cat > apps/notes/namespace.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: notes
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
EOF
git add -A
git commit -m "Postgres für notes, Namespace notes nie prunen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notesExpected (shortened):
✔ applied revision main@sha1:a10caaee769338dafa03c28963848786c0f2ea5a
...
✔ applied revision main@sha1:a10caaee769338dafa03c28963848786c0f2ea5a
Only once the annotation is on the namespace in the cluster is it protected.
Step 2: Now the move.
git mv apps/notes/namespace.yaml apps/notes-db/namespace.yaml
sed -i '/namespace.yaml/d' apps/notes/kustomization.yaml
sed -i 's/^resources:/resources:\n - namespace.yaml/' apps/notes-db/kustomization.yaml
git commit -am "Namespace notes gehört jetzt notes-db"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
kubectl get ns notes -o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}{"\n"}'Expected (shortened):
✔ applied revision main@sha1:8e28df7feaa7b27c26ebfa4bf242bd9257a15682
...
notes-db
The label kustomize.toolkit.fluxcd.io/name names the
Kustomization that an object belongs to. The namespace now belongs to
notes-db and has not been recreated:
kubectl -n notes get podsExpected (shortened): the notes-api pods are older than the last few
minutes, and notes-db is running.
Save your progress? Sign up for free
Drill 3: Connecting notes-api to the database
notes-api needs two new environment variables and an init container
that creates the database schema before startup (migrate).
Replace apps/notes/deployment.yaml:
cat > apps/notes/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
namespace: notes
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
initContainers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.5.3
args: ["migrate"]
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: notes-db
key: password
- name: NOTES_DB_URL
value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
containers:
- name: api
image: ghcr.io/thomaszachmann/notes-api:0.5.3
ports:
- containerPort: 8080
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: notes-api
key: apiKey
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: notes-db
key: password
- name: NOTES_DB_URL
value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
readinessProbe:
httpGet:
path: /healthz
port: 8080
EOF$(DB_PASSWORD) is not a shell variable but Kubernetes
syntax: it inserts a previously defined environment variable.
The Kustomization notes gets dependsOn,
wait and a timeout:
flux create kustomization notes --source=GitRepository/flux-system \
--path=./apps/notes --prune=true --interval=5m --wait=true --timeout=2m \
--depends-on=notes-db --export > clusters/lab/notes.yaml
git commit -am "notes-api mit Postgres"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
flux get kustomizationsExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
flux-system main@sha1:60742105 False True Applied revision: main@sha1:60742105
notes main@sha1:60742105 False True Applied revision: main@sha1:60742105
notes-db main@sha1:60742105 False True Applied revision: main@sha1:60742105
podinfo 6.17.0@sha256:3a5e82b1 False True Applied revision: 6.17.0@sha256:3a5e82b1
registry main@sha1:60742105 False True Applied revision: main@sha1:60742105
Now a note survives a restart:
curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo Flux"}' localhost:30080/notes
kubectl -n notes rollout restart deploy/notes
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/notesExpected (shortened, the ID differs):
{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}
...
[{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}]
Save your progress? Sign up for free
Drill 4: What wait gives you
Without wait, a Kustomization reports Ready
as soon as the API server has accepted the objects, regardless of
whether the pods ever start. Try a version that does not exist:
sed -i 's/notes-api:0.5.3/notes-api:0.9.99/g' apps/notes/deployment.yaml
git commit -am "notes-api 0.9.99"
git push
flux reconcile source git flux-system
flux reconcile kustomization notesThe CLI waits now. After a little over three minutes (two minutes of timeout plus follow-up):
flux get kustomization notesExpected:
NAME REVISION SUSPENDED READY MESSAGE
notes main@sha1:60742105 False False health check failed after 2m0.122046047s: timeout waiting for: [Deployment/notes/notes status: 'InProgress']
kubectl -n notes get pods
curl -s localhost:30080/versionExpected (shortened):
NAME READY STATUS RESTARTS AGE
notes-79c46dfb9c-zkq6d 0/1 Init:ErrImagePull 0 3m20s
notes-86b7bffdbb-4h7pf 1/1 Running 0 3m44s
notes-86b7bffdbb-nh9qc 1/1 Running 0 3m40s
...
0.5.3
- The old pods keep running; the Kubernetes rolling update takes care of that.
- Flux did apply the new manifest.
REVISIONstill shows the old commit, because a revision only counts as “applied” once the health checks have passed. READY=Falseis the signal you will hang alerts on later (Day 11).
git revert --no-edit HEAD
git push
flux reconcile source git flux-system
flux reconcile kustomization notesExpected (shortened):
[main 39634fa] Revert "notes-api 0.9.99"
...
✔ applied revision main@sha1:39634fad5a6246cad3a6348f4a7d2439b4adba6c
Save your progress? Sign up for free
Drill 5: What dependsOn gives
you
Now the database breaks:
sed -i 's/postgres:18.6-alpine/postgres:99-alpine/' apps/notes-db/deployment.yaml
git commit -am "Postgres 99"
git push
flux reconcile source git flux-system
flux get kustomizations notes notes-dbExpected (after a few seconds):
NAME REVISION SUSPENDED READY MESSAGE
notes main@sha1:39634fad False False dependency 'flux-system/notes-db' revision is not up to date
notes-db main@sha1:39634fad False Unknown Reconciliation in progress
notes waits until notes-db has applied the
new revision healthily. An update of notes-api would not be rolled out
now. But dependsOn cannot protect the running database:
kubectl -n notes get podsExpected (shortened):
NAME READY STATUS RESTARTS AGE
notes-78cd6c995-khnw2 0/1 Running 0 105s
notes-78cd6c995-qfd2h 0/1 Running 0 101s
notes-db-784b5556cd-27kqw 0/1 ImagePullBackOff 0 76s
Recreate stopped the old Postgres before the new one was
running. Without a database, the readiness probes of notes-api fail, and
the API is unreachable. Back:
git revert --no-edit HEAD
git push
flux reconcile source git flux-system
flux reconcile kustomization notes-db
flux reconcile kustomization notes
curl -s localhost:30080/notesExpected (shortened):
...
[{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}]
The note is still there; it lives on the volume.
Save your progress? Sign up for free
Drill 6: Variables
The notes-api version appears twice in the Deployment, and each
cluster should be able to have its own number of replicas. For this,
Flux replaces variables of the form ${NAME} after the
Kustomize build:
sed -i 's/notes-api:0.5.3/notes-api:${NOTES_VERSION}/g' apps/notes/deployment.yaml
sed -i 's/replicas: 2/replicas: ${NOTES_REPLICAS:=1}/' apps/notes/deployment.yaml
grep -n -E 'image|replicas' apps/notes/deployment.yamlExpected:
7: replicas: ${NOTES_REPLICAS:=1}
18: image: ghcr.io/thomaszachmann/notes-api:${NOTES_VERSION}
30: image: ghcr.io/thomaszachmann/notes-api:${NOTES_VERSION}
${NOTES_REPLICAS:=1} has a default value,
${NOTES_VERSION} does not. The values come from two
sources: a ConfigMap that applies to the whole cluster, and directly
from the Kustomization.
cat > clusters/lab/cluster-vars.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-vars
namespace: flux-system
data:
CLUSTER_NAME: lab
NOTES_VERSION: 0.5.3
EOF
cat >> clusters/lab/notes.yaml <<'EOF'
postBuild:
substitute:
NOTES_REPLICAS: "2"
substituteFrom:
- kind: ConfigMap
name: cluster-vars
EOFBefore you push, you look at the result. flux build
builds the manifests locally, with the variables from the cluster:
flux build kustomization notes --path ./apps/notes | grep -E 'replicas|image:'Expected:
replicas: 2
image: ghcr.io/thomaszachmann/notes-api:0.5.3
image: ghcr.io/thomaszachmann/notes-api:0.5.3
git add -A
git commit -m "notes: Version und Replicas als Variablen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
kubectl -n notes get deploy notes \
-o jsonpath='{.spec.replicas} {.spec.template.spec.containers[0].image}{"\n"}'Expected (shortened):
✔ applied revision main@sha1:d8043e69aaae6c67bf7369fb241be0e27680f2f2
2 ghcr.io/thomaszachmann/notes-api:0.5.3
flux diff shows what a change would do, again without
pushing. As a test, set three replicas:
sed -i 's/NOTES_REPLICAS: "2"/NOTES_REPLICAS: "3"/' clusters/lab/notes.yaml
flux diff kustomization notes --path ./apps/notes \
--kustomization-file clusters/lab/notes.yamlExpected:
► Deployment/notes/notes drifted
metadata.generation
± value change
- 5
+ 6
spec.replicas
± value change
- 2
+ 3
⚠️ identified at least one change, exiting with non-zero exit code
git checkout clusters/lab/notes.yamlThat discards the change again. flux diff is the tool
for pull requests: it shows before the merge what would happen in the
cluster (Day 13).
Save your progress? Sign up for free
Drill 7: Suspending
During an incident, someone wants to intervene by hand without Flux rolling it back:
flux suspend kustomization notes
kubectl -n notes scale deploy notes --replicas=4
flux reconcile kustomization notes
flux get kustomization notesExpected:
► suspending kustomization notes in flux-system namespace
✔ kustomization suspended
deployment.apps/notes scaled
✗ resource is suspended
NAME REVISION SUSPENDED READY MESSAGE
notes main@sha1:d8043e69 True True Applied revision: main@sha1:d8043e69
The four replicas stay. After the incident:
flux resume kustomization notes
kubectl -n notes get deploy notesExpected (shortened):
✔ applied revision main@sha1:d8043e69aaae6c67bf7369fb241be0e27680f2f2
NAME READY UP-TO-DATE AVAILABLE AGE
notes 2/2 2 2 11m
resume reconciles immediately and restores the state
from Git.
Save your progress? Sign up for free
Checkpoint
flux get kustomizationsExpected: five Kustomizations, all READY
True.
flux get kustomization notesExpected: SUSPENDED False.
curl -s localhost:30080/notesExpected: your note “Hallo Flux”.
diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag04/appsExpected: no output.
Challenge
Someone makes a typo in the ConfigMap:
sed -i 's/ NOTES_VERSION:/ NOTES_VERSON:/' clusters/lab/cluster-vars.yaml
git commit -am "Tippfehler"
git push
flux reconcile kustomization flux-system --with-sourceWhat happens to notes-api? Does the image become
notes-api: without a version?
Solution
flux get kustomization notesExpected:
NAME REVISION SUSPENDED READY MESSAGE
notes main@sha1:d8043e69 False False post build failed for 'Deployment.v1.apps/notes': envsubst error: variable substitution failed: variable not set (strict mode): "NOTES_VERSION"
Flux replaces variables in strict mode: if a variable without a default value is missing, the build aborts and nothing is applied. notes-api keeps running unchanged.
git revert --no-edit HEAD
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notesExpected (shortened):
...
✔ applied revision main@sha1:c1eb001d3263eb43ffb4e4ed64d3aaefddced91e
Save your progress? Sign up for free
Enterprise Note
- Databases do not belong in a Deployment. In operations you use a Postgres operator such as CloudNativePG or a cloud database. A mistake like Drill 5 then does not end the only instance.
- Protect dangerous objects: namespaces, PVCs and
CRDs with data get
kustomize.toolkit.fluxcd.io/prune: disabled. A folder accidentally deleted in Git then does not delete the production data. - Use variables sparingly:
postBuildis handy for a few values per cluster. Differences between environments are better solved with Kustomize overlays (Day 5). There you see every file exactly as it is applied.