Day 15: Competition
Goal: Find and fix five incidents, each of which actually happened while testing this book. · Duration: approx. 120 min · Prerequisite: Days 0–14
Rules of Play
- Every round starts with a situation: what happened, what is the team reporting?
- You find the cause with the tools from the book:
flux get,flux events,flux tree,flux logs,kubectl. Only then do you repair, as always through Git. - The files for the rounds are in
~/flux-lab/code/wettkampf/. - There is a hint for every round. Read it only if you have been stuck for ten minutes.
- Count the rounds you complete without a hint. Five out of five is a win.
cd ~/flux-lab/fleet
git pullRound 1: The maintenance notice
Situation: The team wants a ConfigMap
motd with a maintenance notice in notes. You put the file
into the cluster folder and push:
cp ~/flux-lab/code/wettkampf/runde1/motd.yaml clusters/lab/
git add -A
git commit -m "Runde 1: motd"
git push
flux reconcile kustomization flux-system --with-source
kubectl -n notes get configmap motdExpected (shortened):
✔ applied revision main@sha1:e180e60b8b578a979e3077575e1ec71fbe58b694
Error from server (NotFound): configmaps "motd" not found
flux-system reports applied, no error
message anywhere. Where is the ConfigMap?
Hint: Ask Flux which objects
flux-system manages. And remember what has been different
in clusters/lab since Day 12.
Solution
flux tree kustomization flux-system | grep -c motdExpected:
0
Flux does not know the file. Since Day 12, clusters/lab
has had a kustomization.yaml, and Flux reads only the files
listed under resources:
sed -i 's/ - kro-apis.yaml/ - kro-apis.yaml\n - motd.yaml/' clusters/lab/kustomization.yaml
git commit -am "Runde 1: motd eintragen"
git push
flux reconcile kustomization flux-system --with-source
kubectl -n notes get configmap motd -o jsonpath='{.data.text}{"\n"}'Expected (shortened):
Wartung am Samstag ab 8 Uhr
Prevention: The CI workflow from Day 13 does not
catch this, because a file that is not listed is valid YAML. A step that
compares every file in clusters/lab with the entries under
resources helps.
Save your progress? Sign up for free
Round 2: The SMTP password
Situation: notes gets an SMTP password, encrypted of
course. For this, the platform team creates a new Kustomization
geheim:
export SOPS_AGE_KEY_FILE=~/flux-lab/age-2027.agekey
mkdir -p infrastructure/geheim
cp ~/flux-lab/code/wettkampf/runde2/secret.yaml infrastructure/geheim/
sops --encrypt --in-place infrastructure/geheim/secret.yaml
cp ~/flux-lab/code/wettkampf/runde2/geheim.yaml clusters/lab/
sed -i 's/ - echo.yaml/ - echo.yaml\n - geheim.yaml/' clusters/lab/kustomization.yaml
git add -A
git commit -m "Runde 2: SMTP-Passwort"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization geheim
flux get kustomization geheimExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
geheim main@sha1:171da6cf False True Applied revision: main@sha1:171da6cf
All green. The next day the mail server reports: wrong password.
Hint: Look at what the Secret contains in the cluster, not in Git.
Solution
kubectl -n notes get secret smtp -o jsonpath='{.data.password}' | base64 -d | cut -c1-30; echoExpected:
ENC[AES256_GCM,data:v68fWEKTKQ
The case from Day 8: the new Kustomization lacks
decryption, so Flux applied the encrypted text as the
value. No error, no hint.
sed -i 's/^ name: flux-system$/ name: flux-system\n decryption:\n provider: sops\n secretRef:\n name: sops-age/' \
clusters/lab/geheim.yaml
git commit -am "Runde 2: entschlüsseln"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization geheim
kubectl -n notes get secret smtp -o jsonpath='{.data.password}' | base64 -d; echoExpected (shortened):
smtp-pass-123
Prevention: A patch like on Day 12 that adds
decryption to every Kustomization in
flux-system. Then nobody can forget it anymore.
Save your progress? Sign up for free
Round 3: Team grün is stuck
Situation: A colleague “cleaned up” the synchronization of Team grün and rewrote it:
cp ~/flux-lab/code/wettkampf/runde3/sync.yaml tenants/team-gruen/sync.yaml
git commit -am "Runde 3: team-gruen neu"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenantsTeam grün reports: new versions no longer arrive.
Hint:
flux get kustomizations -n team-gruen. And which user
appears in the message?
Solution
flux get kustomizations -n team-gruenExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
apps 1.0.0@sha256:57c8e47c False False Deployment/team-gruen/notes dry-run failed (Forbidden): deployments.apps "notes" is forbidden: User "system:serviceaccount:team-gruen:default" cannot patch resource "deployments" …
team-gruen:default instead of
team-gruen:team-gruen. The new file lacks
serviceAccountName, and the lockdown from Day 12 then uses
the powerless ServiceAccount default. The good news:
without the lockdown, the Kustomization would have worked with the
controller’s permissions, and nobody would have noticed.
git checkout HEAD~1 -- tenants/team-gruen/sync.yaml
git commit -am "Runde 3: ServiceAccount zurück"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux reconcile kustomization apps -n team-gruen
flux get kustomizations -n team-gruenExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
apps 1.0.0@sha256:57c8e47c False True Applied revision: 1.0.0@sha256:57c8e47c
Save your progress? Sign up for free
Round 4: The silent bot
Situation: dev should run the newest release
candidate. Someone set dev to 0.6.0-rc.2 by hand, and an
old automation branch is still lying around somewhere. Recreate the
situation:
git push origin HEAD~3:refs/heads/flux/notes-updates
sed -i 's/NOTES_VERSION: 0.6.0-rc.4/NOTES_VERSION: 0.6.0-rc.2/' clusters/lab/notes-dev.yaml
git commit -am "Runde 4: dev auf rc.2"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile image update notesExpected (shortened):
...
✔ repository up-to-date
The policy notes-api-rc is at 0.6.0-rc.4,
Git says rc.2, and the automation thinks everything is up
to date.
Hint: Which branch does the automation work on when its push branch already exists?
Solution
git fetch
git log --oneline -1 origin/flux/notes-updatesExpected (shortened):
171da6c Runde 2: SMTP-Passwort
The automation builds on the existing branch
flux/notes-updates. It is old; there, dev is still at
rc.4, so there is nothing to do. The case from Day 10:
git push origin --delete flux/notes-updates
flux reconcile image update notes
git fetch -p
git log --oneline -1 origin/flux/notes-updatesExpected (shortened):
e7d1a5a 0.6.0-rc.2 -> 0.6.0-rc.4
git merge --no-edit origin/flux/notes-updates
git push
git push origin --delete flux/notes-updates
flux reconcile kustomization flux-system --with-sourcePrevention: Enable Automatically delete head branches on GitHub.
Save your progress? Sign up for free
Round 5: The deaf Receiver
Situation: A colleague changed the webhook secret for security reasons:
cd ~/flux-lab
head -c 24 /dev/urandom | base64 > webhook.token
kubectl -n flux-system create secret generic webhook-token \
--from-file=token=webhook.token --dry-run=client -o yaml | kubectl replace -f -Since then, every webhook from GitHub ends in 400. Play
GitHub as on Day 11:
P=$(kubectl -n flux-system get receiver github-fleet -o jsonpath='{.status.webhookPath}')
BODY='{"ref":"refs/heads/main"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$(cat webhook.token)" | cut -d' ' -f2)
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
-H "X-Hub-Signature-256: sha256=$SIG" -d "$BODY" localhost:30900$PExpected:
400
Hint: The notification-controller writes the reason
to its log. And compare what is in the file with what
$(cat …) makes of it.
Solution
kubectl -n flux-system logs deploy/notification-controller --since=1m \
| grep -o '"error":"[^"]*"' | tail -1Expected:
"error":"the GitHub signature header is invalid, err: payload signature check failed"
base64 appends a newline. It ends up in the Secret, but
not in $(cat webhook.token), because the shell strips it.
Two different secrets, so no signature matches:
tr -d '\n' < webhook.token > webhook.tmp && mv webhook.tmp webhook.token
kubectl -n flux-system create secret generic webhook-token \
--from-file=token=webhook.token --dry-run=client -o yaml | kubectl replace -f -
P=$(kubectl -n flux-system get receiver github-fleet -o jsonpath='{.status.webhookPath}')
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$(cat webhook.token)" | cut -d' ' -f2)
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
-H "X-Hub-Signature-256: sha256=$SIG" -d "$BODY" localhost:30900$PExpected (shortened):
secret/webhook-token replaced
200
You have to fetch P again: the Receiver’s path is a hash
that the secret goes into. A new secret also means a new URL, and it has
to be updated in the webhook on GitHub.
Save your progress? Sign up for free
Scoring
| Rounds without a hint | Verdict |
|---|---|
| 5 | You can run Flux. |
| 3–4 | Solid. Reread the days for the rounds you missed. |
| 0–2 | Redo Days 4, 8, 10, 11 and 12; together they cover all five failure patterns. |
All five failures have one thing in common: Flux reported either nothing at all or only in a place where nobody looks on their own. That is the real lesson of this day. With GitOps, things rarely break loudly. More often, quietly, what you expect simply does not happen.