Skip to content
Hands-on IT IT Drills

Day 15: Competition

Goal: Find and fix five incidents, each of which actually happened while testing this book. · Duration: approx. 120 min · Prerequisite: Days 0–14

Rules of Play

  • Every round starts with a situation: what happened, what is the team reporting?
  • You find the cause with the tools from the book: flux get, flux events, flux tree, flux logs, kubectl. Only then do you repair, as always through Git.
  • The files for the rounds are in ~/flux-lab/code/wettkampf/.
  • There is a hint for every round. Read it only if you have been stuck for ten minutes.
  • Count the rounds you complete without a hint. Five out of five is a win.
cd ~/flux-lab/fleet
git pull

Round 1: The maintenance notice

Situation: The team wants a ConfigMap motd with a maintenance notice in notes. You put the file into the cluster folder and push:

cp ~/flux-lab/code/wettkampf/runde1/motd.yaml clusters/lab/
git add -A
git commit -m "Runde 1: motd"
git push
flux reconcile kustomization flux-system --with-source
kubectl -n notes get configmap motd

Expected (shortened):

✔ applied revision main@sha1:e180e60b8b578a979e3077575e1ec71fbe58b694
Error from server (NotFound): configmaps "motd" not found

flux-system reports applied, no error message anywhere. Where is the ConfigMap?

Hint: Ask Flux which objects flux-system manages. And remember what has been different in clusters/lab since Day 12.

Solution

flux tree kustomization flux-system | grep -c motd

Expected:

0

Flux does not know the file. Since Day 12, clusters/lab has had a kustomization.yaml, and Flux reads only the files listed under resources:

sed -i 's/  - kro-apis.yaml/  - kro-apis.yaml\n  - motd.yaml/' clusters/lab/kustomization.yaml
git commit -am "Runde 1: motd eintragen"
git push
flux reconcile kustomization flux-system --with-source
kubectl -n notes get configmap motd -o jsonpath='{.data.text}{"\n"}'

Expected (shortened):

Wartung am Samstag ab 8 Uhr

Prevention: The CI workflow from Day 13 does not catch this, because a file that is not listed is valid YAML. A step that compares every file in clusters/lab with the entries under resources helps.

Save your progress? Sign up for free

Round 2: The SMTP password

Situation: notes gets an SMTP password, encrypted of course. For this, the platform team creates a new Kustomization geheim:

export SOPS_AGE_KEY_FILE=~/flux-lab/age-2027.agekey
mkdir -p infrastructure/geheim
cp ~/flux-lab/code/wettkampf/runde2/secret.yaml infrastructure/geheim/
sops --encrypt --in-place infrastructure/geheim/secret.yaml
cp ~/flux-lab/code/wettkampf/runde2/geheim.yaml clusters/lab/
sed -i 's/  - echo.yaml/  - echo.yaml\n  - geheim.yaml/' clusters/lab/kustomization.yaml
git add -A
git commit -m "Runde 2: SMTP-Passwort"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization geheim
flux get kustomization geheim

Expected (shortened):

NAME    REVISION            SUSPENDED   READY   MESSAGE
geheim  main@sha1:171da6cf  False       True    Applied revision: main@sha1:171da6cf

All green. The next day the mail server reports: wrong password.

Hint: Look at what the Secret contains in the cluster, not in Git.

Solution

kubectl -n notes get secret smtp -o jsonpath='{.data.password}' | base64 -d | cut -c1-30; echo

Expected:

ENC[AES256_GCM,data:v68fWEKTKQ

The case from Day 8: the new Kustomization lacks decryption, so Flux applied the encrypted text as the value. No error, no hint.

sed -i 's/^    name: flux-system$/    name: flux-system\n  decryption:\n    provider: sops\n    secretRef:\n      name: sops-age/' \
  clusters/lab/geheim.yaml
git commit -am "Runde 2: entschlüsseln"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization geheim
kubectl -n notes get secret smtp -o jsonpath='{.data.password}' | base64 -d; echo

Expected (shortened):

smtp-pass-123

Prevention: A patch like on Day 12 that adds decryption to every Kustomization in flux-system. Then nobody can forget it anymore.

Save your progress? Sign up for free

Round 3: Team grün is stuck

Situation: A colleague “cleaned up” the synchronization of Team grün and rewrote it:

cp ~/flux-lab/code/wettkampf/runde3/sync.yaml tenants/team-gruen/sync.yaml
git commit -am "Runde 3: team-gruen neu"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants

Team grün reports: new versions no longer arrive.

Hint: flux get kustomizations -n team-gruen. And which user appears in the message?

Solution

flux get kustomizations -n team-gruen

Expected (shortened):

NAME    REVISION                SUSPENDED   READY   MESSAGE
apps    1.0.0@sha256:57c8e47c   False       False   Deployment/team-gruen/notes dry-run failed (Forbidden): deployments.apps "notes" is forbidden: User "system:serviceaccount:team-gruen:default" cannot patch resource "deployments" …

team-gruen:default instead of team-gruen:team-gruen. The new file lacks serviceAccountName, and the lockdown from Day 12 then uses the powerless ServiceAccount default. The good news: without the lockdown, the Kustomization would have worked with the controller’s permissions, and nobody would have noticed.

git checkout HEAD~1 -- tenants/team-gruen/sync.yaml
git commit -am "Runde 3: ServiceAccount zurück"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux reconcile kustomization apps -n team-gruen
flux get kustomizations -n team-gruen

Expected (shortened):

NAME    REVISION                SUSPENDED   READY   MESSAGE
apps    1.0.0@sha256:57c8e47c   False       True    Applied revision: 1.0.0@sha256:57c8e47c

Save your progress? Sign up for free

Round 4: The silent bot

Situation: dev should run the newest release candidate. Someone set dev to 0.6.0-rc.2 by hand, and an old automation branch is still lying around somewhere. Recreate the situation:

git push origin HEAD~3:refs/heads/flux/notes-updates
sed -i 's/NOTES_VERSION: 0.6.0-rc.4/NOTES_VERSION: 0.6.0-rc.2/' clusters/lab/notes-dev.yaml
git commit -am "Runde 4: dev auf rc.2"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile image update notes

Expected (shortened):

...
✔ repository up-to-date

The policy notes-api-rc is at 0.6.0-rc.4, Git says rc.2, and the automation thinks everything is up to date.

Hint: Which branch does the automation work on when its push branch already exists?

Solution

git fetch
git log --oneline -1 origin/flux/notes-updates

Expected (shortened):

171da6c Runde 2: SMTP-Passwort

The automation builds on the existing branch flux/notes-updates. It is old; there, dev is still at rc.4, so there is nothing to do. The case from Day 10:

git push origin --delete flux/notes-updates
flux reconcile image update notes
git fetch -p
git log --oneline -1 origin/flux/notes-updates

Expected (shortened):

e7d1a5a 0.6.0-rc.2 -> 0.6.0-rc.4
git merge --no-edit origin/flux/notes-updates
git push
git push origin --delete flux/notes-updates
flux reconcile kustomization flux-system --with-source

Prevention: Enable Automatically delete head branches on GitHub.

Save your progress? Sign up for free

Round 5: The deaf Receiver

Situation: A colleague changed the webhook secret for security reasons:

cd ~/flux-lab
head -c 24 /dev/urandom | base64 > webhook.token
kubectl -n flux-system create secret generic webhook-token \
  --from-file=token=webhook.token --dry-run=client -o yaml | kubectl replace -f -

Since then, every webhook from GitHub ends in 400. Play GitHub as on Day 11:

P=$(kubectl -n flux-system get receiver github-fleet -o jsonpath='{.status.webhookPath}')
BODY='{"ref":"refs/heads/main"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$(cat webhook.token)" | cut -d' ' -f2)
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
  -H "X-Hub-Signature-256: sha256=$SIG" -d "$BODY" localhost:30900$P

Expected:

400

Hint: The notification-controller writes the reason to its log. And compare what is in the file with what $(cat …) makes of it.

Solution

kubectl -n flux-system logs deploy/notification-controller --since=1m \
  | grep -o '"error":"[^"]*"' | tail -1

Expected:

"error":"the GitHub signature header is invalid, err: payload signature check failed"

base64 appends a newline. It ends up in the Secret, but not in $(cat webhook.token), because the shell strips it. Two different secrets, so no signature matches:

tr -d '\n' < webhook.token > webhook.tmp && mv webhook.tmp webhook.token
kubectl -n flux-system create secret generic webhook-token \
  --from-file=token=webhook.token --dry-run=client -o yaml | kubectl replace -f -
P=$(kubectl -n flux-system get receiver github-fleet -o jsonpath='{.status.webhookPath}')
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$(cat webhook.token)" | cut -d' ' -f2)
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
  -H "X-Hub-Signature-256: sha256=$SIG" -d "$BODY" localhost:30900$P

Expected (shortened):

secret/webhook-token replaced
200

You have to fetch P again: the Receiver’s path is a hash that the secret goes into. A new secret also means a new URL, and it has to be updated in the webhook on GitHub.

Save your progress? Sign up for free

Scoring

Rounds without a hint Verdict
5 You can run Flux.
3–4 Solid. Reread the days for the rounds you missed.
0–2 Redo Days 4, 8, 10, 11 and 12; together they cover all five failure patterns.

All five failures have one thing in common: Flux reported either nothing at all or only in a place where nobody looks on their own. That is the real lesson of this day. With GitOps, things rarely break loudly. More often, quietly, what you expect simply does not happen.