Day 15: Competition
Goal: Find and fix five incidents, each of which really happened while testing this book or was reproduced. · Duration: approx. 120 min · Prerequisite: Days 0–14
Rules of Play
- Each round starts with a Situation: what happened, what is the team reporting? The commands under “Situation” recreate the incident. Run them without reading the solution.
- You find the cause with the tools from the book:
argocd app get,argocd app list,kubectl get events, the controller logs,git log. Only then do you fix it. - Fixes happen in Git, never with
kubectl edit. Self-heal would revert it anyway. - There is a hint for each round. Only read it if you have been stuck for ten minutes.
- Count the rounds you complete without a hint.
Before you start:
cd ~/argo-lab/argocd-fleet
git pull
argocd app list | awk 'NR>1 {print $5, $6}' | sort | uniq -cExpected: all eleven Applications Synced Healthy.
Round 1: The Vanished Environment
Situation: A colleague is tidying up and standardizing file extensions:
git mv apps/notes/overlays/dev/env.yaml apps/notes/overlays/dev/env.yml
git commit -m "dev: Endung vereinheitlicht"
git push
kubectl -n argocd annotate appset notes \
argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20The team reports: “notes-dev has disappeared from Argo CD. But the app is still running.”
argocd app list -o name | grep notes
kubectl -n notes-dev get deployExpected:
argocd/notes
argocd/notes-rollout
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 23h
notes-db 1/1 1 1 23h
Time limit: 15 min
Hint: What does the ApplicationSet’s generator look for?
Solution
kubectl -n argocd get appset notes -o jsonpath='{.spec.generators[0].git.files[0].path}'
echo
ls apps/notes/overlays/*/env.*Expected:
apps/notes/overlays/*/env.yaml
apps/notes/overlays/dev/env.yml
apps/notes/overlays/prod/env.yaml
The Git generator looks for env.yaml. For
env.yml it generates nothing, so the ApplicationSet deleted
notes-dev. That notes-api and the database are still
running is thanks to preserveResourcesOnDeletion from Day
8. Without the option, the environment would be gone, volume and
all.
git mv apps/notes/overlays/dev/env.yml apps/notes/overlays/dev/env.yaml
git commit -m "dev: env.yaml zurück"
git push
kubectl -n argocd annotate appset notes \
argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20
argocd app get notes-dev | grep -E "Sync Status|Health Status"
kubectl -n notes-dev get deployExpected (abridged):
…
Sync Status: Synced to main (6d8f945)
Health Status: Healthy
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 23h
notes-db 1/1 1 1 24h
The new Application adopted the running objects. No pod was restarted.
Save your progress? Sign up for free
Round 2: The Environment That Won’t Start
Situation: The testers get their own environment
qa on NodePort 30087:
mkdir -p apps/notes/overlays/qa
cat > apps/notes/overlays/qa/env.yaml <<'EOF'
name: notes-qa
namespace: qa
EOF
cat > apps/notes/overlays/qa/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: nodeport.yaml
EOF
cat > apps/notes/overlays/qa/nodeport.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
ports:
- port: 80
targetPort: 8080
nodePort: 30087
EOF
git add -A
git commit -m "Umgebung qa"
git push
kubectl -n argocd annotate appset notes \
argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20The testers report: “Port 30087 does not answer.”
argocd app get notes-qa | grep -E "Sync Status|Health Status"
curl -s -m 3 localhost:30087/version; echo "rc=$?"Expected:
Sync Status: Unknown
Health Status: Unknown
rc=7
Time limit: 15 min
Hint: Unknown means Argo CD has not
even started. Why?
Solution
argocd app get notes-qa | grep InvalidSpecError
argocd proj get notes | grep "^Destinations"Expected:
InvalidSpecError application destination server 'https://kubernetes.default.svc' and namespace 'qa' do not match any of the allowed destinations in project 'notes' 2026-11-15 08:53:05 +0000 UTC
Destinations: https://kubernetes.default.svc,notes*
The project from Day 9 only allows namespaces that start with
notes. The boundary works; it is just that the new
environment does not respect it:
sed -i "s/^namespace: qa$/namespace: notes-qa/" apps/notes/overlays/qa/env.yaml
git commit -am "qa: Namespace notes-qa"
git push
kubectl -n argocd annotate appset notes \
argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 15
argocd app wait notes-qa --health --operation --timeout 300 \
| grep -E "Health Status|^Phase"
curl -s localhost:30087/version; echoExpected (abridged):
…
Health Status: Healthy
Phase: Succeeded
0.5.3
0.5.3, because qa gets the version from
base. The Image Updater only watches notes and
notes-dev. The testers are happy, and the environment goes
away again:
git rm -r apps/notes/overlays/qa
git commit -m "qa entfernt"
git push
kubectl -n argocd annotate appset notes \
argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20
kubectl delete ns notes-qaExpected (abridged):
…
namespace "notes-qa" deleted
Save your progress? Sign up for free
Round 3: The Little Label
Situation: A colleague wants to add a label to the database Secret. The file is readable after all, so he adds the lines directly:
sed -i "s/^ name: notes-db$/ name: notes-db\n labels:\n team: notes/" \
apps/notes/base/db-secret.enc.yaml
head -8 apps/notes/base/db-secret.enc.yaml
git commit -am "notes-db: Label team"
git push
argocd app get notes --refresh
sleep 10Expected (abridged):
apiVersion: v1
kind: Secret
metadata:
name: notes-db
labels:
team: notes
annotations:
argocd.argoproj.io/sync-wave: "-1"
…
Monitoring reports: “notes no longer has a sync status.”
argocd app list | awk 'NR>1 && /notes/ {print $1, $5, $6}'
curl -s localhost:30080/version; echoExpected (abridged):
argocd/notes Unknown Healthy
…
0.7.1
Time limit: 20 min
Hint: What does the Application’s condition say? And
what did you read on Day 10 about the mac field?
Solution
kubectl -n argocd get app notes -o jsonpath='{.status.conditions[0].message}' \
| grep -o "Failed to verify data integrity"
sops decrypt apps/notes/base/db-secret.enc.yaml 2>&1 | head -1Expected (abridged):
Failed to verify data integrity
Failed to verify data integrity
MAC mismatch. File has 06702062AA62759C…, computed CE790C67338B2EC8…
The SOPS checksum also covers the unencrypted fields. Any manual
change makes the file unusable. notes keeps running, because without a
desired state Argo CD does not sync. The fix is git revert;
after that, SOPS itself sets the label:
git revert --no-edit HEAD
sops set apps/notes/base/db-secret.enc.yaml '["metadata"]["labels"]' '{"team": "notes"}'
sops decrypt apps/notes/base/db-secret.enc.yaml | head -8
git commit -am "notes-db: Label team, mit sops"
git push
argocd app get notes --refresh
argocd app get notes-dev --refresh
sleep 15
argocd app list | awk 'NR>1 && /notes/ {print $1, $5, $6}'
kubectl -n notes get secret notes-db --show-labelsExpected (abridged):
[main a27ed25] Revert "notes-db: Label team"
…
apiVersion: v1
kind: Secret
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
labels:
team: notes
…
argocd/notes Synced Healthy
argocd/notes-dev Synced Healthy
argocd/notes-rollout Synced Healthy
NAME TYPE DATA AGE LABELS
notes-db Opaque 1 24h team=notes
Save your progress? Sign up for free
Round 4: The Silent Notification
Situation: The echo service is supposed to move to
its own namespace notify at some point. A colleague changes
the address in advance:
sed -i "s#url: http://echo.echo.svc#url: http://echo.notify.svc#" \
platform/argocd/argocd-notifications-cm.yaml
git commit -am "Notifications: neuer Namespace"
git push
argocd app get argocd-extras --refresh
kubectl -n echo logs deploy/echo | grep -c '"body"'Expected (abridged):
…
10
Your number may differ. Remember it. Then a normal deployment:
sed -i "s/replicas: 1/replicas: 2/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: zwei Repliken"
git push
argocd app get notes-dev --refresh
sleep 30
kubectl -n notes-dev get deploy notes
kubectl -n echo logs deploy/echo | grep -c '"body"'Expected (abridged):
…
NAME READY UP-TO-DATE AVAILABLE AGE
notes 2/2 2 2 24h
10
The team reports: “The deployment went through, but no message arrived.”
Time limit: 15 min
Hint: Who sends the messages, and what does it write to its log?
Solution
kubectl -n argocd logs deploy/argocd-notifications-controller --since=5m \
| grep "request failed" | tail -1Expected:
2026/11/15 08:57:45 [ERR] POST http://echo.notify.svc request failed: Post "http://echo.notify.svc": dial tcp: lookup echo.notify.svc on 10.43.0.10:53: no such host
The notify namespace does not exist. You revert exactly
the commit that changed the file:
C=$(git log -1 --format=%h -- platform/argocd/argocd-notifications-cm.yaml)
echo $C
git revert --no-edit $C
git push
argocd app get argocd-extras --refresh
sleep 30
kubectl -n echo logs deploy/echo | grep -c '"body"'Expected (abridged):
2dfe98f
[main 643ea8a] Revert "Notifications: neuer Namespace"
…
10
Still no new message. The lost one is not delivered later:
git log -1 --format=%h -- apps/notes/overlays/dev/replicas.yaml
kubectl -n argocd get app notes-dev \
-o jsonpath='{.metadata.annotations.notified\.notifications\.argoproj\.io}' \
| tr "," "\n" | grep baf262e | cut -c1-60Expected:
baf262e
"baf262e445f4512bc9781c43f922a80e7e2449a7:on-deployed:[0].y7
Tried it: Argo CD recorded the message for commit
baf262eas sent, even though delivery failed. Because ofoncePer, it does not try a second time. Delivery errors only show up in the controller’s log.
The next deployment gets through again:
sed -i "s/replicas: 2/replicas: 1/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: wieder eine Replik"
git push
argocd app get notes-dev --refresh
sleep 30
kubectl -n echo logs deploy/echo | grep -c '"body"'Expected (abridged):
…
11
Save your progress? Sign up for free
Round 5: The Preview That Never Finishes
Situation: A colleague wants to test the patch
0.7.10 as blue-green:
sed -i "s/notes-api:0.7.0/notes-api:0.7.10/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: 0.7.10"
git push
argocd app get notes-rollout --refresh
sleep 60He reports: “The preview is still not there after a minute. Argo CD
only says Progressing.”
argocd app get notes-rollout | grep -E "Sync Status|Health Status"
curl -s localhost:30085/versionExpected:
Sync Status: Synced to main (0792426)
Health Status: Progressing
0.7.0
Time limit: 15 min
Hint: Look at the preview pods.
Solution
kubectl argo rollouts get rollout notes -n notes-rollout \
| grep -E "Message|preview|ErrImagePull"Expected (abridged):
Message: active service cutover pending
ghcr.io/thomaszachmann/notes-api:0.7.10 (preview)
│ └──⧉ notes-6b8967d675 ReplicaSet ◌ Progressing 59s preview
│ ├──□ notes-6b8967d675-jp7cd Pod ⚠ ErrImagePull 59s ready:0/1
│ ├──□ notes-6b8967d675-lp7mx Pod ⚠ ErrImagePull 59s ready:0/1
…
kubectl -n notes-rollout get events --field-selector reason=Failed \
| grep "0.7.10" | head -1Expected (abridged):
… Warning Failed pod/notes-6b8967d675-jp7cd Failed to pull image "ghcr.io/thomaszachmann/notes-api:0.7.10": rpc error: code = NotFound …
There is no tag 0.7.10, as the list from Day 12 shows.
Traffic on 30085 noticed nothing, because blue-green only
switches over once the preview is ready. Back to the previous state:
git revert --no-edit HEAD
git push
argocd app get notes-rollout --refresh
sleep 60
argocd app get notes-rollout | grep -E "Sync Status|Health Status"
kubectl -n notes-rollout get rs \
-o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,\
IMAGE:.spec.template.spec.containers[0].image | grep -E "NAME|0.7.10|0.7.0"Expected (abridged):
[main cadbc89] Revert "notes-rollout: 0.7.10"
…
Sync Status: Synced to main (cadbc89)
Health Status: Healthy
NAME DESIRED IMAGE
notes-5ff689668b 4 ghcr.io/thomaszachmann/notes-api:0.7.0
notes-6b8967d675 0 ghcr.io/thomaszachmann/notes-api:0.7.10
Save your progress? Sign up for free
Scoring
| Rounds without a hint | Assessment |
|---|---|
| 5 | You run Argo CD, not the other way around. |
| 3–4 | Solid. Reread the days covering the rounds you missed. |
| 0–2 | Work through Days 8 to 13 a second time, this time without the solutions. |
What all rounds have in common: every fix was a commit. Afterwards, Git shows who changed what and when, and who reverted it.