Skip to content
Hands-on IT IT Drills

Day 15: Competition

Goal: Find and fix five incidents, each of which really happened while testing this book or was reproduced. · Duration: approx. 120 min · Prerequisite: Days 0–14

Rules of Play

  • Each round starts with a Situation: what happened, what is the team reporting? The commands under “Situation” recreate the incident. Run them without reading the solution.
  • You find the cause with the tools from the book: argocd app get, argocd app list, kubectl get events, the controller logs, git log. Only then do you fix it.
  • Fixes happen in Git, never with kubectl edit. Self-heal would revert it anyway.
  • There is a hint for each round. Only read it if you have been stuck for ten minutes.
  • Count the rounds you complete without a hint.

Before you start:

cd ~/argo-lab/argocd-fleet
git pull
argocd app list | awk 'NR>1 {print $5, $6}' | sort | uniq -c

Expected: all eleven Applications Synced Healthy.

Round 1: The Vanished Environment

Situation: A colleague is tidying up and standardizing file extensions:

git mv apps/notes/overlays/dev/env.yaml apps/notes/overlays/dev/env.yml
git commit -m "dev: Endung vereinheitlicht"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20

The team reports: “notes-dev has disappeared from Argo CD. But the app is still running.”

argocd app list -o name | grep notes
kubectl -n notes-dev get deploy

Expected:

argocd/notes
argocd/notes-rollout
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
notes      1/1     1            1           23h
notes-db   1/1     1            1           23h

Time limit: 15 min

Hint: What does the ApplicationSet’s generator look for?

Solution

kubectl -n argocd get appset notes -o jsonpath='{.spec.generators[0].git.files[0].path}'
echo
ls apps/notes/overlays/*/env.*

Expected:

apps/notes/overlays/*/env.yaml
apps/notes/overlays/dev/env.yml
apps/notes/overlays/prod/env.yaml

The Git generator looks for env.yaml. For env.yml it generates nothing, so the ApplicationSet deleted notes-dev. That notes-api and the database are still running is thanks to preserveResourcesOnDeletion from Day 8. Without the option, the environment would be gone, volume and all.

git mv apps/notes/overlays/dev/env.yml apps/notes/overlays/dev/env.yaml
git commit -m "dev: env.yaml zurück"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20
argocd app get notes-dev | grep -E "Sync Status|Health Status"
kubectl -n notes-dev get deploy

Expected (abridged):

…
Sync Status:        Synced to main (6d8f945)
Health Status:      Healthy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
notes      1/1     1            1           23h
notes-db   1/1     1            1           24h

The new Application adopted the running objects. No pod was restarted.

Save your progress? Sign up for free

Round 2: The Environment That Won’t Start

Situation: The testers get their own environment qa on NodePort 30087:

mkdir -p apps/notes/overlays/qa
cat > apps/notes/overlays/qa/env.yaml <<'EOF'
name: notes-qa
namespace: qa
EOF
cat > apps/notes/overlays/qa/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - path: nodeport.yaml
EOF
cat > apps/notes/overlays/qa/nodeport.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30087
EOF
git add -A
git commit -m "Umgebung qa"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20

The testers report: “Port 30087 does not answer.”

argocd app get notes-qa | grep -E "Sync Status|Health Status"
curl -s -m 3 localhost:30087/version; echo "rc=$?"

Expected:

Sync Status:        Unknown
Health Status:      Unknown
rc=7

Time limit: 15 min

Hint: Unknown means Argo CD has not even started. Why?

Solution

argocd app get notes-qa | grep InvalidSpecError
argocd proj get notes | grep "^Destinations"

Expected:

InvalidSpecError  application destination server 'https://kubernetes.default.svc' and namespace 'qa' do not match any of the allowed destinations in project 'notes'  2026-11-15 08:53:05 +0000 UTC
Destinations:                https://kubernetes.default.svc,notes*

The project from Day 9 only allows namespaces that start with notes. The boundary works; it is just that the new environment does not respect it:

sed -i "s/^namespace: qa$/namespace: notes-qa/" apps/notes/overlays/qa/env.yaml
git commit -am "qa: Namespace notes-qa"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 15
argocd app wait notes-qa --health --operation --timeout 300 \
  | grep -E "Health Status|^Phase"
curl -s localhost:30087/version; echo

Expected (abridged):

…
Health Status:      Healthy
Phase:              Succeeded
0.5.3

0.5.3, because qa gets the version from base. The Image Updater only watches notes and notes-dev. The testers are happy, and the environment goes away again:

git rm -r apps/notes/overlays/qa
git commit -m "qa entfernt"
git push
kubectl -n argocd annotate appset notes \
  argocd.argoproj.io/application-set-refresh=true --overwrite
sleep 20
kubectl delete ns notes-qa

Expected (abridged):

…
namespace "notes-qa" deleted

Save your progress? Sign up for free

Round 3: The Little Label

Situation: A colleague wants to add a label to the database Secret. The file is readable after all, so he adds the lines directly:

sed -i "s/^    name: notes-db$/    name: notes-db\n    labels:\n        team: notes/" \
  apps/notes/base/db-secret.enc.yaml
head -8 apps/notes/base/db-secret.enc.yaml
git commit -am "notes-db: Label team"
git push
argocd app get notes --refresh
sleep 10

Expected (abridged):

apiVersion: v1
kind: Secret
metadata:
    name: notes-db
    labels:
        team: notes
    annotations:
        argocd.argoproj.io/sync-wave: "-1"
…

Monitoring reports: “notes no longer has a sync status.”

argocd app list | awk 'NR>1 && /notes/ {print $1, $5, $6}'
curl -s localhost:30080/version; echo

Expected (abridged):

argocd/notes Unknown Healthy
…
0.7.1

Time limit: 20 min

Hint: What does the Application’s condition say? And what did you read on Day 10 about the mac field?

Solution

kubectl -n argocd get app notes -o jsonpath='{.status.conditions[0].message}' \
  | grep -o "Failed to verify data integrity"
sops decrypt apps/notes/base/db-secret.enc.yaml 2>&1 | head -1

Expected (abridged):

Failed to verify data integrity
Failed to verify data integrity
MAC mismatch. File has 06702062AA62759C…, computed CE790C67338B2EC8…

The SOPS checksum also covers the unencrypted fields. Any manual change makes the file unusable. notes keeps running, because without a desired state Argo CD does not sync. The fix is git revert; after that, SOPS itself sets the label:

git revert --no-edit HEAD
sops set apps/notes/base/db-secret.enc.yaml '["metadata"]["labels"]' '{"team": "notes"}'
sops decrypt apps/notes/base/db-secret.enc.yaml | head -8
git commit -am "notes-db: Label team, mit sops"
git push
argocd app get notes --refresh
argocd app get notes-dev --refresh
sleep 15
argocd app list | awk 'NR>1 && /notes/ {print $1, $5, $6}'
kubectl -n notes get secret notes-db --show-labels

Expected (abridged):

[main a27ed25] Revert "notes-db: Label team"
…
apiVersion: v1
kind: Secret
metadata:
    name: notes-db
    annotations:
        argocd.argoproj.io/sync-wave: "-1"
    labels:
        team: notes
…
argocd/notes Synced Healthy
argocd/notes-dev Synced Healthy
argocd/notes-rollout Synced Healthy
NAME       TYPE     DATA   AGE   LABELS
notes-db   Opaque   1      24h   team=notes

Save your progress? Sign up for free

Round 4: The Silent Notification

Situation: The echo service is supposed to move to its own namespace notify at some point. A colleague changes the address in advance:

sed -i "s#url: http://echo.echo.svc#url: http://echo.notify.svc#" \
  platform/argocd/argocd-notifications-cm.yaml
git commit -am "Notifications: neuer Namespace"
git push
argocd app get argocd-extras --refresh
kubectl -n echo logs deploy/echo | grep -c '"body"'

Expected (abridged):

…
10

Your number may differ. Remember it. Then a normal deployment:

sed -i "s/replicas: 1/replicas: 2/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: zwei Repliken"
git push
argocd app get notes-dev --refresh
sleep 30
kubectl -n notes-dev get deploy notes
kubectl -n echo logs deploy/echo | grep -c '"body"'

Expected (abridged):

…
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   2/2     2            2           24h
10

The team reports: “The deployment went through, but no message arrived.”

Time limit: 15 min

Hint: Who sends the messages, and what does it write to its log?

Solution

kubectl -n argocd logs deploy/argocd-notifications-controller --since=5m \
  | grep "request failed" | tail -1

Expected:

2026/11/15 08:57:45 [ERR] POST http://echo.notify.svc request failed: Post "http://echo.notify.svc": dial tcp: lookup echo.notify.svc on 10.43.0.10:53: no such host

The notify namespace does not exist. You revert exactly the commit that changed the file:

C=$(git log -1 --format=%h -- platform/argocd/argocd-notifications-cm.yaml)
echo $C
git revert --no-edit $C
git push
argocd app get argocd-extras --refresh
sleep 30
kubectl -n echo logs deploy/echo | grep -c '"body"'

Expected (abridged):

2dfe98f
[main 643ea8a] Revert "Notifications: neuer Namespace"
…
10

Still no new message. The lost one is not delivered later:

git log -1 --format=%h -- apps/notes/overlays/dev/replicas.yaml
kubectl -n argocd get app notes-dev \
  -o jsonpath='{.metadata.annotations.notified\.notifications\.argoproj\.io}' \
  | tr "," "\n" | grep baf262e | cut -c1-60

Expected:

baf262e
"baf262e445f4512bc9781c43f922a80e7e2449a7:on-deployed:[0].y7

Tried it: Argo CD recorded the message for commit baf262e as sent, even though delivery failed. Because of oncePer, it does not try a second time. Delivery errors only show up in the controller’s log.

The next deployment gets through again:

sed -i "s/replicas: 2/replicas: 1/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: wieder eine Replik"
git push
argocd app get notes-dev --refresh
sleep 30
kubectl -n echo logs deploy/echo | grep -c '"body"'

Expected (abridged):

…
11

Save your progress? Sign up for free

Round 5: The Preview That Never Finishes

Situation: A colleague wants to test the patch 0.7.10 as blue-green:

sed -i "s/notes-api:0.7.0/notes-api:0.7.10/" apps/notes-rollout/rollout.yaml
git commit -am "notes-rollout: 0.7.10"
git push
argocd app get notes-rollout --refresh
sleep 60

He reports: “The preview is still not there after a minute. Argo CD only says Progressing.”

argocd app get notes-rollout | grep -E "Sync Status|Health Status"
curl -s localhost:30085/version

Expected:

Sync Status:        Synced to main (0792426)
Health Status:      Progressing
0.7.0

Time limit: 15 min

Hint: Look at the preview pods.

Solution

kubectl argo rollouts get rollout notes -n notes-rollout \
  | grep -E "Message|preview|ErrImagePull"

Expected (abridged):

Message:         active service cutover pending
                 ghcr.io/thomaszachmann/notes-api:0.7.10 (preview)
│  └──⧉ notes-6b8967d675                                         ReplicaSet   ◌ Progressing   59s  preview
│     ├──□ notes-6b8967d675-jp7cd                                Pod          ⚠ ErrImagePull  59s  ready:0/1
│     ├──□ notes-6b8967d675-lp7mx                                Pod          ⚠ ErrImagePull  59s  ready:0/1
…
kubectl -n notes-rollout get events --field-selector reason=Failed \
  | grep "0.7.10" | head -1

Expected (abridged):

… Warning   Failed   pod/notes-6b8967d675-jp7cd   Failed to pull image "ghcr.io/thomaszachmann/notes-api:0.7.10": rpc error: code = NotFound …

There is no tag 0.7.10, as the list from Day 12 shows. Traffic on 30085 noticed nothing, because blue-green only switches over once the preview is ready. Back to the previous state:

git revert --no-edit HEAD
git push
argocd app get notes-rollout --refresh
sleep 60
argocd app get notes-rollout | grep -E "Sync Status|Health Status"
kubectl -n notes-rollout get rs \
  -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,\
IMAGE:.spec.template.spec.containers[0].image | grep -E "NAME|0.7.10|0.7.0"

Expected (abridged):

[main cadbc89] Revert "notes-rollout: 0.7.10"
…
Sync Status:        Synced to main (cadbc89)
Health Status:      Healthy
NAME               DESIRED   IMAGE
notes-5ff689668b   4         ghcr.io/thomaszachmann/notes-api:0.7.0
notes-6b8967d675   0         ghcr.io/thomaszachmann/notes-api:0.7.10

Save your progress? Sign up for free

Scoring

Rounds without a hint Assessment
5 You run Argo CD, not the other way around.
3–4 Solid. Reread the days covering the rounds you missed.
0–2 Work through Days 8 to 13 a second time, this time without the solutions.

What all rounds have in common: every fix was a commit. Afterwards, Git shows who changed what and when, and who reverted it.