Skip to content
Hands-on IT IT Drills

Day 4: Postgres, Sync Waves, and Hooks

Goal: notes-api stores its data in Postgres. Argo CD first creates the database, then a Job migrates the schema, then notes-api starts, and finally a smoke test checks it. · Duration: approx. 60 min · Prerequisite: Day 3

Briefing

  • Without instructions, Argo CD applies all objects of an Application in one go, only roughly sorted by kind (Namespaces and Secrets before Deployments).
  • Sync waves define an order. The annotation argocd.argoproj.io/sync-wave: "-1" comes before wave 0, which comes before 1. Argo CD waits until a wave is Healthy before it touches the next one.
  • Hooks are objects, usually Jobs, that only run during a sync: PreSync before all waves, Sync in the middle of the waves, PostSync afterwards, SyncFail on failure.
  • The hook-delete-policy determines when Argo CD deletes a hook Job again.
 PreSync ─► Wave -1 ─► Wave 0 ─► Wave 1 ─► PostSync
            Postgres    Migration   notes-api   Smoke test
            (Secret,    (Sync hook)
             PVC, DB)

Drill 1: Postgres in Wave -1

Four files go into apps/notes, all with wave -1:

cd ~/argo-lab/argocd-fleet/apps/notes
cat > db-secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-db
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
stringData:
  password: lab-db-pw-123
EOF
cat > db-pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: notes-db
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF
cat > db-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes-db
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  selector:
    app: notes-db
  ports:
    - port: 5432
EOF
cat > db-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-db
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: notes-db
  template:
    metadata:
      labels:
        app: notes-db
    spec:
      containers:
        - name: postgres
          image: postgres:18.6-alpine
          env:
            - name: POSTGRES_USER
              value: notes
            - name: POSTGRES_DB
              value: notes
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-db
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "notes", "-d", "notes"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: notes-db
EOF
  • strategy: Recreate: two Postgres pods must never write to the same volume at the same time. During an update, Kubernetes stops the old pod first.
  • The readiness probe with pg_isready makes sure that Healthy only comes once Postgres accepts connections. Only then does Argo CD move on to the next wave.

notes-api gets the database URL and wave 1. The whole file:

cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
        - name: api
          image: ghcr.io/thomaszachmann/notes-api:0.5.3
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: notes-api
                  key: apiKey
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-db
                  key: password
            - name: NOTES_DB_URL
              value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
EOF

Kubernetes substitutes $(DB_PASSWORD) itself, because the variable appears earlier in the same list. That way the password lives only in the Secret.

Save your progress? Sign up for free

Drill 2: The Migration as a PreSync Hook

notes-api migrate creates the schema. The first idea: a Job that runs before everything else, so PreSync:

cat > migrate-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: notes-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/thomaszachmann/notes-api:0.5.3
          args: ["migrate"]
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: notes-db
                  key: password
            - name: NOTES_DB_URL
              value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
EOF
cd ~/argo-lab/argocd-fleet
git add .
git commit -m "Postgres und Migration (PreSync)"
git push
argocd app get notes --refresh

Wait a minute:

sleep 60
argocd app get notes

Expected (abridged):

Sync Status:        OutOfSync from main (94cb48e)
Health Status:      Healthy

GROUP  KIND                   NAMESPACE  NAME           STATUS     HEALTH   HOOK     MESSAGE
batch  Job                    notes      notes-migrate  Running    Synced   PreSync  job.batch/notes-migrate created
       PersistentVolumeClaim  notes      notes-db       OutOfSync  Missing           
       Secret                 notes      notes-db       OutOfSync  Missing           
…
apps   Deployment             notes      notes-db       OutOfSync  Missing           

The sync is stuck. The Job is running, everything else is still missing. The events show why:

kubectl -n notes get events --field-selector reason=Failed

Expected (abridged):

LAST SEEN   TYPE      REASON   OBJECT                    MESSAGE
6s          Warning   Failed   pod/notes-migrate-6dwjl   Error: secret "notes-db" not found

PreSync really means before everything, even before wave -1. The Secret with the password does not exist yet, and the pod cannot start. Argo CD keeps waiting patiently. You abort the sync:

argocd app terminate-op notes

Expected:

Application 'notes' operation terminating

Save your progress? Sign up for free

Drill 3: The Migration as a Sync Hook in Wave 0

The migration belongs between the database and notes-api. A Sync hook without its own wave runs in wave 0:

sed -i "s/hook: PreSync/hook: Sync/" apps/notes/migrate-job.yaml
head -8 apps/notes/migrate-job.yaml

Expected:

apiVersion: batch/v1
kind: Job
metadata:
  name: notes-migrate
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
git commit -am "Migration als Sync-Hook"
git push
argocd app get notes --refresh
argocd app wait notes --operation --health --timeout 300

Expected (abridged):

…
GROUP  KIND                   NAMESPACE  NAME           STATUS     HEALTH   HOOK  MESSAGE
       Secret                 notes      notes-db       Synced                    secret/notes-db created
       PersistentVolumeClaim  notes      notes-db       Synced     Healthy        persistentvolumeclaim/notes-db created
       Service                notes      notes-db       Synced     Healthy        service/notes-db created
apps   Deployment             notes      notes-db       Synced     Healthy        deployment.apps/notes-db created
       Secret                 notes      notes-api      Synced                    secret/notes-api configured
       Service                notes      notes          Synced     Healthy        service/notes unchanged
batch  Job                    notes      notes-migrate  Succeeded  Synced   Sync  Reached expected number of succeeded pods
apps   Deployment             notes      notes          Synced     Healthy        deployment.apps/notes configured

The order of the lines is the order of the waves: first Postgres, then the remaining objects of wave 0 together with the Job, and finally notes-api. What the Job did:

kubectl -n notes logs job/notes-migrate
kubectl -n notes logs deploy/notes | grep store

Expected (abridged):

time=2026-11-14T08:41:11.202Z level=INFO msg="migration complete" table=notes
Found 2 pods, using pod/notes-75c5f54c4c-fzkvp
time=2026-11-14T08:41:16.323Z level=INFO msg="store ready" store=postgres

Save your progress? Sign up for free

Drill 4: The Data Stays

curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo Argo CD"}' \
  localhost:30080/notes
kubectl -n notes rollout restart deploy/notes
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/notes

Expected (abridged):

{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}
…
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]

Both pods are new, the note is still there. rollout restart sets an annotation that is not in Git. Self-heal therefore leaves it alone (Day 3, Drill 4).

kubectl -n notes exec deploy/notes-db -- \
  psql -U notes -d notes -c "select text from notes"

Expected:

     text      
---------------
 Hallo Argo CD
(1 row)

Save your progress? Sign up for free

Drill 5: A Smoke Test as a PostSync Hook

After every sync, a Job should check whether notes-api answers:

cat > apps/notes/smoke-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: notes-smoke
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: curlimages/curl:8.17.0
          args: ["-sf", "http://notes/notes"]
EOF
git add .
git commit -m "Smoke-Test nach dem Sync"
git push
argocd app get notes --refresh | grep "Sync Status"

Expected:

Sync Status:        Synced to main (a4d8401)

Tried it: The commit lands, the Application is immediately Synced, but no sync runs and the smoke test never starts. Hooks do not count in the comparison. A commit that only changes a hook therefore creates no deviation, and automated has nothing to do. The hook only runs on the next real sync.

You trigger it by hand:

argocd app sync notes

Expected (abridged):

…
batch  Job        notes   notes-migrate  Succeeded   Synced   Sync      Reached expected number of succeeded pods
batch  Job        notes   notes-smoke    Running     Synced   PostSync  job.batch/notes-smoke created
batch  Job        notes   notes-smoke    Succeeded   Synced   PostSync  Reached expected number of succeeded pods
…
Phase:              Succeeded
Message:            successfully synced (no more tasks)
kubectl -n notes get jobs

Expected:

NAME            STATUS     COMPLETIONS   DURATION   AGE
notes-migrate   Complete   1/1           5s         19s

notes-smoke is already gone: HookSucceeded deletes the Job after success. notes-migrate has BeforeHookCreation and stays until the next sync.

Save your progress? Sign up for free

Checkpoint

argocd app get notes | grep -E "Sync Status|Health Status"

Expected: Synced and Healthy.

kubectl -n notes get pods

Expected: two pods notes-…, one pod notes-db-…, all Running, plus notes-migrate-… with Completed.

curl -s localhost:30080/notes

Expected: the note Hallo Argo CD.

Challenge

Break the smoke test by making it query an address that does not exist. What happens to the sync and to notes-api? Then fix it again.

Hint: curl -f fails on an HTTP error. After the fix, look closely at the message for the Job.

Solution

sed -i "s#http://notes/notes#http://notes/gibtsnicht#" apps/notes/smoke-job.yaml
git commit -am "Smoke-Test kaputt"
git push
argocd app get notes --refresh
argocd app sync notes

Expected (abridged):

…  notes-smoke    Failed   Synced    PostSync  Job has reached the specified backoff limit
Phase:              Failed
Message:            one or more synchronization tasks completed unsuccessfully
argocd app get notes | grep -E "Sync Status|Health Status"
curl -s localhost:30080/version

Expected:

Sync Status:        Synced to main (ea5b39e)
Health Status:      Healthy
0.5.3

The sync counts as failed, but notes-api keeps running. A PostSync hook cannot roll anything back; it only reports. HookSucceeded leaves the failed Job in place so you can investigate it. Now the fix:

sed -i "s#http://notes/gibtsnicht#http://notes/notes#" apps/notes/smoke-job.yaml
git commit -am "Smoke-Test repariert"
git push
argocd app get notes --refresh
argocd app sync notes

Expected (abridged):

…  notes-smoke  Failed  SyncFailed  PostSync  error when patching "/dev/shm/3202005619": Job.batch "notes-smoke" is invalid: spec.template: Invalid value: …: field is immutable
Phase:              Failed

The old, failed Job is still there. Argo CD tries to change it, and Kubernetes forbids that for Jobs. The solution is both policies together: delete before the next run, delete after success.

cat > apps/notes/smoke-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: notes-smoke
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: curlimages/curl:8.17.0
          args: ["-sf", "http://notes/notes"]
EOF
git commit -am "Smoke-Test: alten Job vorher löschen"
git push
argocd app get notes --refresh
argocd app sync notes

Expected (abridged):

…
Phase:              Succeeded
Message:            successfully synced (no more tasks)

Save your progress? Sign up for free

Enterprise Note

  • Migrations as hooks are convenient but risky: a rollback via Argo CD does not roll back the schema. Migrations therefore have to be backward compatible.
  • Without an explicit policy, Argo CD sets BeforeHookCreation. If you specify a different one, it replaces the default instead of adding to it.
  • In production, Postgres itself belongs in an operator (CloudNativePG) or a managed service, not in a Deployment with a volume.