Day 4: Postgres, Sync Waves, and Hooks
Goal: notes-api stores its data in Postgres. Argo CD first creates the database, then a Job migrates the schema, then notes-api starts, and finally a smoke test checks it. · Duration: approx. 60 min · Prerequisite: Day 3
Briefing
- Without instructions, Argo CD applies all objects of an Application in one go, only roughly sorted by kind (Namespaces and Secrets before Deployments).
- Sync waves define an order. The annotation
argocd.argoproj.io/sync-wave: "-1"comes before wave0, which comes before1. Argo CD waits until a wave isHealthybefore it touches the next one. - Hooks are objects, usually Jobs, that only run
during a sync:
PreSyncbefore all waves,Syncin the middle of the waves,PostSyncafterwards,SyncFailon failure. - The hook-delete-policy determines when Argo CD deletes a hook Job again.
PreSync ─► Wave -1 ─► Wave 0 ─► Wave 1 ─► PostSync
Postgres Migration notes-api Smoke test
(Secret, (Sync hook)
PVC, DB)
Drill 1: Postgres in Wave -1
Four files go into apps/notes, all with wave
-1:
cd ~/argo-lab/argocd-fleet/apps/notes
cat > db-secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
stringData:
password: lab-db-pw-123
EOF
cat > db-pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
cat > db-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
selector:
app: notes-db
ports:
- port: 5432
EOFcat > db-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: notes-db
template:
metadata:
labels:
app: notes-db
spec:
containers:
- name: postgres
image: postgres:18.6-alpine
env:
- name: POSTGRES_USER
value: notes
- name: POSTGRES_DB
value: notes
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: notes-db
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- containerPort: 5432
readinessProbe:
exec:
command: ["pg_isready", "-U", "notes", "-d", "notes"]
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: notes-db
EOFstrategy: Recreate: two Postgres pods must never write to the same volume at the same time. During an update, Kubernetes stops the old pod first.- The readiness probe with
pg_isreadymakes sure thatHealthyonly comes once Postgres accepts connections. Only then does Argo CD move on to the next wave.
notes-api gets the database URL and wave 1. The whole
file:
cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
replicas: 2
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: api
image: ghcr.io/thomaszachmann/notes-api:0.5.3
ports:
- containerPort: 8080
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: notes-api
key: apiKey
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: notes-db
key: password
- name: NOTES_DB_URL
value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
readinessProbe:
httpGet:
path: /healthz
port: 8080
EOFKubernetes substitutes $(DB_PASSWORD) itself, because
the variable appears earlier in the same list. That way the password
lives only in the Secret.
Save your progress? Sign up for free
Drill 2: The Migration as a PreSync Hook
notes-api migrate creates the schema. The first idea: a
Job that runs before everything else, so PreSync:
cat > migrate-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: notes-migrate
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 1
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.5.3
args: ["migrate"]
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: notes-db
key: password
- name: NOTES_DB_URL
value: postgres://notes:$(DB_PASSWORD)@notes-db:5432/notes?sslmode=disable
EOF
cd ~/argo-lab/argocd-fleet
git add .
git commit -m "Postgres und Migration (PreSync)"
git push
argocd app get notes --refreshWait a minute:
sleep 60
argocd app get notesExpected (abridged):
Sync Status: OutOfSync from main (94cb48e)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
batch Job notes notes-migrate Running Synced PreSync job.batch/notes-migrate created
PersistentVolumeClaim notes notes-db OutOfSync Missing
Secret notes notes-db OutOfSync Missing
…
apps Deployment notes notes-db OutOfSync Missing
The sync is stuck. The Job is running, everything else is still missing. The events show why:
kubectl -n notes get events --field-selector reason=FailedExpected (abridged):
LAST SEEN TYPE REASON OBJECT MESSAGE
6s Warning Failed pod/notes-migrate-6dwjl Error: secret "notes-db" not found
PreSync really means before everything, even before wave
-1. The Secret with the password does not exist yet, and
the pod cannot start. Argo CD keeps waiting patiently. You abort the
sync:
argocd app terminate-op notesExpected:
Application 'notes' operation terminating
Save your progress? Sign up for free
Drill 3: The Migration as a Sync Hook in Wave 0
The migration belongs between the database and notes-api. A
Sync hook without its own wave runs in wave
0:
sed -i "s/hook: PreSync/hook: Sync/" apps/notes/migrate-job.yaml
head -8 apps/notes/migrate-job.yamlExpected:
apiVersion: batch/v1
kind: Job
metadata:
name: notes-migrate
annotations:
argocd.argoproj.io/hook: Sync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
git commit -am "Migration als Sync-Hook"
git push
argocd app get notes --refresh
argocd app wait notes --operation --health --timeout 300Expected (abridged):
…
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Secret notes notes-db Synced secret/notes-db created
PersistentVolumeClaim notes notes-db Synced Healthy persistentvolumeclaim/notes-db created
Service notes notes-db Synced Healthy service/notes-db created
apps Deployment notes notes-db Synced Healthy deployment.apps/notes-db created
Secret notes notes-api Synced secret/notes-api configured
Service notes notes Synced Healthy service/notes unchanged
batch Job notes notes-migrate Succeeded Synced Sync Reached expected number of succeeded pods
apps Deployment notes notes Synced Healthy deployment.apps/notes configured
The order of the lines is the order of the waves: first Postgres,
then the remaining objects of wave 0 together with the Job,
and finally notes-api. What the Job did:
kubectl -n notes logs job/notes-migrate
kubectl -n notes logs deploy/notes | grep storeExpected (abridged):
time=2026-11-14T08:41:11.202Z level=INFO msg="migration complete" table=notes
Found 2 pods, using pod/notes-75c5f54c4c-fzkvp
time=2026-11-14T08:41:16.323Z level=INFO msg="store ready" store=postgres
Save your progress? Sign up for free
Drill 4: The Data Stays
curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo Argo CD"}' \
localhost:30080/notes
kubectl -n notes rollout restart deploy/notes
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/notesExpected (abridged):
{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}
…
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]
Both pods are new, the note is still there.
rollout restart sets an annotation that is not in Git.
Self-heal therefore leaves it alone (Day 3, Drill 4).
kubectl -n notes exec deploy/notes-db -- \
psql -U notes -d notes -c "select text from notes"Expected:
text
---------------
Hallo Argo CD
(1 row)
Save your progress? Sign up for free
Drill 5: A Smoke Test as a PostSync Hook
After every sync, a Job should check whether notes-api answers:
cat > apps/notes/smoke-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: notes-smoke
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: smoke
image: curlimages/curl:8.17.0
args: ["-sf", "http://notes/notes"]
EOF
git add .
git commit -m "Smoke-Test nach dem Sync"
git push
argocd app get notes --refresh | grep "Sync Status"Expected:
Sync Status: Synced to main (a4d8401)
Tried it: The commit lands, the Application is immediately
Synced, but no sync runs and the smoke test never starts. Hooks do not count in the comparison. A commit that only changes a hook therefore creates no deviation, andautomatedhas nothing to do. The hook only runs on the next real sync.
You trigger it by hand:
argocd app sync notesExpected (abridged):
…
batch Job notes notes-migrate Succeeded Synced Sync Reached expected number of succeeded pods
batch Job notes notes-smoke Running Synced PostSync job.batch/notes-smoke created
batch Job notes notes-smoke Succeeded Synced PostSync Reached expected number of succeeded pods
…
Phase: Succeeded
Message: successfully synced (no more tasks)
kubectl -n notes get jobsExpected:
NAME STATUS COMPLETIONS DURATION AGE
notes-migrate Complete 1/1 5s 19s
notes-smoke is already gone: HookSucceeded
deletes the Job after success. notes-migrate has
BeforeHookCreation and stays until the next sync.
Save your progress? Sign up for free
Checkpoint
argocd app get notes | grep -E "Sync Status|Health Status"Expected: Synced and Healthy.
kubectl -n notes get podsExpected: two pods notes-…, one pod
notes-db-…, all Running, plus
notes-migrate-… with Completed.
curl -s localhost:30080/notesExpected: the note Hallo Argo CD.
Challenge
Break the smoke test by making it query an address that does not exist. What happens to the sync and to notes-api? Then fix it again.
Hint: curl -f fails on an HTTP error.
After the fix, look closely at the message for the Job.
Solution
sed -i "s#http://notes/notes#http://notes/gibtsnicht#" apps/notes/smoke-job.yaml
git commit -am "Smoke-Test kaputt"
git push
argocd app get notes --refresh
argocd app sync notesExpected (abridged):
… notes-smoke Failed Synced PostSync Job has reached the specified backoff limit
Phase: Failed
Message: one or more synchronization tasks completed unsuccessfully
argocd app get notes | grep -E "Sync Status|Health Status"
curl -s localhost:30080/versionExpected:
Sync Status: Synced to main (ea5b39e)
Health Status: Healthy
0.5.3
The sync counts as failed, but notes-api keeps running. A PostSync
hook cannot roll anything back; it only reports.
HookSucceeded leaves the failed Job in place so you can
investigate it. Now the fix:
sed -i "s#http://notes/gibtsnicht#http://notes/notes#" apps/notes/smoke-job.yaml
git commit -am "Smoke-Test repariert"
git push
argocd app get notes --refresh
argocd app sync notesExpected (abridged):
… notes-smoke Failed SyncFailed PostSync error when patching "/dev/shm/3202005619": Job.batch "notes-smoke" is invalid: spec.template: Invalid value: …: field is immutable
Phase: Failed
The old, failed Job is still there. Argo CD tries to change it, and Kubernetes forbids that for Jobs. The solution is both policies together: delete before the next run, delete after success.
cat > apps/notes/smoke-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: notes-smoke
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: smoke
image: curlimages/curl:8.17.0
args: ["-sf", "http://notes/notes"]
EOF
git commit -am "Smoke-Test: alten Job vorher löschen"
git push
argocd app get notes --refresh
argocd app sync notesExpected (abridged):
…
Phase: Succeeded
Message: successfully synced (no more tasks)
Save your progress? Sign up for free
Enterprise Note
- Migrations as hooks are convenient but risky: a rollback via Argo CD does not roll back the schema. Migrations therefore have to be backward compatible.
- Without an explicit policy, Argo CD sets
BeforeHookCreation. If you specify a different one, it replaces the default instead of adding to it. - In production, Postgres itself belongs in an operator (CloudNativePG) or a managed service, not in a Deployment with a volume.