Day 12: Image Updater and Notifications
Goal: A new notes-api image lands in the fleet repo as a commit without manual work. After every deployment, Argo CD sends a message to a webhook. · Duration: approx. 50 min · Prerequisite: Day 8
Briefing
- So far you change the image tag by hand in Git. The Argo CD Image Updater regularly asks the registry for new tags and enters the matching one itself.
- Since version 1.0, the configuration lives in its own
ImageUpdaterobject, no longer in annotations on the Application. - There are two ways to write back:
argocdchanges the Application in the cluster,gitmakes a commit to the fleet repo. Only the second one is GitOps. - Notifications are part of Argo CD and have been
running since Day 1 as
argocd-notifications-controller. A trigger says when something happens, a template what is sent, a service where to. - An Application subscribes to a trigger with an annotation.
ghcr.io ──tags──► Image Updater ──git push──► Fleet repo ──► Argo CD ──sync──► notes
│
└─ Notification ─► echo
Drill 1: Install the Image Updater
mkdir -p ~/argo-lab/tag12
cd ~/argo-lab/tag12
IU_URL=https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater
curl -sLO $IU_URL/v1.5.0/config/install.yaml
kubectl apply -n argocd -f install.yaml
kubectl -n argocd rollout status deploy argocd-image-updater-controller --timeout 120sExpected (abridged):
customresourcedefinition.apiextensions.k8s.io/imageupdaters.argocd-image-updater.argoproj.io created
serviceaccount/argocd-image-updater-controller created
…
deployment.apps/argocd-image-updater-controller created
…
deployment "argocd-image-updater-controller" successfully rolled out
The controller runs in the argocd namespace and only
sees objects there. That is correct, because that is where the
Applications live too.
Which tags does notes-api have? The ghcr.io registry hands them out without logging in. You only need an anonymous token:
TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:thomaszachmann/notes-api:pull" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
curl -s -H "Authorization: Bearer $TOKEN" \
"https://ghcr.io/v2/thomaszachmann/notes-api/tags/list?n=1000" \
| python3 -c 'import sys,json; print([t for t in json.load(sys.stdin)["tags"]
if t[0]=="0"])'Expected:
['0.5.1', '0.5', '0.6.0-rc.1', '0.5.2', '0.5.3', '0.6.0-rc.2', '0.6.0-rc.3', '0.6.0-rc.4', '0.7.0', '0.7', '0.7.1']
The production Application notes runs
0.5.3, notes-dev has run 0.7.1
since Day 6. The 0.7 line is approved, so production should
get every patch from it too.
Save your progress? Sign up for free
Drill 2: Write Back to the Application
First an attempt with the default method argocd. The
file deliberately does not go into the repo:
cat > notes-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
name: notes
namespace: argocd
spec:
applicationRefs:
- namePattern: notes
images:
- alias: api
imageName: ghcr.io/thomaszachmann/notes-api:~0.7
EOF
kubectl apply -f notes-updater.yamlnamePatternselects Applications by their name, here exactlynotes.~0.7is a version constraint: everything from0.7.0and below0.8.0. The release candidates0.6.0-rc.*drop out anyway.aliasis a freely chosen name for the image within theImageUpdater.
The controller checks every two minutes. You wait for two runs:
sleep 180
kubectl -n argocd logs deploy/argocd-image-updater-controller \
| grep -E "Setting new image" | cut -c1-110
kubectl -n argocd get app notes -o jsonpath='{.spec.source.kustomize}'
echo
kubectl -n notes get deploy notes \
-o jsonpath='{.spec.template.spec.containers[0].image}'
echoExpected:
time="2026-11-15T07:44:31Z" level=info msg="Setting new image to ghcr.io/thomaszachmann/notes-api:0.7.1"
time="2026-11-15T07:46:32Z" level=info msg="Setting new image to ghcr.io/thomaszachmann/notes-api:0.7.1"
ghcr.io/thomaszachmann/notes-api:0.5.3
Tried it: The Image Updater sets
0.7.1every two minutes, and shortly afterwards0.5.3is back. Theargocdmethod writes the image into the Application. But that Application belongs to the ApplicationSet from Day 8, which immediately enforces its template again. Two controllers fight over the same field, and notes is rolled out again every time.
Save your progress? Sign up for free
Drill 3: Write Back to Git
With method: git, the Image Updater makes a commit.
writeBackTarget: kustomization writes the image into the
overlay’s kustomization.yaml as if you had run
kustomize edit set image. It pushes with the deploy key
that Argo CD has known since Day 2. That is why you gave it write access
on Day 0.
This time the object goes into the fleet repo and replaces the attempt:
cd ~/argo-lab/argocd-fleet
cat > applications/notes-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
name: notes
namespace: argocd
spec:
writeBackConfig:
method: git
gitConfig:
branch: main
writeBackTarget: kustomization
applicationRefs:
- namePattern: notes
images:
- alias: api
imageName: ghcr.io/thomaszachmann/notes-api:~0.7
EOF
git add -A
git commit -m "ImageUpdater notes"
git push
argocd app get root --refresh
sleep 150Now someone else has written to the repo. You fetch their commit:
git pull
git log -2 --format="%h %an: %s"
cat apps/notes/overlays/prod/kustomization.yamlExpected (abridged):
9bd76bb argocd-image-updater: build: automatic update of notes
…
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: ghcr.io/thomaszachmann/notes-api
newTag: 0.7.1
From now on: before each of your own changes, run
git pull first. Otherwise GitHub rejects your
git push because the Image Updater was faster.
argocd app get notes | grep -E "Sync Status|Health Status"
curl -s localhost:30080/version; echo
curl -s localhost:30080/notes; echoExpected:
Sync Status: Synced to main (9bd76bb)
Health Status: Healthy
0.7.1
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"},{"id":"21d3c5b0-d13b-4f1a-8c0e-a9350c1eb22e","text":"neuer Key"}]
Production runs 0.7.1, and both notes are still there.
In the next run, the Image Updater finds nothing left to do:
kubectl -n argocd logs deploy/argocd-image-updater-controller --since=150s \
| grep "Processing results" | cut -c1-140Expected:
time="…" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=
Save your progress? Sign up for free
Drill 4: A Receiver for Messages
Instead of Slack or Teams, the lab uses a small service in the cluster that writes every request to its log. Like everything else, it comes in via Git:
mkdir -p platform/echo
cat > platform/echo/echo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
spec:
replicas: 1
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- name: echo
image: mendhak/http-https-echo:43
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: echo
spec:
selector:
app: echo
ports:
- port: 80
targetPort: 8080
EOF
cat > applications/echo.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: echo
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: platform/echo
destination:
server: https://kubernetes.default.svc
namespace: echo
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOFSave your progress? Sign up for free
Drill 5: Service, Trigger, Template
The configuration lives in argocd-notifications-cm. The
installation created it empty, so, like argocd-rbac-cm on
Day 9, it goes into platform/argocd as a complete file:
cat > platform/argocd/argocd-notifications-cm.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
labels:
app.kubernetes.io/name: argocd-notifications-cm
app.kubernetes.io/part-of: argocd
data:
service.webhook.echo: |
url: http://echo.echo.svc
headers:
- name: Content-Type
value: application/json
trigger.on-deployed: |
- when: >-
app.status.operationState.phase in ['Succeeded'] and
app.status.health.status == 'Healthy'
oncePer: app.status.operationState.syncResult.revision
send: [deployed]
template.deployed: |
webhook:
echo:
method: POST
body: |
{
"app": "{{.app.metadata.name}}",
"revision": "{{.app.status.operationState.syncResult.revision}}",
"images": "{{.app.status.summary.images}}"
}
EOF
git add -A
git commit -m "Notifications an echo"
git push
argocd app get root --refresh
argocd app get argocd-extras --refresh
sleep 30
kubectl -n echo get podsExpected (abridged):
NAME READY STATUS RESTARTS AGE
echo-74669bfbb7-sf67p 1/1 Running 0 27s
service.webhook.echodefines a webhook namedecho.trigger.on-deployedfires when a sync has succeeded and the Application is healthy.oncePermakes sure this happens only once per commit.template.deployedbuilds the text. It has access to the whole Application as.app.
Whether the condition currently holds for an Application is revealed by the CLI in the notification controller’s container:
kubectl -n argocd exec deploy/argocd-notifications-controller -- \
argocd admin notifications trigger run on-deployed notes-devExpected:
CONDITION RESULT
app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' true
Save your progress? Sign up for free
Drill 6: Subscribe
The notes Applications are generated by the ApplicationSet. It would
remove an annotation added by hand, so the annotation belongs in the
template. In applications/notes-appset.yaml, add two lines
under template.metadata, so that the beginning of the
template looks like this:
template:
metadata:
name: "{{.name}}"
annotations:
notifications.argoproj.io/subscribe.on-deployed.echo: ""The structure is
subscribe.<trigger>.<service>. For Slack, the
value would be the channel. A webhook does not need one, so it stays
empty.
git pull
git commit -am "notes: Benachrichtigung bei Deployment"
git push
argocd app get root --refreshNow a real deployment: notes-dev gets two replicas.
sed -i "s/replicas: 1/replicas: 2/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: zwei Repliken"
git push
git log -1 --format=%h
argocd app get notes-dev --refresh
sleep 30
kubectl -n echo logs deploy/echo | grep '"body"' | tail -1Expected (abridged):
36ca1fe
…
"body": "{\n \"app\": \"notes-dev\",\n \"revision\": \"36ca1fe94ba4eca675ea5a9d683d8fd52c16de5f\",\n \"images\": \"[ghcr.io/thomaszachmann/notes-api:0.7.1 postgres:18.6-alpine]\"\n}\n",
Argo CD remembers which messages have already been sent in an annotation on the Application:
kubectl -n argocd get app notes-dev \
-o jsonpath='{.metadata.annotations.notified\.notifications\.argoproj\.io}'
echoExpected (abridged):
{"36ca1fe94ba4eca675ea5a9d683d8fd52c16de5f:on-deployed:[0].y7b5…:echo:":1794729675,…}
Set the replicas back to one. That also produces a message:
sed -i "s/replicas: 2/replicas: 1/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: wieder eine Replik"
git pushTried it: A typo in the condition, such as missing quotes around
Succeeded, produces no error. The controller only writesTrigger 'on-deployed' FAILEDto the log, and no message ever arrives.trigger runthen showsfalse.
Save your progress? Sign up for free
Checkpoint
kubectl -n argocd get imageupdater notesExpected: READY is True.
grep -A2 "images:" ~/argo-lab/argocd-fleet/apps/notes/overlays/prod/kustomization.yamlExpected: newTag: 0.7.1.
kubectl -n argocd get app notes -o jsonpath='{.metadata.annotations}' \
| grep -o 'subscribe[^"]*'Expected: subscribe.on-deployed.echo.
kubectl -n echo logs deploy/echo | grep -c '"body"'Expected: at least 3.
Challenge
Let the Image Updater watch notes-dev as well. What does
it write to the repo?
Hint: namePattern understands wildcards
such as *.
Solution
cd ~/argo-lab/argocd-fleet
git pull
sed -i 's/namePattern: notes$/namePattern: "notes*"/' applications/notes-updater.yaml
grep -n namePattern applications/notes-updater.yaml
git commit -am "ImageUpdater auch für notes-dev"
git push
argocd app get root --refresh
sleep 150
kubectl -n argocd logs deploy/argocd-image-updater-controller --since=150s \
| grep -E "considering|Processing results" | cut -c1-140
kubectl -n argocd get imageupdater notesExpected (abridged):
13: - namePattern: "notes*"
…
time="…" level=info msg="Starting image update cycle, considering 2 application(s) for update" controller=imageupdater co
time="…" level=info msg="Processing results: applications=2 images_considered=2 images_skipped=0 images_updated=0 errors=
NAME APPS IMAGES LAST CHECKED READY
notes 2 2 27s True
Nothing. notes-dev already runs 0.7.1, the
highest tag under ~0.7. Only a tag 0.7.2 in
the registry triggers a commit, and then for both environments.
Save your progress? Sign up for free
Enterprise Note
- The Image Updater writes directly to
main. With a protected branch, it writes to a separate one (branch: "main:image-updater"), and a human merges via pull request. - Besides
semver, there aredigestfor mutable tags such aslatest, andnewest-build. Fixed version numbers withsemverare the easiest to trace. - Private registries need a pull secret that the Image Updater is allowed to read.
- For Slack, Teams, email, and many others, Argo CD ships ready-made
services. A catalog of ready-made triggers and templates is in the Argo
CD repo under
notifications_catalog. Credentials such as a Slack token belong inargocd-notifications-secret, never in the ConfigMap.