Skip to content
Hands-on IT IT Drills

Day 12: Image Updater and Notifications

Goal: A new notes-api image lands in the fleet repo as a commit without manual work. After every deployment, Argo CD sends a message to a webhook. · Duration: approx. 50 min · Prerequisite: Day 8

Briefing

  • So far you change the image tag by hand in Git. The Argo CD Image Updater regularly asks the registry for new tags and enters the matching one itself.
  • Since version 1.0, the configuration lives in its own ImageUpdater object, no longer in annotations on the Application.
  • There are two ways to write back: argocd changes the Application in the cluster, git makes a commit to the fleet repo. Only the second one is GitOps.
  • Notifications are part of Argo CD and have been running since Day 1 as argocd-notifications-controller. A trigger says when something happens, a template what is sent, a service where to.
  • An Application subscribes to a trigger with an annotation.
 ghcr.io ──tags──► Image Updater ──git push──► Fleet repo ──► Argo CD ──sync──► notes
                                                                │
                                                                └─ Notification ─► echo

Drill 1: Install the Image Updater

mkdir -p ~/argo-lab/tag12
cd ~/argo-lab/tag12
IU_URL=https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater
curl -sLO $IU_URL/v1.5.0/config/install.yaml
kubectl apply -n argocd -f install.yaml
kubectl -n argocd rollout status deploy argocd-image-updater-controller --timeout 120s

Expected (abridged):

customresourcedefinition.apiextensions.k8s.io/imageupdaters.argocd-image-updater.argoproj.io created
serviceaccount/argocd-image-updater-controller created
…
deployment.apps/argocd-image-updater-controller created
…
deployment "argocd-image-updater-controller" successfully rolled out

The controller runs in the argocd namespace and only sees objects there. That is correct, because that is where the Applications live too.

Which tags does notes-api have? The ghcr.io registry hands them out without logging in. You only need an anonymous token:

TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:thomaszachmann/notes-api:pull" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://ghcr.io/v2/thomaszachmann/notes-api/tags/list?n=1000" \
  | python3 -c 'import sys,json; print([t for t in json.load(sys.stdin)["tags"]
    if t[0]=="0"])'

Expected:

['0.5.1', '0.5', '0.6.0-rc.1', '0.5.2', '0.5.3', '0.6.0-rc.2', '0.6.0-rc.3', '0.6.0-rc.4', '0.7.0', '0.7', '0.7.1']

The production Application notes runs 0.5.3, notes-dev has run 0.7.1 since Day 6. The 0.7 line is approved, so production should get every patch from it too.

Save your progress? Sign up for free

Drill 2: Write Back to the Application

First an attempt with the default method argocd. The file deliberately does not go into the repo:

cat > notes-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: notes
  namespace: argocd
spec:
  applicationRefs:
    - namePattern: notes
      images:
        - alias: api
          imageName: ghcr.io/thomaszachmann/notes-api:~0.7
EOF
kubectl apply -f notes-updater.yaml
  • namePattern selects Applications by their name, here exactly notes.
  • ~0.7 is a version constraint: everything from 0.7.0 and below 0.8.0. The release candidates 0.6.0-rc.* drop out anyway.
  • alias is a freely chosen name for the image within the ImageUpdater.

The controller checks every two minutes. You wait for two runs:

sleep 180
kubectl -n argocd logs deploy/argocd-image-updater-controller \
  | grep -E "Setting new image" | cut -c1-110
kubectl -n argocd get app notes -o jsonpath='{.spec.source.kustomize}'
echo
kubectl -n notes get deploy notes \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
echo

Expected:

time="2026-11-15T07:44:31Z" level=info msg="Setting new image to ghcr.io/thomaszachmann/notes-api:0.7.1"
time="2026-11-15T07:46:32Z" level=info msg="Setting new image to ghcr.io/thomaszachmann/notes-api:0.7.1"

ghcr.io/thomaszachmann/notes-api:0.5.3

Tried it: The Image Updater sets 0.7.1 every two minutes, and shortly afterwards 0.5.3 is back. The argocd method writes the image into the Application. But that Application belongs to the ApplicationSet from Day 8, which immediately enforces its template again. Two controllers fight over the same field, and notes is rolled out again every time.

Save your progress? Sign up for free

Drill 3: Write Back to Git

With method: git, the Image Updater makes a commit. writeBackTarget: kustomization writes the image into the overlay’s kustomization.yaml as if you had run kustomize edit set image. It pushes with the deploy key that Argo CD has known since Day 2. That is why you gave it write access on Day 0.

This time the object goes into the fleet repo and replaces the attempt:

cd ~/argo-lab/argocd-fleet
cat > applications/notes-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: notes
  namespace: argocd
spec:
  writeBackConfig:
    method: git
    gitConfig:
      branch: main
      writeBackTarget: kustomization
  applicationRefs:
    - namePattern: notes
      images:
        - alias: api
          imageName: ghcr.io/thomaszachmann/notes-api:~0.7
EOF
git add -A
git commit -m "ImageUpdater notes"
git push
argocd app get root --refresh
sleep 150

Now someone else has written to the repo. You fetch their commit:

git pull
git log -2 --format="%h %an: %s"
cat apps/notes/overlays/prod/kustomization.yaml

Expected (abridged):

9bd76bb argocd-image-updater: build: automatic update of notes
…
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: ghcr.io/thomaszachmann/notes-api
    newTag: 0.7.1

From now on: before each of your own changes, run git pull first. Otherwise GitHub rejects your git push because the Image Updater was faster.

argocd app get notes | grep -E "Sync Status|Health Status"
curl -s localhost:30080/version; echo
curl -s localhost:30080/notes; echo

Expected:

Sync Status:        Synced to main (9bd76bb)
Health Status:      Healthy
0.7.1
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"},{"id":"21d3c5b0-d13b-4f1a-8c0e-a9350c1eb22e","text":"neuer Key"}]

Production runs 0.7.1, and both notes are still there. In the next run, the Image Updater finds nothing left to do:

kubectl -n argocd logs deploy/argocd-image-updater-controller --since=150s \
  | grep "Processing results" | cut -c1-140

Expected:

time="…" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=

Save your progress? Sign up for free

Drill 4: A Receiver for Messages

Instead of Slack or Teams, the lab uses a small service in the cluster that writes every request to its log. Like everything else, it comes in via Git:

mkdir -p platform/echo
cat > platform/echo/echo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: mendhak/http-https-echo:43
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 8080
EOF
cat > applications/echo.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: echo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [email protected]:<du>/argocd-fleet.git
    targetRevision: main
    path: platform/echo
  destination:
    server: https://kubernetes.default.svc
    namespace: echo
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
EOF

Save your progress? Sign up for free

Drill 5: Service, Trigger, Template

The configuration lives in argocd-notifications-cm. The installation created it empty, so, like argocd-rbac-cm on Day 9, it goes into platform/argocd as a complete file:

cat > platform/argocd/argocd-notifications-cm.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-notifications-cm
    app.kubernetes.io/part-of: argocd
data:
  service.webhook.echo: |
    url: http://echo.echo.svc
    headers:
      - name: Content-Type
        value: application/json
  trigger.on-deployed: |
    - when: >-
        app.status.operationState.phase in ['Succeeded'] and
        app.status.health.status == 'Healthy'
      oncePer: app.status.operationState.syncResult.revision
      send: [deployed]
  template.deployed: |
    webhook:
      echo:
        method: POST
        body: |
          {
            "app": "{{.app.metadata.name}}",
            "revision": "{{.app.status.operationState.syncResult.revision}}",
            "images": "{{.app.status.summary.images}}"
          }
EOF
git add -A
git commit -m "Notifications an echo"
git push
argocd app get root --refresh
argocd app get argocd-extras --refresh
sleep 30
kubectl -n echo get pods

Expected (abridged):

NAME                    READY   STATUS    RESTARTS   AGE
echo-74669bfbb7-sf67p   1/1     Running   0          27s
  • service.webhook.echo defines a webhook named echo.
  • trigger.on-deployed fires when a sync has succeeded and the Application is healthy. oncePer makes sure this happens only once per commit.
  • template.deployed builds the text. It has access to the whole Application as .app.

Whether the condition currently holds for an Application is revealed by the CLI in the notification controller’s container:

kubectl -n argocd exec deploy/argocd-notifications-controller -- \
  argocd admin notifications trigger run on-deployed notes-dev

Expected:

CONDITION                                                                                   RESULT
app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'  true

Save your progress? Sign up for free

Drill 6: Subscribe

The notes Applications are generated by the ApplicationSet. It would remove an annotation added by hand, so the annotation belongs in the template. In applications/notes-appset.yaml, add two lines under template.metadata, so that the beginning of the template looks like this:

  template:
    metadata:
      name: "{{.name}}"
      annotations:
        notifications.argoproj.io/subscribe.on-deployed.echo: ""

The structure is subscribe.<trigger>.<service>. For Slack, the value would be the channel. A webhook does not need one, so it stays empty.

git pull
git commit -am "notes: Benachrichtigung bei Deployment"
git push
argocd app get root --refresh

Now a real deployment: notes-dev gets two replicas.

sed -i "s/replicas: 1/replicas: 2/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: zwei Repliken"
git push
git log -1 --format=%h
argocd app get notes-dev --refresh
sleep 30
kubectl -n echo logs deploy/echo | grep '"body"' | tail -1

Expected (abridged):

36ca1fe
…
    "body": "{\n  \"app\": \"notes-dev\",\n  \"revision\": \"36ca1fe94ba4eca675ea5a9d683d8fd52c16de5f\",\n  \"images\": \"[ghcr.io/thomaszachmann/notes-api:0.7.1 postgres:18.6-alpine]\"\n}\n",

Argo CD remembers which messages have already been sent in an annotation on the Application:

kubectl -n argocd get app notes-dev \
  -o jsonpath='{.metadata.annotations.notified\.notifications\.argoproj\.io}'
echo

Expected (abridged):

{"36ca1fe94ba4eca675ea5a9d683d8fd52c16de5f:on-deployed:[0].y7b5…:echo:":1794729675,…}

Set the replicas back to one. That also produces a message:

sed -i "s/replicas: 2/replicas: 1/" apps/notes/overlays/dev/replicas.yaml
git commit -am "notes-dev: wieder eine Replik"
git push

Tried it: A typo in the condition, such as missing quotes around Succeeded, produces no error. The controller only writes Trigger 'on-deployed' FAILED to the log, and no message ever arrives. trigger run then shows false.

Save your progress? Sign up for free

Checkpoint

kubectl -n argocd get imageupdater notes

Expected: READY is True.

grep -A2 "images:" ~/argo-lab/argocd-fleet/apps/notes/overlays/prod/kustomization.yaml

Expected: newTag: 0.7.1.

kubectl -n argocd get app notes -o jsonpath='{.metadata.annotations}' \
  | grep -o 'subscribe[^"]*'

Expected: subscribe.on-deployed.echo.

kubectl -n echo logs deploy/echo | grep -c '"body"'

Expected: at least 3.

Challenge

Let the Image Updater watch notes-dev as well. What does it write to the repo?

Hint: namePattern understands wildcards such as *.

Solution

cd ~/argo-lab/argocd-fleet
git pull
sed -i 's/namePattern: notes$/namePattern: "notes*"/' applications/notes-updater.yaml
grep -n namePattern applications/notes-updater.yaml
git commit -am "ImageUpdater auch für notes-dev"
git push
argocd app get root --refresh
sleep 150
kubectl -n argocd logs deploy/argocd-image-updater-controller --since=150s \
  | grep -E "considering|Processing results" | cut -c1-140
kubectl -n argocd get imageupdater notes

Expected (abridged):

13:    - namePattern: "notes*"
…
time="…" level=info msg="Starting image update cycle, considering 2 application(s) for update" controller=imageupdater co
time="…" level=info msg="Processing results: applications=2 images_considered=2 images_skipped=0 images_updated=0 errors=
NAME    APPS   IMAGES   LAST CHECKED   READY
notes   2      2        27s            True

Nothing. notes-dev already runs 0.7.1, the highest tag under ~0.7. Only a tag 0.7.2 in the registry triggers a commit, and then for both environments.

Save your progress? Sign up for free

Enterprise Note

  • The Image Updater writes directly to main. With a protected branch, it writes to a separate one (branch: "main:image-updater"), and a human merges via pull request.
  • Besides semver, there are digest for mutable tags such as latest, and newest-build. Fixed version numbers with semver are the easiest to trace.
  • Private registries need a pull secret that the Image Updater is allowed to read.
  • For Slack, Teams, email, and many others, Argo CD ships ready-made services. A catalog of ready-made triggers and templates is in the Argo CD repo under notifications_catalog. Credentials such as a Slack token belong in argocd-notifications-secret, never in the ConfigMap.