Skip to content
Hands-on IT IT Drills

Day 9: Image Automation I – Watching Images

Goal: Add the two image controllers, give the deploy key write access and teach Flux to look for new notes-api versions on ghcr. · Duration: approx. 60 min · Prerequisite: Day 8

Briefing

So far you change the notes-api version in Git by hand. Image automation takes this over in two stages, split across two controllers:

Controller Object Task Day
image-reflector-controller ImageRepository regularly read the tags of a registry 9
image-reflector-controller ImagePolicy select the matching tag from the tags 9
image-automation-controller ImageUpdateAutomation write the selected tag to Git and push 10
  • Neither controller is part of the standard installation. You add them with flux bootstrap, using the same command as on Day 1 with one more option.
  • The automation writes to Git. For that, Flux needs a deploy key that is allowed to write. Since Day 1 it may only read.
  • Important: Flux never changes the image in the cluster directly. It writes a commit, and that takes the normal path. Git remains the single source of truth.

Drill 1: Adding the controllers

Before you continue, you fetch the state from GitHub. The bootstrap is about to commit to the repo, and your local clone should not lag behind:

cd ~/flux-lab/fleet
git pull
flux bootstrap github --owner=$GITHUB_USER --repository=flux-fleet \
  --branch=main --path=clusters/lab --personal --private=true \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key

Expected (shortened):

✔ committed component manifests to "main" ("1b0b397f3b2a2d2cf131314e8438805b077d03ee")
► pushing component manifests to "https://github.com/<du>/flux-fleet.git"
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
✔ source secret up to date
...
✔ image-automation-controller: deployment ready
✔ image-reflector-controller: deployment ready
...
✔ all components are healthy

Two new controllers, but one line deserves a second look: source secret up to date. The bootstrap kept the existing deploy key; --read-write-key applies only to new keys. On GitHub (flux-fleet → Settings → Deploy keys) it still says “Read-only”.

You delete the Secret with the key, and the next bootstrap generates a new one:

kubectl -n flux-system delete secret flux-system
flux bootstrap github --owner=$GITHUB_USER --repository=flux-fleet \
  --branch=main --path=clusters/lab --personal --private=true \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key

Expected (shortened):

secret "flux-system" deleted from flux-system namespace
...
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlz...
✔ configured deploy key "flux-system-main-flux-system-./clusters/lab" for "https://github.com/<du>/flux-fleet"
...
✔ all components are healthy

The bootstrap replaced the old deploy key on GitHub with the new one. It is now listed there without “Read-only”.

git pull
kubectl -n flux-system get deploy

Expected (the age differs):

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
helm-controller               1/1     1            1           7h
image-automation-controller   1/1     1            1           3m
image-reflector-controller    1/1     1            1           3m
kustomize-controller          1/1     1            1           7h
notification-controller       1/1     1            1           75m
source-controller             1/1     1            1           7h

Save your progress? Sign up for free

Drill 2: Watching a repository

The objects for image automation belong in the cluster folder, because they affect all environments:

F=clusters/lab/notes-images.yaml
flux create image repository notes-api --image=ghcr.io/thomaszachmann/notes-api \
  --interval=10m --export > $F
flux create image policy notes-api --image-ref=notes-api \
  --select-semver=">=0.5.0" --export >> $F
cat $F

Expected:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
  name: notes-api
  namespace: flux-system
spec:
  image: ghcr.io/thomaszachmann/notes-api
  interval: 10m0s
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: notes-api
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: notes-api
  policy:
    semver:
      range: '>=0.5.0'
git add -A
git commit -m "notes-api: Images beobachten"
git push
flux reconcile kustomization flux-system --with-source
flux get image repository
flux get image policy

Expected (shortened):

NAME        LAST SCAN               SUSPENDED   READY   MESSAGE
notes-api   2026-11-12T16:51:49Z    False       True    successful scan: found 130 tags with checksum sha256:8e4aea79…
NAME        IMAGE                               TAG     READY   MESSAGE
notes-api   ghcr.io/thomaszachmann/notes-api    0.7.1   True    Latest image tag for ghcr.io/thomaszachmann/notes-api resolved to 0.7.1

Among all tags, the policy found the highest version that matches >=0.5.0. Pre-releases like 0.6.0-rc.4 do not count with SemVer as long as the range itself does not contain one.

Save your progress? Sign up for free

Drill 3: What is all this junk?

130 tags for a handful of versions? A look at the most recently read ones:

kubectl -n flux-system get imagerepository notes-api \
  -o jsonpath='{.status.lastScanResult.latestTags}{"\n"}' | tr , '\n' | head -3

Expected:

["sha256-fadb2fe90ea04229a9586865474eea5f0b91806cbca535affe6fb48c31f07c4e"
"sha256-f72f0f1c3a640086eaacd443e4dbaeda0bf21be31acb00ba8ed59d9cc9eccc15"
"sha256-f214133d6effdf6d330a7f0adf06b3ca03350b9a291ea3c10d948e3bce5ed7c1"
  • sha256-… are not images but attestations such as SBOMs and provenance records. The CI attaches them to every image it builds (see “Secure supply chains hands-on”).
  • On top of that come tags like sha-fda8903: one image per commit.

They do not hurt the policy, because SemVer ignores everything that is not a version. But the controller stores every tag, at every scan. An exclusionList filters them out while reading. In clusters/lab/notes-images.yaml, add below image::

  image: ghcr.io/thomaszachmann/notes-api
  exclusionList:
    - ^.*\.sig$
    - ^sha

The first line is the default value (cosign signatures). Setting the list overrides it, so it has to be included.

git commit -am "notes-api: nur Versions-Tags"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile image repository notes-api
kubectl -n flux-system get imagerepository notes-api \
  -o jsonpath='{.status.lastScanResult.latestTags}{"\n"}'

Expected (shortened):

✔ scan fetched 13 tags
["main","latest","0.7.1","0.7.0","0.7","0.6.0-rc.4","0.6.0-rc.3","0.6.0-rc.2","0.6.0-rc.1","0.5.3"]

13 instead of 130. main and latest are moving tags. The policy skips them, and they should never appear in Git.

Save your progress? Sign up for free

Drill 4: Other selection rules

SemVer is the most common rule, but not the only one. To try it out, you create two policies directly in the cluster, without --export. They are removed again afterwards.

Only patches of 0.5, via a regular expression and numeric comparison:

flux create image policy nur-05 --image-ref=notes-api \
  --filter-regex='^0\.5\.(?P<p>[0-9]+)$' --filter-extract='$p' --select-numeric=asc
  • --filter-regex lets through only tags that match. (?P<p>…) captures the patch number.
  • --filter-extract='$p' compares only this part; --select-numeric=asc takes the highest.

The same technique works for tags like main-20261112-1530 that are not SemVer versions: extract the timestamp, sort numerically.

flux get image policy

Expected:

NAME        IMAGE                               TAG     READY   MESSAGE
notes-api   ghcr.io/thomaszachmann/notes-api    0.7.1   True    Latest image tag for ghcr.io/thomaszachmann/notes-api resolved to 0.7.1
nur-05      ghcr.io/thomaszachmann/notes-api    0.5.3   True    Latest image tag for ghcr.io/thomaszachmann/notes-api resolved to 0.5.3
flux delete image policy nur-05 --silent

Expected:

► deleting image policy nur-05 in flux-system namespace
✔ image policy deleted

Save your progress? Sign up for free

Checkpoint

flux get image repository

Expected: notes-api, READY True, found 13 tags.

flux get image policy

Expected: exactly one policy notes-api with 0.7.1.

diff ~/flux-lab/fleet/clusters/lab/notes-images.yaml \
  ~/flux-lab/code/loesung/tag09/clusters/lab/notes-images.yaml

Expected: no output.

Challenge

The QA team wants to test the release candidates of the 0.6 series as soon as they appear, but never a final 0.7. Write a policy notes-api-rc for that, this time in Git. Which tag does it select?

Hint: A SemVer range includes pre-releases only if it contains one itself. -0 is the smallest possible one.

Solution

flux create image policy notes-api-rc --image-ref=notes-api \
  --select-semver=">=0.6.0-0 <0.7.0" --export >> clusters/lab/notes-images.yaml
git commit -am "notes-api: Policy für Release-Kandidaten"
git push
flux reconcile kustomization flux-system --with-source
flux get image policy

Expected (shortened):

NAME            IMAGE                               TAG         READY   MESSAGE
notes-api       ghcr.io/thomaszachmann/notes-api    0.7.1       True    Latest image tag for ghcr.io/thomaszachmann/notes-api resolved to 0.7.1
notes-api-rc    ghcr.io/thomaszachmann/notes-api    0.6.0-rc.4  True    Latest image tag for ghcr.io/thomaszachmann/notes-api resolved to 0.6.0-rc.4
  • >=0.6.0-0 opens the range for pre-releases from 0.6.0 upward.
  • <0.7.0 excludes 0.7.0 and everything above it.
  • If a 0.6.0-rc.5 appears, the policy selects it at the next scan. If there were a final 0.6.0, it would be higher than any RC and would be selected.

The checkpoint above now shows two policies. You run the comparison with the companion code against tag10, where the policy is already included.

Save your progress? Sign up for free

Enterprise Note

  • Private registries: the ImageRepository gets a secretRef with Docker credentials or, in the cloud, provider: aws|azure|gcp. Then the controller obtains a short-lived token via workload identity.
  • Scan interval and rate limits: many ImageRepositories with a short interval quickly hit limits on Docker Hub. Ten minutes or more is common. For immediate updates, the registry sends a webhook to a Flux Receiver (Day 11).
  • Pinning digests: spec.digestReflectionPolicy: Always on the ImagePolicy provides the digest in addition to the tag. With that, the automation can write notes-api:0.7.1@sha256:…, and an overwritten tag cannot sneak in unnoticed.