Skip to content
Hands-on IT IT Drills

Day 10: Image Automation II – Flux Commits

Goal: Let Flux write the selected version into the fleet repo itself, first directly to main, then as a branch for a pull request, and know the pitfalls along the way. · Duration: approx. 60 min · Prerequisite: Day 9

Briefing

  • The ImageUpdateAutomation clones the fleet repo, searches a folder for setter markers, replaces the values behind them with the result of an ImagePolicy, commits and pushes.
  • A setter marker is a YAML comment at the end of a line:
  NOTES_VERSION: 0.7.0 # {"$imagepolicy": "flux-system:notes-api:tag"}
                         └ Namespace ─┘ └ Policy ┘ └ part ┘
  • The last part says what gets inserted: tag (only 0.7.1), name (only the image without the tag) or nothing (the entire image with tag).
  • The marker works in any YAML file: in Deployments, in HelmReleases under values and, as here, in the ConfigMap with the variables from Day 4.
 ImagePolicy notes-api: 0.7.1
        │
 ImageUpdateAutomation ── clones ── flux-fleet (main)
        │ replaces 0.7.0 → 0.7.1 in clusters/lab/cluster-vars.yaml
        └── commit + push ──► main  or  flux/notes-updates (pull request)
                                │
                     GitRepository flux-system ─► Kustomization notes ─► 0.7.1

Drill 1: The marker

So that there is something to do in a moment, you set prod back to 0.7.0 and put the marker behind it. The single quotes around the sed expression protect the dollar sign:

cd ~/flux-lab/fleet
sed -i 's/  NOTES_VERSION: 0.7.1/  NOTES_VERSION: 0.7.0 # {"$imagepolicy": "flux-system:notes-api:tag"}/' \
  clusters/lab/cluster-vars.yaml
cat clusters/lab/cluster-vars.yaml

Expected:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-vars
  namespace: flux-system
data:
  CLUSTER_NAME: lab
  NOTES_VERSION: 0.7.0 # {"$imagepolicy": "flux-system:notes-api:tag"}
git commit -am "notes: 0.7.0 mit Setter-Markierung"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s localhost:30080/version; echo

Expected (shortened):

...
0.7.0

Save your progress? Sign up for free

Drill 2: The automation

flux create image update notes --git-repo-ref=flux-system \
  --git-repo-path=./clusters/lab --checkout-branch=main --push-branch=main \
  --author-name=fluxcdbot --author-email=[email protected] \
  --commit-template='{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}}' \
  --interval=5m --export > clusters/lab/notes-automation.yaml
cat clusters/lab/notes-automation.yaml

Expected:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
  name: notes
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: [email protected]
        name: fluxcdbot
      messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println
        .NewValue}}{{end}}'
    push:
      branch: main
  interval: 5m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  update:
    path: ./clusters/lab
    strategy: Setters
  • update.path limits the search for markers to clusters/lab.
  • The messageTemplate is a Go template. .Changed.Changes contains every single replacement with old and new value.
  • The automation pushes using the Secret of the GitRepository flux-system, that is, using the deploy key, which has been allowed to write since Day 9.
git add -A
git commit -m "notes: Image-Updates automatisch"
git push
flux reconcile kustomization flux-system --with-source

A few seconds later:

flux events --for ImageUpdateAutomation/notes

Expected (shortened):

Normal  Succeeded  ImageUpdateAutomation/notes  pushed commit 'a414d2c' to branch 'main'
                                                0.7.0 -> 0.7.1
git pull
git log --oneline -3
git log -1 --format='%an <%ae>%n%B'
grep NOTES_VERSION clusters/lab/cluster-vars.yaml

Expected (the hashes differ):

a414d2c 0.7.0 -> 0.7.1
8a38360 notes: Image-Updates automatisch
bcd5771 notes: 0.7.0 mit Setter-Markierung
fluxcdbot <[email protected]>
0.7.0 -> 0.7.1

  NOTES_VERSION: 0.7.1 # {"$imagepolicy": "flux-system:notes-api:tag"}

The commit by fluxcdbot takes the normal path: the GitRepository fetches it, the Kustomization applies it.

flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s localhost:30080/version; echo

Expected (shortened):

✔ applied revision main@sha1:a414d2ce60a83b10468091e592790c9d12723359
0.7.1

From now on: if notes-api 0.7.2 appears in ghcr, the ImageRepository finds the tag at the next scan, the policy selects it, the automation commits, and prod rolls out. Nobody has to do anything.

Save your progress? Sign up for free

Drill 3: Better via pull request

Directly to main is convenient, but bold for prod. The alternative: the automation pushes to its own branch, and a human merges.

The automation gets a different push branch. checkout stays main:

sed -i '/    push:/{n;s#branch: main#branch: flux/notes-updates#}' \
  clusters/lab/notes-automation.yaml
grep -n branch clusters/lab/notes-automation.yaml

Expected:

11:        branch: main
19:      branch: flux/notes-updates

Back to 0.7.0 again, so there is something to do:

sed -i 's/NOTES_VERSION: 0.7.1/NOTES_VERSION: 0.7.0/' clusters/lab/cluster-vars.yaml
git commit -am "notes: Updates per Pull Request"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
flux reconcile image update notes
git fetch
git branch -r

Expected (shortened):

...
  origin/HEAD -> origin/main
  origin/flux/notes-updates
  origin/main
git diff main origin/flux/notes-updates

Expected (shortened):

-  NOTES_VERSION: 0.7.0 # {"$imagepolicy": "flux-system:notes-api:tag"}
+  NOTES_VERSION: 0.7.1 # {"$imagepolicy": "flux-system:notes-api:tag"}

prod keeps running on 0.7.0 until the merge. On GitHub the repo now shows you a notice “flux/notes-updates had recent pushes” with the button Compare & pull request. That is the team workflow: open a pull request, review, merge. In the lab you merge directly:

git merge --no-edit origin/flux/notes-updates
git push
git push origin --delete flux/notes-updates
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s localhost:30080/version; echo

Expected (shortened):

Updating 005b0f8..3f4f7a7
Fast-forward
 clusters/lab/cluster-vars.yaml | 2 +-
...
0.7.1

Deleting the branch is not tidying up for its own sake. The challenge shows why.

Save your progress? Sign up for free

Checkpoint

flux get image update

Expected: notes, READY True, repository up-to-date.

git log --format='%an' | sort | uniq -c

Expected (the numbers differ): commits by Flux (bootstrap), Flux Lab (you) and fluxcdbot (automation).

diff ~/flux-lab/fleet/clusters/lab/notes-automation.yaml \
  ~/flux-lab/code/loesung/tag10/clusters/lab/notes-automation.yaml

Expected: no output.

Challenge

From now on, dev should always run the newest release candidate, through the policy notes-api-rc from Day 9. Put a marker in clusters/lab/notes-dev.yaml and bring the automation’s pull request into main.

Hint: In dev, substitute takes precedence over substituteFrom (Day 5). Start with 0.6.0-rc.1 so the automation has something to do.

Solution

sed -i 's/^    substituteFrom:/    substitute:\n      NOTES_VERSION: 0.6.0-rc.1 # {"$imagepolicy": "flux-system:notes-api-rc:tag"}\n    substituteFrom:/' \
  clusters/lab/notes-dev.yaml
tail -7 clusters/lab/notes-dev.yaml

Expected:

  wait: true
  postBuild:
    substitute:
      NOTES_VERSION: 0.6.0-rc.1 # {"$imagepolicy": "flux-system:notes-api-rc:tag"}
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
git commit -am "notes-dev: Release-Kandidaten"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile image update notes
git fetch -p
git log --oneline -1 origin/flux/notes-updates

Expected (shortened):

2dedf40 0.6.0-rc.1 -> 0.6.0-rc.4

If you had left the branch in place after the merge in Drill 3, nothing would appear here:

Tried it: If the push branch already exists, the automation builds on it and not on main. The old, long-merged branch does not know the new marker in notes-dev.yaml, so the automation reports repository up-to-date. Only after git push origin --delete flux/notes-updates does the new commit appear. On GitHub you enable Settings → General → Automatically delete head branches for this.

Merging:

git diff main origin/flux/notes-updates

Expected (shortened):

-      NOTES_VERSION: 0.6.0-rc.1 # {"$imagepolicy": "flux-system:notes-api-rc:tag"}
+      NOTES_VERSION: 0.6.0-rc.4 # {"$imagepolicy": "flux-system:notes-api-rc:tag"}
-      - kind: ConfigMap
-        name: cluster-vars
+    - kind: ConfigMap
+      name: cluster-vars

The automation rewrites the whole file and indents the list differently in the process. For YAML that is equivalent, but it clutters the diff. If you want peace, format lists right away the way the automation writes them.

git merge --no-edit origin/flux/notes-updates
git push
git push origin --delete flux/notes-updates
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-dev
curl -s localhost:30081/version; echo

Expected (shortened):

...
0.6.0-rc.4

Save your progress? Sign up for free

Enterprise Note

  • Opening pull requests automatically: Flux only pushes the branch. In many teams, the pull request is opened by a small GitHub Action that reacts to pushes to flux/**. Alternatively, Renovate takes over the image updates entirely, and Flux sticks to rolling out (see “Renovate hands-on”).
  • Many updates at once: if several images appear in quick succession, the automation waits a while with spec.policyChangeDelay (new in Flux 2.10) and bundles them into one commit.
  • Signed commits: spec.git.commit.signingKey signs the bot’s commits with GPG. Together with branch protection (“signed commits only”), everyone knows that a commit really comes from Flux.