Skip to content
Hands-on IT IT Drills

Day 2: The First App from Git

Goal: Roll out notes-api from the fleet repo and experience how Flux rolls back drift, picks up changes and cleans up what was deleted. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

  • From today on, the path of a change is always: edit the file, commit, push. You no longer need kubectl apply for notes-api.
  • For this, notes-api gets its own Kustomization. It points to the folder apps/notes in the fleet repo. The object itself lives in clusters/lab, which flux-system applies anyway.
  • Reconcile is the reconciliation between Git and the cluster. It runs as soon as a new commit arrives, and additionally at every interval. The interval run catches what someone changed by hand in the cluster.
 flux-fleet/
 ├── apps/
 │   └── notes/            ← what should run (manifests)
 └── clusters/lab/
     ├── flux-system/      ← Flux itself
     └── notes.yaml        ← Kustomization "notes": apply apps/notes

Drill 1: Manifests into the repo

The files from Day 0 move into the fleet repo:

cd ~/flux-lab/fleet
mkdir -p apps
mv ~/flux-lab/notes apps/notes
ls apps/notes

Expected:

deployment.yaml  kustomization.yaml  namespace.yaml  secret.yaml  service.yaml

You do not have to type the Flux Kustomization by hand. flux create builds it, and --export writes it out instead of creating it in the cluster:

flux create kustomization notes --source=GitRepository/flux-system \
  --path=./apps/notes --prune=true --interval=5m --export > clusters/lab/notes.yaml
cat clusters/lab/notes.yaml

Expected:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: notes
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./apps/notes
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  • sourceRef points to the same source as flux-system, that is, to the fleet repo. You do not need a second source.
  • path is relative to the root folder of the repo.
  • With --export, Git remains the single source of truth. A flux create without --export only creates the object in the cluster, and at the next reconciliation of flux-system it would stay there unnoticed.

Save your progress? Sign up for free

Drill 2: Commit and roll out

git add -A
git commit -m "notes-api aus Git"
git push

Expected (shortened):

[main 652cb0c] notes-api aus Git
 6 files changed, 66 insertions(+)
...
To https://github.com/<du>/flux-fleet.git
   3fb103a..652cb0c  main -> main

Within a minute, Flux fetches the commit. Impatient? Then trigger the reconciliation yourself:

flux reconcile kustomization flux-system --with-source
flux get kustomizations

Expected (shortened):

✔ applied revision main@sha1:652cb0c1a76befdab2c3a9c9778668451402ef6c
NAME        REVISION            SUSPENDED   READY   MESSAGE
flux-system main@sha1:652cb0c1  False       True    Applied revision: main@sha1:652cb0c1
notes       main@sha1:652cb0c1  False       True    Applied revision: main@sha1:652cb0c1

Two Kustomizations, one commit. The order: flux-system applies clusters/lab/notes.yaml, which creates the Kustomization notes, and that one applies apps/notes.

kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/version

Expected:

deployment "notes" successfully rolled out
0.5.1

Save your progress? Sign up for free

Drill 3: Rolling back drift

The hurried colleague from Day 0 is back:

kubectl -n notes set image deploy/notes api=ghcr.io/thomaszachmann/notes-api:0.7.1
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/version

Expected (shortened):

deployment.apps/notes image updated
...
0.7.1

This time it does not stay that way. After five minutes at the latest (interval: 5m), Flux reconciles. You bring it forward:

flux reconcile kustomization notes
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/version

Expected (shortened):

✔ applied revision main@sha1:652cb0c1a76befdab2c3a9c9778668451402ef6c
...
0.5.1
flux events --for Kustomization/notes

Expected (shortened):

5s   Normal  Progressing             Kustomization/notes  Deployment/notes/notes configured
5s   Normal  ReconciliationSucceeded Kustomization/notes  Reconciliation finished in 899.497735ms, next run in 5m0s

configured means: Flux reset something on the Deployment. In a reconciliation without drift, this line is missing.

The same applies to every field that is in Git, including the number of replicas:

kubectl -n notes scale deploy notes --replicas=3
flux reconcile kustomization notes
kubectl -n notes get deploy notes

Expected (shortened):

deployment.apps/notes scaled
...
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/1     1            1           4m

Save your progress? Sign up for free

Drill 4: Changing through Git

This is what an update looks like from today on:

sed -i 's/notes-api:0.5.1/notes-api:0.5.3/' apps/notes/deployment.yaml
git commit -am "notes-api 0.5.3"
git push
flux reconcile kustomization notes --with-source
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/version

Expected (shortened):

✔ applied revision main@sha1:e6bec635d8c91f12015ea043a19ddb4e1a88927f
...
0.5.3

git log is now the log of your changes to the cluster, with author, time and reason.

Save your progress? Sign up for free

Drill 5: Prune

Whatever disappears from Git, Flux also removes from the cluster. Take out the Service:

git rm -q apps/notes/service.yaml
sed -i '/service.yaml/d' apps/notes/kustomization.yaml
git commit -am "notes: Service entfernt"
git push
flux reconcile kustomization notes --with-source
kubectl -n notes get svc

Expected (shortened):

✔ applied revision main@sha1:71ba4957ccea1ff66aa0010a38f4db6a64c9913a
No resources found in notes namespace.
flux events --for Kustomization/notes | grep deleted

Expected:

8s   Normal  Progressing  Kustomization/notes  Service/notes/notes deleted

You bring the Service back with the tool Git has ready for every mistake:

git revert --no-edit HEAD
git push
flux reconcile kustomization notes --with-source
kubectl -n notes get svc

Expected (shortened):

[main 6a2ee93] Revert "notes: Service entfernt"
...
NAME    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
notes   NodePort   10.43.30.162   <none>        80:30080/TCP   3s

With GitOps, a rollback is not a separate tool but a commit.

Save your progress? Sign up for free

Checkpoint

flux get kustomizations

Expected: flux-system and notes, both READY True, with the same revision.

curl -s localhost:30080/version

Expected: 0.5.3.

Challenge

A colleague wants to set notes-api to two replicas and makes a typo:

sed -i 's/replicas: 1/replicas: zwei/' apps/notes/deployment.yaml
git commit -am "notes: zwei Replicas"
git push

What happens to the running notes-api? Find the error message and repair the state.

Hint: flux get has a MESSAGE column. And before Flux applies anything, it does a trial run.

Solution

flux reconcile kustomization flux-system --with-source
flux get kustomization notes

Expected (shortened):

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   main@sha1:6a2ee938  False       False   Deployment/notes/notes dry-run failed: failed to create typed patch object (notes/notes; apps/v1, Kind=Deployment): .spec.replicas: expected numeric (int or float), got string
  • READY is False, but REVISION still shows the old commit. Flux checks the whole folder with a dry run against the API server before it applies anything. If a single object fails, it applies nothing at all.
  • So notes-api keeps running unchanged:
kubectl -n notes get deploy notes

Expected:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/1     1            1           4m30s

As always, the repair goes through Git:

sed -i 's/replicas: zwei/replicas: 2/' apps/notes/deployment.yaml
git commit -am "notes: zwei Replicas, diesmal als Zahl"
git push
flux reconcile kustomization notes --with-source
kubectl -n notes get deploy notes

Expected (shortened):

✔ applied revision main@sha1:eb05203916fbb52802e0b86886d26b36c43afa23
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   2/2     2            2           5m36s
  • By the way, a flux reconcile on the broken Kustomization notes would have waited until it is ready, that is, until the five-minute timeout expired. That is why the solution triggers flux-system: it fetches the commit, and notes follows on its own.

The state after the challenge is in the companion code:

diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag02/apps

Expected: no output.

Save your progress? Sign up for free

Enterprise Note

  • Replicas and autoscalers: Flux resets every field that is in Git. If a HorizontalPodAutoscaler scales the Deployment, leave replicas out of Git. Otherwise the HPA and Flux fight at every reconciliation.
  • Emergency intervention: if someone has to intervene by hand during an incident, flux suspend kustomization notes pauses the reconciliation, flux resume resumes it (Day 4). Without suspend, Flux rolls back the intervention after the next interval.
  • Branch protection: whoever may push to main may change the cluster. In operations you protect main and allow changes only through pull requests with review.