Day 2: The First App from Git
Goal: Roll out notes-api from the fleet repo and experience how Flux rolls back drift, picks up changes and cleans up what was deleted. · Duration: approx. 60 min · Prerequisite: Day 1
Briefing
- From today on, the path of a change is always: edit the
file, commit, push. You no longer need
kubectl applyfor notes-api. - For this, notes-api gets its own Kustomization. It
points to the folder
apps/notesin the fleet repo. The object itself lives inclusters/lab, whichflux-systemapplies anyway. - Reconcile is the reconciliation between Git and the cluster. It runs as soon as a new commit arrives, and additionally at every interval. The interval run catches what someone changed by hand in the cluster.
flux-fleet/
├── apps/
│ └── notes/ ← what should run (manifests)
└── clusters/lab/
├── flux-system/ ← Flux itself
└── notes.yaml ← Kustomization "notes": apply apps/notes
Drill 1: Manifests into the repo
The files from Day 0 move into the fleet repo:
cd ~/flux-lab/fleet
mkdir -p apps
mv ~/flux-lab/notes apps/notes
ls apps/notesExpected:
deployment.yaml kustomization.yaml namespace.yaml secret.yaml service.yaml
You do not have to type the Flux Kustomization by hand.
flux create builds it, and --export writes it
out instead of creating it in the cluster:
flux create kustomization notes --source=GitRepository/flux-system \
--path=./apps/notes --prune=true --interval=5m --export > clusters/lab/notes.yaml
cat clusters/lab/notes.yamlExpected:
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: notes
namespace: flux-system
spec:
interval: 5m0s
path: ./apps/notes
prune: true
sourceRef:
kind: GitRepository
name: flux-system
sourceRefpoints to the same source asflux-system, that is, to the fleet repo. You do not need a second source.pathis relative to the root folder of the repo.- With
--export, Git remains the single source of truth. Aflux createwithout--exportonly creates the object in the cluster, and at the next reconciliation offlux-systemit would stay there unnoticed.
Save your progress? Sign up for free
Drill 2: Commit and roll out
git add -A
git commit -m "notes-api aus Git"
git pushExpected (shortened):
[main 652cb0c] notes-api aus Git
6 files changed, 66 insertions(+)
...
To https://github.com/<du>/flux-fleet.git
3fb103a..652cb0c main -> main
Within a minute, Flux fetches the commit. Impatient? Then trigger the reconciliation yourself:
flux reconcile kustomization flux-system --with-source
flux get kustomizationsExpected (shortened):
✔ applied revision main@sha1:652cb0c1a76befdab2c3a9c9778668451402ef6c
NAME REVISION SUSPENDED READY MESSAGE
flux-system main@sha1:652cb0c1 False True Applied revision: main@sha1:652cb0c1
notes main@sha1:652cb0c1 False True Applied revision: main@sha1:652cb0c1
Two Kustomizations, one commit. The order: flux-system
applies clusters/lab/notes.yaml, which creates the
Kustomization notes, and that one applies
apps/notes.
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/versionExpected:
deployment "notes" successfully rolled out
0.5.1
Save your progress? Sign up for free
Drill 3: Rolling back drift
The hurried colleague from Day 0 is back:
kubectl -n notes set image deploy/notes api=ghcr.io/thomaszachmann/notes-api:0.7.1
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/versionExpected (shortened):
deployment.apps/notes image updated
...
0.7.1
This time it does not stay that way. After five minutes at the latest
(interval: 5m), Flux reconciles. You bring it forward:
flux reconcile kustomization notes
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/versionExpected (shortened):
✔ applied revision main@sha1:652cb0c1a76befdab2c3a9c9778668451402ef6c
...
0.5.1
flux events --for Kustomization/notesExpected (shortened):
5s Normal Progressing Kustomization/notes Deployment/notes/notes configured
5s Normal ReconciliationSucceeded Kustomization/notes Reconciliation finished in 899.497735ms, next run in 5m0s
configured means: Flux reset something on the
Deployment. In a reconciliation without drift, this line is missing.
The same applies to every field that is in Git, including the number of replicas:
kubectl -n notes scale deploy notes --replicas=3
flux reconcile kustomization notes
kubectl -n notes get deploy notesExpected (shortened):
deployment.apps/notes scaled
...
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 4m
Save your progress? Sign up for free
Drill 4: Changing through Git
This is what an update looks like from today on:
sed -i 's/notes-api:0.5.1/notes-api:0.5.3/' apps/notes/deployment.yaml
git commit -am "notes-api 0.5.3"
git push
flux reconcile kustomization notes --with-source
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/versionExpected (shortened):
✔ applied revision main@sha1:e6bec635d8c91f12015ea043a19ddb4e1a88927f
...
0.5.3
git log is now the log of your changes to the cluster,
with author, time and reason.
Save your progress? Sign up for free
Drill 5: Prune
Whatever disappears from Git, Flux also removes from the cluster. Take out the Service:
git rm -q apps/notes/service.yaml
sed -i '/service.yaml/d' apps/notes/kustomization.yaml
git commit -am "notes: Service entfernt"
git push
flux reconcile kustomization notes --with-source
kubectl -n notes get svcExpected (shortened):
✔ applied revision main@sha1:71ba4957ccea1ff66aa0010a38f4db6a64c9913a
No resources found in notes namespace.
flux events --for Kustomization/notes | grep deletedExpected:
8s Normal Progressing Kustomization/notes Service/notes/notes deleted
You bring the Service back with the tool Git has ready for every mistake:
git revert --no-edit HEAD
git push
flux reconcile kustomization notes --with-source
kubectl -n notes get svcExpected (shortened):
[main 6a2ee93] Revert "notes: Service entfernt"
...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
notes NodePort 10.43.30.162 <none> 80:30080/TCP 3s
With GitOps, a rollback is not a separate tool but a commit.
Save your progress? Sign up for free
Checkpoint
flux get kustomizationsExpected: flux-system and notes, both
READY True, with the same revision.
curl -s localhost:30080/versionExpected: 0.5.3.
Challenge
A colleague wants to set notes-api to two replicas and makes a typo:
sed -i 's/replicas: 1/replicas: zwei/' apps/notes/deployment.yaml
git commit -am "notes: zwei Replicas"
git pushWhat happens to the running notes-api? Find the error message and repair the state.
Hint: flux get has a
MESSAGE column. And before Flux applies anything, it does a
trial run.
Solution
flux reconcile kustomization flux-system --with-source
flux get kustomization notesExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
notes main@sha1:6a2ee938 False False Deployment/notes/notes dry-run failed: failed to create typed patch object (notes/notes; apps/v1, Kind=Deployment): .spec.replicas: expected numeric (int or float), got string
READYisFalse, butREVISIONstill shows the old commit. Flux checks the whole folder with a dry run against the API server before it applies anything. If a single object fails, it applies nothing at all.- So notes-api keeps running unchanged:
kubectl -n notes get deploy notesExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 4m30s
As always, the repair goes through Git:
sed -i 's/replicas: zwei/replicas: 2/' apps/notes/deployment.yaml
git commit -am "notes: zwei Replicas, diesmal als Zahl"
git push
flux reconcile kustomization notes --with-source
kubectl -n notes get deploy notesExpected (shortened):
✔ applied revision main@sha1:eb05203916fbb52802e0b86886d26b36c43afa23
NAME READY UP-TO-DATE AVAILABLE AGE
notes 2/2 2 2 5m36s
- By the way, a
flux reconcileon the broken Kustomizationnoteswould have waited until it is ready, that is, until the five-minute timeout expired. That is why the solution triggersflux-system: it fetches the commit, andnotesfollows on its own.
The state after the challenge is in the companion code:
diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag02/appsExpected: no output.
Save your progress? Sign up for free
Enterprise Note
- Replicas and autoscalers: Flux resets every field
that is in Git. If a HorizontalPodAutoscaler scales the Deployment,
leave
replicasout of Git. Otherwise the HPA and Flux fight at every reconciliation. - Emergency intervention: if someone has to intervene
by hand during an incident,
flux suspend kustomization notespauses the reconciliation,flux resumeresumes it (Day 4). Withoutsuspend, Flux rolls back the intervention after the next interval. - Branch protection: whoever may push to
mainmay change the cluster. In operations you protectmainand allow changes only through pull requests with review.