Day 13: CI and Operations
Goal: Have GitHub Actions check the fleet repo on every push, investigate, monitor and upgrade Flux in operations, and know how to get rid of it again. · Duration: approx. 75 min · Prerequisite: Day 12
Briefing
- Flux checks manifests only in the cluster, via dry run. So a typo is
only noticed once it is already on
main. A CI checks beforehand, on every push and every pull request. - The check takes three steps: Kustomize builds every folder,
flux envsubstsubstitutes variables, and kubeconform checks every object against the schema of its kind. For the Flux kinds, the Flux project ships the schemas. - In operations you ask three questions: where does this object come from? What is broken right now? How long does reconciliation take? The Flux CLI has a command for each.
Drill 1: What the CI should check
Before a workflow runs, you try the check on the VM:
cd /tmp
curl -fsSL https://github.com/yannh/kubeconform/releases/download/v0.8.0/kubeconform-linux-amd64.tar.gz \
| tar xz -C ~/.local/bin kubeconform
mkdir -p /tmp/flux-schemas
curl -fsSL https://github.com/fluxcd/flux2/releases/download/v2.10.0/crd-schemas.tar.gz \
| tar xz -C /tmp/flux-schemas
ls /tmp/flux-schemas | head -3Expected:
_definitions.json
alert-notification-v1.json
alert-notification-v1beta3.json
kubeconform knows the Kubernetes kinds on its own
(default). For everything else it gets a template under
which it finds the schema:
cd ~/flux-lab/fleet
S='-schema-location default -schema-location /tmp/flux-schemas/{{.ResourceKind}}{{.KindSuffix}}.json'
kubectl kustomize clusters/lab | kubeconform -strict -summary -skip Secret $SExpected (shortened):
stdin - CustomResourceDefinition alerts.notification.toolkit.fluxcd.io failed validation: could not find schema for CustomResourceDefinition
...
Summary: 64 resources found parsing stdin - Valid: 50, Invalid: 0, Errors: 14, Skipped: 0
The 14 errors are the CRDs from gotk-components.yaml.
There is no schema for the kind CustomResourceDefinition,
so you skip them like the Secrets. You leave Secrets out because SOPS
extends them with a sops field that Kubernetes does not
know.
kubectl kustomize clusters/lab \
| kubeconform -strict -summary -skip Secret,CustomResourceDefinition $SExpected:
Summary: 64 resources found parsing stdin - Valid: 50, Invalid: 0, Errors: 0, Skipped: 14
Now the notes overlays:
kubectl kustomize apps/notes/overlays/prod | kubeconform -strict -summary -skip Secret $SExpected (shortened):
stdin - Deployment notes is invalid: … at '/spec/replicas': got string, want null or integer
Summary: 3 resources found parsing stdin - Valid: 1, Invalid: 1, Errors: 0, Skipped: 1
Correctly detected: the folder has
replicas: ${NOTES_REPLICAS:=1}, a string. Only Flux inserts
a number during reconciliation. flux envsubst does the same
locally. For NOTES_VERSION there is no default value, so
the CI gets a placeholder:
kubectl kustomize apps/notes/overlays/prod | NOTES_VERSION=0.0.0-ci flux envsubst --strict \
| kubeconform -strict -summary -skip Secret $SExpected:
Summary: 3 resources found parsing stdin - Valid: 2, Invalid: 0, Errors: 0, Skipped: 1
Without the placeholder, the line
image: ghcr.io/thomaszachmann/notes-api: would have been
left over, and the colon at the end turns that into broken YAML.
Save your progress? Sign up for free
Drill 2: The workflow
A token may only change workflow files if it has the permission
Workflows: Read and write (Day 0). If it is missing,
GitHub rejects the push, and you add it in the browser to the token
flux-lab:
! [remote rejected] main -> main (refusing to allow a Personal Access Token to create or update workflow `.github/workflows/validate.yaml` without `workflow` scope)
The workflow is a file in the repo:
mkdir -p .github/workflows
cat > .github/workflows/validate.yaml <<'EOF'
name: validate
on:
pull_request:
push:
branches: [main]
jobs:
manifests:
runs-on: ubuntu-24.04
env:
NOTES_VERSION: 0.0.0-ci
SCHEMAS: -schema-location default -schema-location /tmp/flux-schemas/{{.ResourceKind}}{{.KindSuffix}}.json
steps:
- uses: actions/checkout@v5
- uses: fluxcd/flux2/[email protected]
with:
version: 2.10.0
- name: kubeconform und Flux-Schemas
run: |
curl -fsSL https://github.com/yannh/kubeconform/releases/download/v0.8.0/kubeconform-linux-amd64.tar.gz | tar xz -C /usr/local/bin kubeconform
mkdir -p /tmp/flux-schemas
curl -fsSL https://github.com/fluxcd/flux2/releases/download/v2.10.0/crd-schemas.tar.gz | tar xz -C /tmp/flux-schemas
- name: Cluster lab
run: kubectl kustomize clusters/lab | kubeconform -strict -summary -skip Secret,CustomResourceDefinition $SCHEMAS
- name: Apps und Infrastruktur
run: |
for d in $(find apps infrastructure -name kustomization.yaml -exec dirname {} \;); do
echo "== $d"
kubectl kustomize $d | flux envsubst --strict | kubeconform -strict -summary -skip Secret $SCHEMAS
done
- name: Mandanten
run: kubeconform -strict -summary $SCHEMAS tenants
EOF
git add -A
git commit -m "CI: Manifeste prüfen"
git pushfluxcd/flux2/actioninstalls the Flux CLI in the desired version.- The loop finds every folder with a
kustomization.yamlunderappsandinfrastructure, including new ones, without anyone having to adjust the workflow. tenantshas nokustomization.yaml, so kubeconform checks the files directly.
On GitHub, a validate run appears under
flux-fleet → Actions. Without a browser you see the
same with the GitHub CLI on your laptop, if you have it there:
gh run list -R <du>/flux-fleet -L 1
Expected (shortened):
completed success CI: Manifeste prüfen validate main push 43275465547 15s …
The log of the step “Apps und Infrastruktur” has one summary per folder:
== apps/notes/overlays/prod
Summary: 3 resources found parsing stdin - Valid: 2, Invalid: 0, Errors: 0, Skipped: 1
...
== infrastructure/registry
Summary: 3 resources found parsing stdin - Valid: 3, Invalid: 0, Errors: 0, Skipped: 0
About 15 seconds for the whole repo. A green check mark appears on the commit, next to the check marks of the Kustomizations from Day 11.
Save your progress? Sign up for free
Drill 3: Where does this come from?
A colleague asks: “Who actually manages the Deployment
notes?”
flux trace deployment notes -n notesExpected (shortened):
Object: Deployment/notes
Namespace: notes
Status: Managed by Flux
---
Kustomization: notes
Namespace: flux-system
Path: ./apps/notes/overlays/prod
Revision: main@sha1:3c42d9b575c2de949e5d2b925296bb4ee7271798
Status: Last reconciled at 2026-11-12 17:17:20 +0000 UTC
Message: Applied revision: main@sha1:3c42d9b575c2de949e5d2b925296bb4ee7271798
---
GitRepository: flux-system
Namespace: flux-system
URL: ssh://[email protected]/<du>/flux-fleet
Branch: main
...
From the object through the Kustomization and the path to the repo
and commit. flux tree goes the opposite way, from the
Kustomization to its objects:
flux tree kustomization notesExpected:
Kustomization/flux-system/notes
├── Secret/notes/notes-api
├── Service/notes/notes
└── Deployment/notes/notes
Save your progress? Sign up for free
Drill 4: What is broken?
The fastest overview of the whole cluster:
flux get all -A --status-selector ready=falseExpected: no output. Anything that appears here needs attention.
flux stats -AExpected (shortened):
RECONCILERS RUNNING FAILING SUSPENDED STORAGE
GitRepository 1 0 0 57.6 KiB
OCIRepository 4 0 0 2.9 KiB
...
Kustomization 12 0 0 -
HelmRelease 1 0 0 -
...
The logs of one object, without digging through the logs of the whole controller:
flux logs --kind=Kustomization --name=notes --since=10m | tail -3Expected (shortened):
… info Kustomization/notes.flux-system - All dependencies are ready, proceeding with reconciliation
… info Kustomization/notes.flux-system - server-side apply completed
… info Kustomization/notes.flux-system - Reconciliation finished in 1.023620861s, next run in 5m0s
Save your progress? Sign up for free
Drill 5: Metrics
Every controller provides Prometheus metrics on port 8080. Without Prometheus you can fetch them through the API server:
P=$(kubectl -n flux-system get pod -l app=kustomize-controller \
-o jsonpath='{.items[0].metadata.name}')
kubectl get --raw /api/v1/namespaces/flux-system/pods/$P:8080/proxy/metrics \
| grep '^gotk_reconcile_duration_seconds_count' | head -3Expected (shortened):
gotk_reconcile_duration_seconds_count{kind="Kustomization",name="apps",namespace="team-blau"} 4
gotk_reconcile_duration_seconds_count{kind="Kustomization",name="apps",namespace="team-gruen"} 3
...
gotk_reconcile_duration_seconds shows how often and how
long each object is reconciled. If the duration rises, a repo is getting
too large or a health check is hanging.
Save your progress? Sign up for free
Drill 6: Upgrade
A Flux upgrade is a bootstrap with a newer CLI. You already have the newest one, so you check that a bootstrap breaks nothing, above all not the patches from Day 12:
flux bootstrap github --owner=$GITHUB_USER --repository=flux-fleet \
--branch=main --path=clusters/lab --personal --private=true \
--components-extra=image-reflector-controller,image-automation-controller \
--read-write-keyExpected (shortened):
✔ component manifests are up to date
...
✔ source secret up to date
...
✔ sync manifests are up to date
git pull
grep -c 'op: add' clusters/lab/flux-system/kustomization.yamlExpected:
3
The bootstrap only rewrites gotk-components.yaml and
gotk-sync.yaml. The kustomization.yaml next to
them with your patches stays as it is.
For a real upgrade, you download the new Flux version as on Day 0,
replace ~/.local/bin/flux and run the same command. Read
the release notes beforehand. Flux 2.10, for example, promoted the
notification APIs to v1.
Save your progress? Sign up for free
Drill 7: Uninstalling
What happens if you remove Flux? A trial run shows it:
flux uninstall --dry-run --silent | grep -v 'deleted (dry run)'Expected:
► deleting components in flux-system namespace
► deleting toolkit.fluxcd.io finalizers in all namespaces
► deleting toolkit.fluxcd.io custom resource definitions
✔ uninstall finished
The order is crucial: first Flux removes the finalizers, then the CRDs. Without finalizers, no Kustomization removes its objects when it is deleted. So notes, Postgres and the teams keep running; it is just that nobody manages them anymore.
Save your progress? Sign up for free
Checkpoint
flux get all -A --status-selector ready=falseExpected: no output.
diff ~/flux-lab/fleet/.github/workflows/validate.yaml \
~/flux-lab/code/loesung/tag13/.github/workflows/validate.yamlExpected: no output.
Challenge
A colleague pushes a typo: replicas: zwei in the
notes-db Deployment. Who notices first, the CI or Flux? And what do you
see on GitHub?
Solution
cd ~/flux-lab/fleet
sed -i 's/replicas: 1/replicas: zwei/' apps/notes-db/base/deployment.yaml
git commit -am "notes-db: zwei Replicas"
git pushAfter about 15 seconds the validate run is red. In the
log of the step “Apps und Infrastruktur”:
== apps/notes-db/base
stdin - Deployment notes-db is invalid: problem validating schema. … at '/spec/replicas': got string, want null or integer
##[error]Process completed with exit code 1.
Within a minute, Flux reports the error too:
flux get kustomization notes-dbExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
notes-db main@sha1:9a6e2ef2 False False Deployment/notes/notes-db dry-run failed: … .spec.replicas: expected numeric (int or float), got string
- The CI was faster, but both found the error, and nothing broke. Flux checks via dry run and applies nothing.
- The real gain comes with pull requests: the CI checks
before the merge. With branch protection and the rule “status
validatemust be green”, such a commit never even reachesmain.
git revert --no-edit HEAD
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-dbSave your progress? Sign up for free
Enterprise Note
flux diffin the pull request: with access to a cluster,flux diff kustomizationshows in the pull request what would change. For this the CI needs read access to the cluster. Many teams give it its own kubeconfig with read-only permissions.- Monitoring: the Flux project maintains an example
(
flux2-monitoring-example) with Prometheus, kube-state-metrics and ready-made Grafana dashboards. From it comes the metricgotk_resource_infowith the state of every Flux object, the best foundation for alerts. - Flux Operator: instead of
flux bootstrap, the Flux Operator manages the installation through a singleFluxInstanceobject, with automatic upgrades within a version series. For many clusters this is the simpler way.