Skip to content
Hands-on IT IT Drills

Day 13: CI and Operations

Goal: Have GitHub Actions check the fleet repo on every push, investigate, monitor and upgrade Flux in operations, and know how to get rid of it again. · Duration: approx. 75 min · Prerequisite: Day 12

Briefing

  • Flux checks manifests only in the cluster, via dry run. So a typo is only noticed once it is already on main. A CI checks beforehand, on every push and every pull request.
  • The check takes three steps: Kustomize builds every folder, flux envsubst substitutes variables, and kubeconform checks every object against the schema of its kind. For the Flux kinds, the Flux project ships the schemas.
  • In operations you ask three questions: where does this object come from? What is broken right now? How long does reconciliation take? The Flux CLI has a command for each.

Drill 1: What the CI should check

Before a workflow runs, you try the check on the VM:

cd /tmp
curl -fsSL https://github.com/yannh/kubeconform/releases/download/v0.8.0/kubeconform-linux-amd64.tar.gz \
  | tar xz -C ~/.local/bin kubeconform
mkdir -p /tmp/flux-schemas
curl -fsSL https://github.com/fluxcd/flux2/releases/download/v2.10.0/crd-schemas.tar.gz \
  | tar xz -C /tmp/flux-schemas
ls /tmp/flux-schemas | head -3

Expected:

_definitions.json
alert-notification-v1.json
alert-notification-v1beta3.json

kubeconform knows the Kubernetes kinds on its own (default). For everything else it gets a template under which it finds the schema:

cd ~/flux-lab/fleet
S='-schema-location default -schema-location /tmp/flux-schemas/{{.ResourceKind}}{{.KindSuffix}}.json'
kubectl kustomize clusters/lab | kubeconform -strict -summary -skip Secret $S

Expected (shortened):

stdin - CustomResourceDefinition alerts.notification.toolkit.fluxcd.io failed validation: could not find schema for CustomResourceDefinition
...
Summary: 64 resources found parsing stdin - Valid: 50, Invalid: 0, Errors: 14, Skipped: 0

The 14 errors are the CRDs from gotk-components.yaml. There is no schema for the kind CustomResourceDefinition, so you skip them like the Secrets. You leave Secrets out because SOPS extends them with a sops field that Kubernetes does not know.

kubectl kustomize clusters/lab \
  | kubeconform -strict -summary -skip Secret,CustomResourceDefinition $S

Expected:

Summary: 64 resources found parsing stdin - Valid: 50, Invalid: 0, Errors: 0, Skipped: 14

Now the notes overlays:

kubectl kustomize apps/notes/overlays/prod | kubeconform -strict -summary -skip Secret $S

Expected (shortened):

stdin - Deployment notes is invalid: … at '/spec/replicas': got string, want null or integer
Summary: 3 resources found parsing stdin - Valid: 1, Invalid: 1, Errors: 0, Skipped: 1

Correctly detected: the folder has replicas: ${NOTES_REPLICAS:=1}, a string. Only Flux inserts a number during reconciliation. flux envsubst does the same locally. For NOTES_VERSION there is no default value, so the CI gets a placeholder:

kubectl kustomize apps/notes/overlays/prod | NOTES_VERSION=0.0.0-ci flux envsubst --strict \
  | kubeconform -strict -summary -skip Secret $S

Expected:

Summary: 3 resources found parsing stdin - Valid: 2, Invalid: 0, Errors: 0, Skipped: 1

Without the placeholder, the line image: ghcr.io/thomaszachmann/notes-api: would have been left over, and the colon at the end turns that into broken YAML.

Save your progress? Sign up for free

Drill 2: The workflow

A token may only change workflow files if it has the permission Workflows: Read and write (Day 0). If it is missing, GitHub rejects the push, and you add it in the browser to the token flux-lab:

 ! [remote rejected] main -> main (refusing to allow a Personal Access Token to create or update workflow `.github/workflows/validate.yaml` without `workflow` scope)

The workflow is a file in the repo:

mkdir -p .github/workflows
cat > .github/workflows/validate.yaml <<'EOF'
name: validate
on:
  pull_request:
  push:
    branches: [main]
jobs:
  manifests:
    runs-on: ubuntu-24.04
    env:
      NOTES_VERSION: 0.0.0-ci
      SCHEMAS: -schema-location default -schema-location /tmp/flux-schemas/{{.ResourceKind}}{{.KindSuffix}}.json
    steps:
      - uses: actions/checkout@v5
      - uses: fluxcd/flux2/[email protected]
        with:
          version: 2.10.0
      - name: kubeconform und Flux-Schemas
        run: |
          curl -fsSL https://github.com/yannh/kubeconform/releases/download/v0.8.0/kubeconform-linux-amd64.tar.gz | tar xz -C /usr/local/bin kubeconform
          mkdir -p /tmp/flux-schemas
          curl -fsSL https://github.com/fluxcd/flux2/releases/download/v2.10.0/crd-schemas.tar.gz | tar xz -C /tmp/flux-schemas
      - name: Cluster lab
        run: kubectl kustomize clusters/lab | kubeconform -strict -summary -skip Secret,CustomResourceDefinition $SCHEMAS
      - name: Apps und Infrastruktur
        run: |
          for d in $(find apps infrastructure -name kustomization.yaml -exec dirname {} \;); do
            echo "== $d"
            kubectl kustomize $d | flux envsubst --strict | kubeconform -strict -summary -skip Secret $SCHEMAS
          done
      - name: Mandanten
        run: kubeconform -strict -summary $SCHEMAS tenants
EOF
git add -A
git commit -m "CI: Manifeste prüfen"
git push
  • fluxcd/flux2/action installs the Flux CLI in the desired version.
  • The loop finds every folder with a kustomization.yaml under apps and infrastructure, including new ones, without anyone having to adjust the workflow.
  • tenants has no kustomization.yaml, so kubeconform checks the files directly.

On GitHub, a validate run appears under flux-fleet → Actions. Without a browser you see the same with the GitHub CLI on your laptop, if you have it there:

gh run list -R <du>/flux-fleet -L 1

Expected (shortened):

completed   success CI: Manifeste prüfen    validate    main    push    43275465547 15s …

The log of the step “Apps und Infrastruktur” has one summary per folder:

== apps/notes/overlays/prod
Summary: 3 resources found parsing stdin - Valid: 2, Invalid: 0, Errors: 0, Skipped: 1
...
== infrastructure/registry
Summary: 3 resources found parsing stdin - Valid: 3, Invalid: 0, Errors: 0, Skipped: 0

About 15 seconds for the whole repo. A green check mark appears on the commit, next to the check marks of the Kustomizations from Day 11.

Save your progress? Sign up for free

Drill 3: Where does this come from?

A colleague asks: “Who actually manages the Deployment notes?”

flux trace deployment notes -n notes

Expected (shortened):

Object:          Deployment/notes
Namespace:       notes
Status:          Managed by Flux
---
Kustomization:   notes
Namespace:       flux-system
Path:            ./apps/notes/overlays/prod
Revision:        main@sha1:3c42d9b575c2de949e5d2b925296bb4ee7271798
Status:          Last reconciled at 2026-11-12 17:17:20 +0000 UTC
Message:         Applied revision: main@sha1:3c42d9b575c2de949e5d2b925296bb4ee7271798
---
GitRepository:   flux-system
Namespace:       flux-system
URL:             ssh://[email protected]/<du>/flux-fleet
Branch:          main
...

From the object through the Kustomization and the path to the repo and commit. flux tree goes the opposite way, from the Kustomization to its objects:

flux tree kustomization notes

Expected:

Kustomization/flux-system/notes
├── Secret/notes/notes-api
├── Service/notes/notes
└── Deployment/notes/notes

Save your progress? Sign up for free

Drill 4: What is broken?

The fastest overview of the whole cluster:

flux get all -A --status-selector ready=false

Expected: no output. Anything that appears here needs attention.

flux stats -A

Expected (shortened):

RECONCILERS             RUNNING FAILING SUSPENDED   STORAGE
GitRepository           1       0       0           57.6 KiB
OCIRepository           4       0       0           2.9 KiB
...
Kustomization           12      0       0           -
HelmRelease             1       0       0           -
...

The logs of one object, without digging through the logs of the whole controller:

flux logs --kind=Kustomization --name=notes --since=10m | tail -3

Expected (shortened):

… info Kustomization/notes.flux-system - All dependencies are ready, proceeding with reconciliation
… info Kustomization/notes.flux-system - server-side apply completed
… info Kustomization/notes.flux-system - Reconciliation finished in 1.023620861s, next run in 5m0s

Save your progress? Sign up for free

Drill 5: Metrics

Every controller provides Prometheus metrics on port 8080. Without Prometheus you can fetch them through the API server:

P=$(kubectl -n flux-system get pod -l app=kustomize-controller \
  -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw /api/v1/namespaces/flux-system/pods/$P:8080/proxy/metrics \
  | grep '^gotk_reconcile_duration_seconds_count' | head -3

Expected (shortened):

gotk_reconcile_duration_seconds_count{kind="Kustomization",name="apps",namespace="team-blau"} 4
gotk_reconcile_duration_seconds_count{kind="Kustomization",name="apps",namespace="team-gruen"} 3
...

gotk_reconcile_duration_seconds shows how often and how long each object is reconciled. If the duration rises, a repo is getting too large or a health check is hanging.

Save your progress? Sign up for free

Drill 6: Upgrade

A Flux upgrade is a bootstrap with a newer CLI. You already have the newest one, so you check that a bootstrap breaks nothing, above all not the patches from Day 12:

flux bootstrap github --owner=$GITHUB_USER --repository=flux-fleet \
  --branch=main --path=clusters/lab --personal --private=true \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key

Expected (shortened):

✔ component manifests are up to date
...
✔ source secret up to date
...
✔ sync manifests are up to date
git pull
grep -c 'op: add' clusters/lab/flux-system/kustomization.yaml

Expected:

3

The bootstrap only rewrites gotk-components.yaml and gotk-sync.yaml. The kustomization.yaml next to them with your patches stays as it is.

For a real upgrade, you download the new Flux version as on Day 0, replace ~/.local/bin/flux and run the same command. Read the release notes beforehand. Flux 2.10, for example, promoted the notification APIs to v1.

Save your progress? Sign up for free

Drill 7: Uninstalling

What happens if you remove Flux? A trial run shows it:

flux uninstall --dry-run --silent | grep -v 'deleted (dry run)'

Expected:

► deleting components in flux-system namespace
► deleting toolkit.fluxcd.io finalizers in all namespaces
► deleting toolkit.fluxcd.io custom resource definitions
✔ uninstall finished

The order is crucial: first Flux removes the finalizers, then the CRDs. Without finalizers, no Kustomization removes its objects when it is deleted. So notes, Postgres and the teams keep running; it is just that nobody manages them anymore.

Save your progress? Sign up for free

Checkpoint

flux get all -A --status-selector ready=false

Expected: no output.

diff ~/flux-lab/fleet/.github/workflows/validate.yaml \
  ~/flux-lab/code/loesung/tag13/.github/workflows/validate.yaml

Expected: no output.

Challenge

A colleague pushes a typo: replicas: zwei in the notes-db Deployment. Who notices first, the CI or Flux? And what do you see on GitHub?

Solution

cd ~/flux-lab/fleet
sed -i 's/replicas: 1/replicas: zwei/' apps/notes-db/base/deployment.yaml
git commit -am "notes-db: zwei Replicas"
git push

After about 15 seconds the validate run is red. In the log of the step “Apps und Infrastruktur”:

== apps/notes-db/base
stdin - Deployment notes-db is invalid: problem validating schema. … at '/spec/replicas': got string, want null or integer
##[error]Process completed with exit code 1.

Within a minute, Flux reports the error too:

flux get kustomization notes-db

Expected (shortened):

NAME        REVISION            SUSPENDED   READY   MESSAGE
notes-db    main@sha1:9a6e2ef2  False       False   Deployment/notes/notes-db dry-run failed: … .spec.replicas: expected numeric (int or float), got string
  • The CI was faster, but both found the error, and nothing broke. Flux checks via dry run and applies nothing.
  • The real gain comes with pull requests: the CI checks before the merge. With branch protection and the rule “status validate must be green”, such a commit never even reaches main.
git revert --no-edit HEAD
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db

Save your progress? Sign up for free

Enterprise Note

  • flux diff in the pull request: with access to a cluster, flux diff kustomization shows in the pull request what would change. For this the CI needs read access to the cluster. Many teams give it its own kubeconfig with read-only permissions.
  • Monitoring: the Flux project maintains an example (flux2-monitoring-example) with Prometheus, kube-state-metrics and ready-made Grafana dashboards. From it comes the metric gotk_resource_info with the state of every Flux object, the best foundation for alerts.
  • Flux Operator: instead of flux bootstrap, the Flux Operator manages the installation through a single FluxInstance object, with automatic upgrades within a version series. For many clusters this is the simpler way.