Skip to content
Hands-on IT IT Drills

Day 1: Bootstrap

Goal: Install Flux with a single command, create the fleet repo and understand why Flux manages itself from Git from now on. · Duration: approx. 45 min · Prerequisite: Day 0

Briefing

  • flux bootstrap github does four things at once: it creates the repo, writes the Flux manifests into it, installs Flux in the cluster and connects the two through a deploy key, an SSH key that is valid only for this one repo.
  • Flux consists of controllers. Each one takes care of one kind of object:
Controller Responsible for
source-controller Fetching sources: GitRepository, OCIRepository, HelmRepository, Bucket
kustomize-controller Applying manifests from a source: Kustomization
helm-controller Installing Helm charts: HelmRelease
notification-controller Sending and receiving notifications: Provider, Alert, Receiver
  • After the bootstrap there is exactly one source and one Kustomization, both named flux-system. They point to clusters/lab in the fleet repo, and that is also where the manifests of Flux itself live. So Flux manages itself from Git.
 GitHub: flux-fleet ─────────────────────────────────┐
  └─ clusters/lab/                                   │ fetch every minute
      └─ flux-system/  (Flux itself)                 ▼
                                    source-controller: GitRepository flux-system
                                                     │ artifact (tar.gz)
                                                     ▼
                                    kustomize-controller: Kustomization flux-system
                                                     │ apply every 10 min
                                                     ▼
                                                  Cluster

Drill 1: Bootstrap

flux bootstrap github --owner=$GITHUB_USER --repository=flux-fleet \
  --branch=main --path=clusters/lab --personal --private=true

Expected (shortened, hashes and keys differ):

► connecting to github.com
✔ repository "https://github.com/<du>/flux-fleet" created
► cloning branch "main" from Git repository "https://github.com/<du>/flux-fleet.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("05554f17511b194c00d5558b6f1cd8c13eca348d")
► pushing component manifests to "https://github.com/<du>/flux-fleet.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlz...
✔ configured deploy key "flux-system-main-flux-system-./clusters/lab" for "https://github.com/<du>/flux-fleet"
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("3fb103a568d48a36a87ba45f05dfeada2fefa241")
► pushing sync manifests to "https://github.com/<du>/flux-fleet.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
  • --personal means: the repo belongs to your user, not to an organization.
  • --path=clusters/lab is the folder for this one cluster. A second cluster would get clusters/prod in the same repo.
  • The token is only needed for the bootstrap: create the repo, commit, add the deploy key. It does not end up in the cluster. From now on Flux itself reads the repo through the deploy key, and that key may only read.

If the bootstrap fails

  • authorization failed: Write access to repository not granted: the token lacks Contents: Read and write.
  • GET …/keys: 403 Resource not accessible by personal access token: the token lacks Administration: Read and write.

Add the permissions (token → Edit → Update) and run the same command again. The bootstrap is repeatable: whatever already exists it leaves in place, and it reports, for example, component manifests are up to date.

Save your progress? Sign up for free

Drill 2: What is in the repo

You clone the fleet repo into the VM. So that git logs in to GitHub without writing the token to a file, it gets a small helper that reads the two environment variables from Day 0:

git config --global user.name "Flux Lab"
git config --global user.email "[email protected]"
git config --global credential.helper \
  '!f() { echo username=$GITHUB_USER; echo password=$GITHUB_TOKEN; }; f'
cd ~/flux-lab
git clone -q https://github.com/$GITHUB_USER/flux-fleet.git fleet
cd fleet
git log --oneline

Expected (the hashes differ):

3fb103a Add Flux sync manifests
05554f1 Add Flux v2.10.0 component manifests
find clusters -type f

Expected:

clusters/lab/flux-system/gotk-components.yaml
clusters/lab/flux-system/gotk-sync.yaml
clusters/lab/flux-system/kustomization.yaml
  • gotk-components.yaml (gotk = GitOps Toolkit) contains everything Flux needs: CRDs, controllers, permissions. About 6,600 lines, generated by flux bootstrap. You never edit it by hand.
  • gotk-sync.yaml contains the two objects that connect Flux with the repo:
cat clusters/lab/flux-system/gotk-sync.yaml

Expected:

# This manifest was generated by flux. DO NOT EDIT.
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: ssh://[email protected]/<du>/flux-fleet
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/lab
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
Field Meaning
GitRepository.spec.interval: 1m0s Check every minute whether main has a new commit
secretRef: flux-system The private part of the deploy key is stored in this Secret
url: ssh://… Flux reads over SSH, not over HTTPS
Kustomization.spec.interval: 10m0s Reconcile the cluster with Git every 10 minutes, even without a new commit
path: ./clusters/lab Apply this folder, with all subfolders
prune: true Whatever disappears from Git also disappears from the cluster

Next to the Flux Kustomization, the folder also contains a kustomization.yaml. The names are easy to confuse:

  • kustomization.yaml is a file for the Kustomize tool (Day 0). It lists which files belong together.
  • Kustomization (kustomize.toolkit.fluxcd.io) is an object in the cluster. It tells Flux which folder from which source to apply.

On GitHub you see the deploy key under flux-fleet → Settings → Deploy keys, marked “Read-only”.

Save your progress? Sign up for free

Drill 3: Looking at Flux

kubectl -n flux-system get pods

Expected (names and age differ):

NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-7879bfd657-csxff           1/1     Running   0          3m
kustomize-controller-6cbbc9f58d-9c8nl      1/1     Running   0          3m
notification-controller-7f7c8ccffc-g9fql   1/1     Running   0          3m
source-controller-648554b6c9-4wvvx         1/1     Running   0          3m

flux get shows the Flux objects with their state:

flux get all

Expected:

NAME                        REVISION            SUSPENDED   READY   MESSAGE
gitrepository/flux-system   main@sha1:3fb103a5  False       True    stored artifact for revision 'main@sha1:3fb103a5'

NAME                        REVISION            SUSPENDED   READY   MESSAGE
kustomization/flux-system   main@sha1:3fb103a5  False       True    Applied revision: main@sha1:3fb103a5
  • REVISION is the commit that Flux last fetched or applied. If both lines match git log, the cluster is up to date with Git.
  • stored artifact: the source-controller has stored the commit as an archive in its storage. The kustomize-controller picks it up there; it never talks to GitHub itself.
flux check

Expected (shortened):

► checking version in cluster
✔ distribution: flux-v2.10.0
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.7.1
...
✔ all checks passed

Save your progress? Sign up for free

Drill 4: Flux heals itself

Flux lives in Git, so Flux also restores itself. Delete a controller:

kubectl -n flux-system delete deploy notification-controller

Expected:

deployment.apps "notification-controller" deleted from flux-system namespace

It would be back within ten minutes at the latest. You do not have to wait that long:

flux reconcile kustomization flux-system --with-source

Expected:

► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:3fb103a568d48a36a87ba45f05dfeada2fefa241
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:3fb103a568d48a36a87ba45f05dfeada2fefa241
  • reconcile means: reconcile now, not only at the next interval. For this the CLI only sets an annotation on the object; the controller does the rest.
  • --with-source fetches the newest commit first. Without the option, Flux applies the last fetched state again.
kubectl -n flux-system get deploy notification-controller

Expected:

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
notification-controller   1/1     1            1           14s

What Flux did is in the events:

flux events --for Kustomization/flux-system

Expected (shortened):

14s   Normal  Progressing             Kustomization/flux-system  Deployment/flux-system/notification-controller created
1s    Normal  ReconciliationSucceeded Kustomization/flux-system  Reconciliation finished in 15.641814202s, next run in 10m0s

Save your progress? Sign up for free

Checkpoint

flux get kustomizations

Expected: one line flux-system with READY True.

git -C ~/flux-lab/fleet log --oneline | wc -l

Expected: 2.

kubectl -n flux-system get secret flux-system -o jsonpath='{.data}' | grep -o '"[a-z_.-]*":'

Expected:

"identity":
"identity.pub":
"known_hosts":

The Secret contains the private key (identity), the public key and the host keys of github.com. You do not display the values themselves.

Challenge

How many Kubernetes objects did the Kustomization flux-system create? Find the number without counting gotk-components.yaml.

Hint: Every Kustomization keeps an inventory of the objects it manages in its status.

Solution

kubectl -n flux-system get kustomization flux-system \
  -o jsonpath='{.status.inventory.entries[*].id}' | wc -w

Expected (the number may differ depending on the version):

34

The inventory is the reason why prune works: Flux knows exactly what it created itself, and deletes only that, never other objects.

flux tree shows the same inventory as a tree:

flux tree kustomization flux-system | head -5

Expected:

Kustomization/flux-system/flux-system
├── CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io
├── CustomResourceDefinition/buckets.source.toolkit.fluxcd.io
├── CustomResourceDefinition/externalartifacts.source.toolkit.fluxcd.io
├── CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io

Save your progress? Sign up for free

Enterprise Note

  • Deploy keys instead of tokens in the cluster: a deploy key is valid for only one repo and may only read here. If the cluster is compromised, the attacker cannot write anything to the repo. After that you only need the token from Day 0 for bootstrap runs.
  • Bootstrap is repeatable: a Flux upgrade is the same command with a newer
    1. It writes a new gotk-components.yaml to the repo, and Flux updates itself (Day 13).
  • Without bootstrap: flux install installs only the controllers, without a repo. Some companies separate this deliberately and install Flux via a Helm chart or with the Flux Operator, so that Flux itself is not managed from the fleet repo.