Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: A lab VM with k3s, the Flux CLI and Helm, a GitHub token and notes-api deployed once by hand in the cluster, so you see which problem Flux solves. · Duration: approx. 45 min · Prerequisite: an account on github.com

Briefing

  • GitOps means: the desired state of the cluster lives in a Git repo. A program in the cluster reads the repo and continuously reconciles the cluster with it. Whoever wants to change something changes Git, not the cluster.
  • Flux is such a program. It consists of several small controllers that run in the flux-system namespace. It is controlled through its own Kubernetes objects such as GitRepository or Kustomization.
  • The Flux CLI flux installs Flux, creates these objects and shows their state. It is not required; everything also works with kubectl. But it saves a lot of typing.
  • In this book the repo is called the fleet repo (flux-fleet). It is private in your GitHub account.
 Laptop ──ssh──► VM flux
                  └─ k3s
                      ├─ flux-system: Flux controllers (from Day 1)
                      └─ notes:       notes-api (by hand today, from Git from Day 2)
                         ▲
 github.com/<du>/flux-fleet ── Flux pulls ──┘
 curl ─► http://<VM-IP>:30080  notes-api

Drill 1: The lab VM

On the laptop (Multipass from multipass.run):

multipass launch 24.04 --name flux --cpus 2 --memory 4G --disk 30G
multipass exec flux -- bash -c "echo '$(cat ~/.ssh/id_ed25519.pub)' \
  >> ~/.ssh/authorized_keys"
multipass info flux | grep IPv4
ssh ubuntu@<IP aus der Ausgabe>
  • 4 GB are enough for Flux, notes-api, a small registry and later kro. The Flux controllers together need about 300 MB.
  • Any other Linux VM with sudo and internet access will do as well. The VM must reach github.com; GitHub does not need to reach the VM.

From here on, everything runs in the VM.

Save your progress? Sign up for free

Drill 2: k3s

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 \
  sh -s - --disable traefik --disable servicelb

Expected (shortened):

[INFO]  Using v1.37.2+k3s1 as release
...
[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s
  • get.k3s.io is the official installation script. It downloads k3s, creates the service and links kubectl.
  • --disable traefik --disable servicelb: you do not need an ingress controller or a load balancer; everything runs through NodePorts.

The credentials belong to root. You make yourself a copy:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
export KUBECONFIG=~/.kube/config
kubectl get nodes

Expected:

NAME   STATUS   ROLES           AGE   VERSION
flux   Ready    control-plane   20s   v1.37.2+k3s1

KUBECONFIG is mandatory with k3s. Without the variable, its kubectl always reads /etc/rancher/k3s/k3s.yaml and fails with permission denied. The Flux CLI reads the same variable.

Save your progress? Sign up for free

Drill 3: Flux CLI and Helm

Both programs come as ready-made binaries from GitHub:

mkdir -p ~/.local/bin ~/flux-lab
cd ~/flux-lab
curl -fsSLO https://github.com/fluxcd/flux2/releases/download/v2.10.0/flux_2.10.0_linux_amd64.tar.gz
tar xzf flux_2.10.0_linux_amd64.tar.gz -C ~/.local/bin flux
rm flux_2.10.0_linux_amd64.tar.gz

You need Helm from Day 6 on to package a chart:

curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
tar xzf helm-v4.3.1-linux-amd64.tar.gz
mv linux-amd64/helm ~/.local/bin/
rm -r linux-amd64 helm-v4.3.1-linux-amd64.tar.gz

Ubuntu adds ~/.local/bin to the PATH at login as soon as the folder exists. If you log out once and back in, the shell finds both programs:

exit
ssh ubuntu@<VM-IP>
flux version --client

Expected:

flux: v2.10.0
helm version --short

Expected:

v4.3.1+gc0d08d0

The Flux CLI completes commands with the Tab key once you enable it:

echo 'source <(flux completion bash)' >> ~/.bashrc
source ~/.bashrc

Now flux checks whether your cluster fits:

flux check --pre

Expected:

► checking prerequisites
✔ Kubernetes 1.37.2+k3s1 >=1.35.0-0
✔ prerequisites checks passed

Flux 2.10 supports the three newest Kubernetes versions, that is 1.35 to 1.37.

Save your progress? Sign up for free

Drill 4: The GitHub token

Tomorrow Flux creates a repo in your account and writes to it. For that it needs a token. You create it in the browser:

  1. github.com → profile picture → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token.
  2. Name flux-lab, expiration 30 days.
  3. Repository access: All repositories. The repo does not exist yet, after all.
  4. Permissions → Repository permissions:
    • Administration: Read and write (create the repo, add the deploy key)
    • Contents: Read and write (commit files)
    • Commit statuses: Read and write (from Day 11)
    • Workflows: Read and write (for the CI on Day 13)
  5. Generate token and copy the token. GitHub shows it only once.

In the VM you put it into a file that only you may read. read -s shows nothing while you paste, and the token does not end up in the shell history:

read -s T && echo "$T" > ~/.github-token && chmod 600 ~/.github-token

Paste the token, press Enter. Then your username and the token belong in environment variables, because that is exactly where the Flux CLI looks for them. Replace <du> with your GitHub username:

echo 'export GITHUB_USER=<du>' >> ~/.bashrc
echo 'export GITHUB_TOKEN=$(cat ~/.github-token)' >> ~/.bashrc
source ~/.bashrc
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" https://api.github.com/user \
  | grep '"login"'

Expected (with your name):

  "login": "<du>",

If you see "message": "Bad credentials" instead, the token was copied incorrectly. If the token lacks a permission, you only notice it during the bootstrap. Day 1 shows the messages for that.

Save your progress? Sign up for free

Drill 5: The companion code

git clone -q https://github.com/thomaszachmann/flux-zum-anfassen.git ~/flux-lab/code
ls ~/flux-lab/code ~/flux-lab/code/loesung

Expected:

/home/ubuntu/flux-lab/code:
LICENSE  loesung  README.md  wettkampf

/home/ubuntu/flux-lab/code/loesung:
pruefung  tag03  tag05  tag07  tag09  tag11  tag13
tag02     tag04  tag06  tag08  tag10  tag12  tag14
  • loesung/tagNN contains the state of your fleet repo at the end of each day. You write everything yourself and compare with diff at the end.
  • You only need wettkampf/ on Day 15.

You do not have to build an image. notes-api is ready in the GitHub Container Registry: ghcr.io/thomaszachmann/notes-api, in versions 0.5 to 0.7.

Save your progress? Sign up for free

Drill 6: notes-api by hand

Before Flux comes into play, you bring notes-api into the cluster the way it is usual without GitOps: write manifests, kubectl apply. The four files will later live in the fleet repo, today in ~/flux-lab/notes.

mkdir -p ~/flux-lab/notes
cd ~/flux-lab/notes
cat > namespace.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: notes
EOF
cat > secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-api
  namespace: notes
stringData:
  apiKey: lab-api-key-123
EOF

The API key is still in plain text here. With a Secret in a Git repo it cannot stay that way; Day 8 cleans it up.

cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  namespace: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
        - name: api
          image: ghcr.io/thomaszachmann/notes-api:0.5.1
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: notes-api
                  key: apiKey
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
EOF
cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: notes
  namespace: notes
spec:
  type: NodePort
  selector:
    app: notes
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080
EOF

A kustomization.yaml lists the files. kubectl understands it with -k, and Flux works the same way from tomorrow on:

cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - secret.yaml
  - deployment.yaml
  - service.yaml
EOF
cd ~/flux-lab
kubectl apply -k notes
kubectl -n notes rollout status deploy/notes

Expected:

namespace/notes created
secret/notes-api created
service/notes created
deployment.apps/notes created
Waiting for deployment "notes" rollout to finish: 0 of 1 updated replicas are available...
deployment "notes" successfully rolled out
curl -s localhost:30080/version

Expected:

0.5.1

If curl reports nothing (exit code 7), k3s is still setting up the NodePort. Wait a few seconds and try again.

Save your progress? Sign up for free

Drill 7: The drift

A colleague is in a hurry and moves notes-api to a new version directly in the cluster:

kubectl -n notes set image deploy/notes api=ghcr.io/thomaszachmann/notes-api:0.7.1
kubectl -n notes rollout status deploy/notes
curl -s localhost:30080/version

Expected (shortened):

deployment.apps/notes image updated
...
deployment "notes" successfully rolled out
0.7.1

Your files still say 0.5.1. The cluster and the files have diverged, and nobody notices. It only becomes visible when someone looks:

kubectl diff -k notes

Expected (shortened):

-        image: ghcr.io/thomaszachmann/notes-api:0.7.1
+        image: ghcr.io/thomaszachmann/notes-api:0.5.1

This divergence is called drift. Without GitOps, it depends on discipline whether someone notices it and rolls it back. From Day 2 on, Flux does that on its own every few minutes.

Save your progress? Sign up for free

Checkpoint

kubectl get nodes

Expected: one node flux with STATUS Ready.

flux check --pre

Expected: ✔ prerequisites checks passed.

echo "$GITHUB_USER ${#GITHUB_TOKEN}"

Expected: your GitHub name and a number greater than zero, the length of the token. That way the token itself stays invisible.

Challenge

Roll back the drift by hand and check that kubectl diff is silent afterwards.

Hint: kubectl diff exits with exit code 0 when there is nothing to show, and with 1 when it finds differences.

Solution

kubectl apply -k notes
kubectl -n notes rollout status deploy/notes
kubectl diff -k notes; echo "Exit-Code $?"

Expected (shortened):

deployment.apps/notes configured
...
Exit-Code 0

You no longer need the namespace. Tomorrow Flux creates it again:

kubectl delete -k notes

Save your progress? Sign up for free

Enterprise Note

  • Push or pull? A CI pipeline that runs kubectl apply needs write access to the cluster and only runs when someone pushes. Flux, on the other hand, pulls from the inside: the cluster only needs read access to the repo, and no machine outside holds cluster credentials.
  • Token or GitHub App? A personal token is tied to a person. If they leave the company, Flux stops. In operations you use a token of a technical account or a GitHub App, which flux bootstrap github also supports.
  • Flux or Argo CD? Both implement GitOps. Flux gets by without its own UI and is controlled entirely through Kubernetes objects. Argo CD brings a web UI and its own permission model.