Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: Two lab VMs, k3s, the argocd CLI and Helm, a fleet repo with a deploy key, and notes-api deployed once by hand in the cluster, so you see which problem Argo CD solves. · Duration: approx. 45 min · Prerequisite: an account on github.com

Briefing

  • GitOps means: the desired state of the cluster lives in a Git repo. A program in the cluster continuously compares the repo with the cluster and reconciles. If you want to change something, you change Git, not the cluster.
  • Argo CD is such a program. It runs in the argocd namespace, has a web UI, a CLI, and its own Kubernetes objects such as Application.
  • In this book the repo is called the fleet repo (argocd-fleet). It is private in your GitHub account. Argo CD reads it through a deploy key, an SSH key that is valid for this one repo only.
 Laptop ──ssh──► VM argo                       VM prod (from Day 11)
                  └─ k3s                          └─ k3s
                      ├─ argocd: Argo CD (from Day 1) ───► manages prod
                      └─ notes:  notes-api (by hand today, from Git from Day 2)
                         ▲
 github.com/<du>/argocd-fleet ── Argo CD fetches via SSH ──┘
 curl ─► http://<argo-IP>:30080  notes-api

Drill 1: The Lab VMs

You need two Linux VMs with Ubuntu 24.04 on the same network:

VM vCPU RAM Disk from
argo 4 8 GB 40 GB Day 0
prod 2 4 GB 30 GB Day 11

On your laptop you can do this, for example, with Multipass from multipass.run:

multipass launch 24.04 --name argo --cpus 4 --memory 8G --disk 40G
multipass launch 24.04 --name prod --cpus 2 --memory 4G --disk 30G
multipass info argo | grep IPv4
  • Any other Linux VM with sudo and internet access works too. The VMs must reach github.com; GitHub does not need to reach them.
  • You only need prod on Day 11. Until then it can stay off.

From here on, everything runs in the VM argo. In the book it has the address 172.16.0.182, and prod has 172.16.0.123. Use your own addresses.

Save your progress? Sign up for free

Drill 2: k3s

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 \
  sh -s - --disable traefik --disable servicelb

Expected (abridged):

[INFO]  Using v1.37.2+k3s1 as release
…
[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s
  • get.k3s.io is the official installation script. It downloads k3s, creates the service, and links kubectl.
  • --disable traefik --disable servicelb: you do not need an ingress controller or a load balancer; everything runs through NodePorts.

The credentials belong to root. You make yourself a copy:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
export KUBECONFIG=~/.kube/config
kubectl get nodes

Expected:

NAME   STATUS   ROLES           AGE   VERSION
argo   Ready    control-plane   21s   v1.37.2+k3s1

KUBECONFIG is mandatory with k3s. Without the variable, its kubectl always reads /etc/rancher/k3s/k3s.yaml and fails with permission denied. The argocd CLI reads the same variable.

Save your progress? Sign up for free

Drill 3: argocd CLI and Helm

Both programs come as ready-made binaries. The argocd CLI is a single file on GitHub:

mkdir -p ~/.local/bin ~/argo-lab
cd ~/argo-lab
curl -fsSL -o argocd \
  https://github.com/argoproj/argo-cd/releases/download/v3.5.7/argocd-linux-amd64
install -m 755 argocd ~/.local/bin/argocd
rm argocd

You need Helm on Day 1 for the installation and on Day 5 for your own charts:

curl -fsSLO https://get.helm.sh/helm-v4.3.2-linux-amd64.tar.gz
tar xzf helm-v4.3.2-linux-amd64.tar.gz
mv linux-amd64/helm ~/.local/bin/
rm -r linux-amd64 helm-v4.3.2-linux-amd64.tar.gz

Ubuntu adds ~/.local/bin to the PATH at login as soon as the folder exists. Log out once and log back in:

exit
ssh [email protected]
argocd version --client --short

Expected:

argocd: v3.5.7+6d73296
helm version --short

Expected:

v4.3.2+g8346aaa

Save your progress? Sign up for free

Drill 4: Fleet Repo and Deploy Key

Create a private repo in the browser: github.com → New repository → name argocd-fleet, Private, without a README. With the GitHub CLI on your laptop you can also use gh repo create argocd-fleet --private.

In the VM, you generate a separate key just for this repo:

ssh-keygen -q -t ed25519 -N "" -C argocd-fleet -f ~/.ssh/argocd-fleet
cat ~/.ssh/argocd-fleet.pub

Expected (your key will differ):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFaWVjfNDS1nnR5RjlbUFtrY8/U1V30V864SgtlgfuyJ argocd-fleet

You add the public part to the repo: Settings → Deploy keys → Add deploy key, title argo-lab, paste the key, check Allow write access. The key may write because you push from the VM and Argo CD makes commits itself on Day 12.

git should use exactly this key for github.com:

cat > ~/.ssh/config <<'EOF'
Host github.com
  IdentityFile ~/.ssh/argocd-fleet
EOF
chmod 600 ~/.ssh/config
ssh-keyscan -t ed25519 github.com >> ~/.ssh/known_hosts
ssh -T [email protected]

Expected (abridged):

# github.com:22 SSH-2.0-11529790
Hi <du>/argocd-fleet! You've successfully authenticated, but GitHub does not provide shell access.

GitHub shows the repo name instead of your account. This tells you that a deploy key is in effect and not a personal key. ssh-keyscan fetches GitHub’s host key. If needed, compare it with the fingerprints at docs.github.com → “GitHub’s SSH key fingerprints”.

Now you clone the empty repo and configure git:

cd ~/argo-lab
git clone [email protected]:<du>/argocd-fleet.git
git config --global user.name "Dein Name"
git config --global user.email "[email protected]"
git config --global init.defaultBranch main

Expected:

Cloning into 'argocd-fleet'...
warning: You appear to have cloned an empty repository.

Save your progress? Sign up for free

Drill 5: The Companion Code

git clone -q https://github.com/thomaszachmann/argocd-zum-anfassen.git ~/argo-lab/code
ls ~/argo-lab/code

Expected:

LICENSE  README.md  loesung  wettkampf
  • loesung/tagNN contains the state of your fleet repo at the end of each day. You write everything yourself and compare with diff at the end.
  • You only need wettkampf/ on Day 15.

You do not need to build an image. notes-api is ready in the GitHub Container Registry: ghcr.io/thomaszachmann/notes-api, among others in versions 0.5.1 to 0.7.1.

Save your progress? Sign up for free

Drill 6: notes-api by Hand

Before Argo CD comes into play, you bring notes-api into the cluster the way it is usually done without GitOps: write a manifest, kubectl apply. Without a database, notes-api keeps the notes in memory, which is enough for today. Postgres comes on Day 4.

mkdir -p ~/argo-lab/hand
cd ~/argo-lab/hand
cat > notes.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: notes
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
  namespace: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
        - name: api
          image: ghcr.io/thomaszachmann/notes-api:0.5.1
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              value: lab-api-key-123
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: notes
  namespace: notes
spec:
  type: NodePort
  selector:
    app: notes
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080
EOF
kubectl apply -f notes.yaml
kubectl -n notes rollout status deploy/notes

Expected:

namespace/notes created
deployment.apps/notes created
service/notes created
Waiting for deployment "notes" rollout to finish: 0 of 1 updated replicas are available...
deployment "notes" successfully rolled out
curl -s localhost:30080/version

Expected:

0.5.1

If curl does not answer, k3s is still setting up the NodePort. Wait a few seconds and try again.

Save your progress? Sign up for free

Drill 7: The Drift

A colleague is in a hurry and bumps notes-api to a new version directly in the cluster:

kubectl -n notes set image deploy/notes api=ghcr.io/thomaszachmann/notes-api:0.7.1
kubectl -n notes rollout status deploy/notes

Expected (abridged):

deployment.apps/notes image updated
…
deployment "notes" successfully rolled out
curl -s localhost:30080/version

Expected:

0.7.1

Your file still says 0.5.1. The cluster and the file have diverged, and nobody notices. It only becomes visible when someone checks:

kubectl diff -f notes.yaml

Expected (abridged):

-        image: ghcr.io/thomaszachmann/notes-api:0.7.1
+        image: ghcr.io/thomaszachmann/notes-api:0.5.1

This deviation is called drift. Without GitOps, it depends on discipline whether anyone notices it and reverts it. From Day 3 on, Argo CD does that on its own.

Save your progress? Sign up for free

Checkpoint

kubectl get nodes

Expected: one node argo with STATUS Ready.

argocd version --client --short

Expected: argocd: v3.5.7+6d73296.

git -C ~/argo-lab/argocd-fleet remote -v

Expected: [email protected]:<du>/argocd-fleet.git twice, once with (fetch), once with (push).

Challenge

Revert the drift by hand and check that kubectl diff is silent afterwards. Then clean up the notes namespace, because from Day 2 on Argo CD creates it.

Hint: kubectl apply enforces the state of the file. kubectl diff exits without output when nothing differs.

Solution

kubectl apply -f notes.yaml
kubectl -n notes rollout status deploy/notes
kubectl diff -f notes.yaml

Expected (abridged):

namespace/notes unchanged
deployment.apps/notes configured
service/notes unchanged
…
deployment "notes" successfully rolled out

After that, kubectl diff no longer prints anything. Clean up:

kubectl delete -f notes.yaml

Expected:

namespace "notes" deleted
deployment.apps "notes" deleted from notes namespace
service "notes" deleted from notes namespace

Save your progress? Sign up for free

Enterprise Note

  • In company clusters, a platform team installs Argo CD. Teams get projects and permissions, not cluster-admin (Day 9).
  • Deploy keys are valid for exactly one repo. For many repos, you use a GitHub App or a machine account with a read token, never a person’s key.
  • With GitOps, running kubectl directly against the production cluster is the exception, for example in an emergency. Many teams give humans only read access there.