Skip to content
Hands-on IT IT Drills

Day 11: Multi-Cluster

Goal: Argo CD on argo manages a second cluster on prod. An ApplicationSet distributes platform services to all matching clusters. · Duration: approx. 50 min · Prerequisite: Day 8

Briefing

  • So far Argo CD only writes to its own cluster. It is called in-cluster and is reachable at https://kubernetes.default.svc.
  • You register additional clusters with argocd cluster add. Argo CD creates a ServiceAccount there and stores its token as a Secret in the argocd namespace.
  • One Argo CD for many clusters is called hub and spoke. No Argo CD runs on prod, only k3s.
  • The cluster generator creates one Application per registered cluster. Labels on the cluster filter which clusters are meant.
  • The matrix generator combines two generators: every entry of one with every entry of the other. Two services times two clusters result in four Applications.
 VM argo                                       VM prod
 └─ k3s                                         └─ k3s
     └─ Argo CD ── https://172.16.0.123:6443 ──►    ├─ ns podinfo
         ├─ in-cluster (stufe=dev)                  └─ ns plattform-info
         └─ prod       (stufe=prod)

Drill 1: k3s on prod

Start the prod VM if it is off, and log in:

In the VM prod, you install k3s exactly as on Day 0:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 \
  sh -s - --disable traefik --disable servicelb
sudo k3s kubectl get nodes

Expected (abridged):

[INFO]  Using v1.37.2+k3s1 as release
…
[INFO]  systemd: Starting k3s
NAME   STATUS   ROLES           AGE   VERSION
prod   Ready    control-plane   20s   v1.37.2+k3s1

That is all you need on prod. Argo CD handles everything else from argo. Log out again:

exit

Save your progress? Sign up for free

Drill 2: Credentials to argo

Argo CD needs admin access to prod once to set itself up there. So the file /etc/rancher/k3s/k3s.yaml from prod has to get to argo. On your laptop, you pipe it from one VM to the other without storing it in between:

ssh [email protected] "sudo cat /etc/rancher/k3s/k3s.yaml" \
  | ssh [email protected] "cat > ~/.kube/prod.yaml && chmod 600 ~/.kube/prod.yaml"

From here on, back in the VM argo. In the file, the server points to 127.0.0.1, and cluster, user, and context are all called default, just like in your own ~/.kube/config. You change both:

sed -i "s/127.0.0.1/172.16.0.123/; s/: default$/: prod/" ~/.kube/prod.yaml
grep -nE "server:|name:|cluster:|user:|current-context" ~/.kube/prod.yaml
kubectl --kubeconfig ~/.kube/prod.yaml get nodes

Expected:

5:    server: https://172.16.0.123:6443
6:  name: prod
9:    cluster: prod
10:    user: prod
11:  name: prod
12:current-context: prod
15:- name: prod
16:  user:
NAME   STATUS   ROLES           AGE   VERSION
prod   Ready    control-plane   34s   v1.37.2+k3s1

The k3s certificate is also valid for the VM’s IP address. That is why access works without --insecure-skip-tls-verify.

Save your progress? Sign up for free

Drill 3: Register the Cluster

argocd cluster add prod --kubeconfig ~/.kube/prod.yaml --name prod -y
argocd cluster list

Expected (abridged):

{"level":"info","msg":"ServiceAccount \"argocd-manager\" created in namespace \"kube-system\"",…}
{"level":"info","msg":"ClusterRole \"argocd-manager-role\" created",…}
{"level":"info","msg":"ClusterRoleBinding \"argocd-manager-role-binding\" created",…}
{"level":"info","msg":"Created bearer token secret \"argocd-manager-long-lived-token\" for ServiceAccount \"argocd-manager\"",…}
Cluster 'https://172.16.0.123:6443' added
SERVER                          NAME        VERSION  STATUS      MESSAGE  PROJECT
https://172.16.0.123:6443       prod        v1.37.2  Successful
https://kubernetes.default.svc  in-cluster  v1.37.2  Successful
  • The first prod is the context in the file, --name prod is the name in Argo CD.
  • On prod, there is now the ServiceAccount argocd-manager with full permissions. Argo CD uses its token, not your admin file.

The access is stored as a Secret with a special label:

kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=cluster

Expected (abridged):

NAME                              TYPE     DATA   AGE
cluster-172.16.0.123-3405776406   Opaque   3      1s

From now on, you only need ~/.kube/prod.yaml for your own looks at prod.

Save your progress? Sign up for free

Drill 4: Cluster Generator

podinfo comes back, this time as a platform service on every cluster. So that the folder is the same for every cluster, the manifests live as a plain file in the fleet repo:

cd ~/argo-lab/argocd-fleet
mkdir -p addons/podinfo
cat > addons/podinfo/podinfo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.17.0
          ports:
            - containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
spec:
  type: NodePort
  selector:
    app: podinfo
  ports:
    - port: 9898
      targetPort: 9898
      nodePort: 30090
EOF

The ApplicationSet has a cluster generator without a filter. The generator provides {{.name}} and {{.server}} for each cluster:

cat > applications/podinfo-appset.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: podinfo
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters: {}
  template:
    metadata:
      name: "podinfo-{{.name}}"
    spec:
      project: default
      source:
        repoURL: [email protected]:<du>/argocd-fleet.git
        targetRevision: main
        path: addons/podinfo
      destination:
        server: "{{.server}}"
        namespace: podinfo
      syncPolicy:
        automated:
          selfHeal: true
          prune: true
        syncOptions:
          - CreateNamespace=true
EOF
git add -A
git commit -m "podinfo auf allen Clustern"
git push
argocd app get root --refresh
sleep 30
kubectl -n argocd get app -o custom-columns=NAME:.metadata.name,\
SERVER:.spec.destination.server,SYNC:.status.sync.status

Expected:

NAME                 SERVER                           SYNC
argocd-extras        https://kubernetes.default.svc   Synced
notes                https://kubernetes.default.svc   Synced
notes-dev            https://kubernetes.default.svc   Synced
podinfo-in-cluster   https://kubernetes.default.svc   Synced
podinfo-prod         https://172.16.0.123:6443        Synced
root                 https://kubernetes.default.svc   Synced

Two Applications from one template. The local cluster counts too, even though there is no Secret for it. Both podinfos answer:

curl -s 172.16.0.123:30090 | grep '"version'
curl -s 172.16.0.182:30090 | grep '"version'

Expected:

  "version": "6.17.0",
  "version": "6.17.0",

Save your progress? Sign up for free

Drill 5: Labels on the Cluster

Platform services should not always run everywhere. For that, the clusters get labels:

argocd cluster set prod --label stufe=prod
argocd cluster set in-cluster --label stufe=dev
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=cluster -L stufe

Expected (abridged):

Cluster 'prod' updated.
Cluster 'in-cluster' updated.
NAME                                        TYPE     DATA   AGE   STUFE
cluster-172.16.0.123-3405776406             Opaque   3      60s   prod
cluster-kubernetes.default.svc-3396314289   Opaque   3      0s    dev

Argo CD has now also created a Secret for in-cluster, because the label has to be stored somewhere.

The podinfo ApplicationSet has served its purpose. Its Applications have the finalizer from Day 8, so podinfo disappears from both clusters on removal:

git rm applications/podinfo-appset.yaml
git commit -m "podinfo-ApplicationSet entfernt"
git push
argocd app get root --refresh
sleep 25
argocd app list -o name
kubectl --kubeconfig ~/.kube/prod.yaml -n podinfo get all

Expected:

argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/root
No resources found in podinfo namespace.

Save your progress? Sign up for free

Drill 6: Matrix Generator

Now the usual form for platform services: every folder under addons is a service, and every service goes to every cluster with stufe=prod. A second service joins, a small ConfigMap:

mkdir -p addons/plattform-info
cat > addons/plattform-info/configmap.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: plattform-info
data:
  team: plattform
  kontakt: [email protected]
EOF

The matrix generator combines a Git generator for folders with the cluster generator:

cat > applications/addons-appset.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: addons
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: [email protected]:<du>/argocd-fleet.git
              revision: main
              directories:
                - path: "addons/*"
          - clusters:
              selector:
                matchLabels:
                  stufe: prod
  template:
    metadata:
      name: "{{.path.basename}}-{{.name}}"
    spec:
      project: default
      source:
        repoURL: [email protected]:<du>/argocd-fleet.git
        targetRevision: main
        path: "{{.path.path}}"
      destination:
        server: "{{.server}}"
        namespace: "{{.path.basename}}"
      syncPolicy:
        automated:
          selfHeal: true
          prune: true
        syncOptions:
          - CreateNamespace=true
EOF
git add -A
git commit -m "Addons per Matrix auf prod"
git push
argocd app get root --refresh
sleep 30
kubectl -n argocd get app -o custom-columns=NAME:.metadata.name,\
SERVER:.spec.destination.server,NS:.spec.destination.namespace

Expected:

NAME                  SERVER                           NS
argocd-extras         https://kubernetes.default.svc   argocd
notes                 https://kubernetes.default.svc   notes
notes-dev             https://kubernetes.default.svc   notes-dev
plattform-info-prod   https://172.16.0.123:6443        plattform-info
podinfo-prod          https://172.16.0.123:6443        podinfo
root                  https://kubernetes.default.svc   argocd
  • directories instead of files: the Git generator delivers every matching folder. {{.path.basename}} is the folder name; here it serves as both name and namespace.
  • {{.name}} and {{.server}} come from the cluster generator. The template sees the parameters of both generators.
  • Two folders times one cluster result in two Applications. in-cluster is missing, because it has stufe=dev.
kubectl --kubeconfig ~/.kube/prod.yaml -n plattform-info get cm plattform-info
kubectl --kubeconfig ~/.kube/prod.yaml -n podinfo get deploy

Expected:

NAME             DATA   AGE
plattform-info   2      27s
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
podinfo   1/1     1            1           29s

On argo, the empty podinfo namespace from the first attempt is still there. You clean it up:

kubectl delete ns podinfo

Save your progress? Sign up for free

Checkpoint

argocd cluster list

Expected: prod and in-cluster, both Successful.

argocd cluster get prod -o json | grep -A2 '"labels"'

Expected: "stufe": "prod".

argocd app list -o name | grep prod

Expected: argocd/plattform-info-prod and argocd/podinfo-prod.

Challenge

The addons should run on every cluster that has a stufe label at all, regardless of its value. A new cluster then gets them automatically as soon as it carries the label.

Hint: Besides matchLabels, a label selector also knows matchExpressions with the operator Exists.

Solution

In applications/addons-appset.yaml, you replace the selector. Afterwards the part looks like this:

          - clusters:
              selector:
                matchExpressions:
                  - key: stufe
                    operator: Exists
git commit -am "Addons auf allen Clustern mit Stufe"
git push
argocd app get root --refresh
sleep 30
argocd app list -o name
curl -s localhost:30090 | grep '"version'

Expected (abridged):

argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/plattform-info-in-cluster
argocd/plattform-info-prod
argocd/podinfo-in-cluster
argocd/podinfo-prod
argocd/root
  "version": "6.17.0",

Four Applications: two services times two clusters.

Save your progress? Sign up for free

Enterprise Note

  • argocd-manager has cluster-admin permissions on the target cluster. Whoever may read Secrets in argocd therefore has all clusters. The argocd namespace must therefore be protected like a vault.
  • The notes project from Day 9 only allows https://kubernetes.default.svc. That way a team cannot accidentally deploy to prod. If it should be allowed to, the cluster is explicitly added to destinations.
  • Cluster Secrets can also be created declaratively, for example encrypted with SOPS from Day 10. Then the list of clusters is in Git as well.
  • With very many clusters, a single application controller becomes a bottleneck. Argo CD then distributes the clusters across multiple controller replicas (sharding).