Day 11: Multi-Cluster
Goal: Argo CD on
argomanages a second cluster onprod. An ApplicationSet distributes platform services to all matching clusters. · Duration: approx. 50 min · Prerequisite: Day 8
Briefing
- So far Argo CD only writes to its own cluster. It is called
in-clusterand is reachable athttps://kubernetes.default.svc. - You register additional clusters with
argocd cluster add. Argo CD creates a ServiceAccount there and stores its token as a Secret in theargocdnamespace. - One Argo CD for many clusters is called hub and
spoke. No Argo CD runs on
prod, only k3s. - The cluster generator creates one Application per registered cluster. Labels on the cluster filter which clusters are meant.
- The matrix generator combines two generators: every entry of one with every entry of the other. Two services times two clusters result in four Applications.
VM argo VM prod
└─ k3s └─ k3s
└─ Argo CD ── https://172.16.0.123:6443 ──► ├─ ns podinfo
├─ in-cluster (stufe=dev) └─ ns plattform-info
└─ prod (stufe=prod)
Drill 1: k3s on prod
Start the prod VM if it is off, and log in:
In the VM prod, you install k3s exactly
as on Day 0:
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 \
sh -s - --disable traefik --disable servicelb
sudo k3s kubectl get nodesExpected (abridged):
[INFO] Using v1.37.2+k3s1 as release
…
[INFO] systemd: Starting k3s
NAME STATUS ROLES AGE VERSION
prod Ready control-plane 20s v1.37.2+k3s1
That is all you need on prod. Argo CD handles everything
else from argo. Log out again:
exitSave your progress? Sign up for free
Drill 2: Credentials to argo
Argo CD needs admin access to prod once to set itself up
there. So the file /etc/rancher/k3s/k3s.yaml from
prod has to get to argo. On your
laptop, you pipe it from one VM to the other without storing it
in between:
ssh [email protected] "sudo cat /etc/rancher/k3s/k3s.yaml" \
| ssh [email protected] "cat > ~/.kube/prod.yaml && chmod 600 ~/.kube/prod.yaml"From here on, back in the VM argo. In
the file, the server points to 127.0.0.1, and cluster,
user, and context are all called default, just like in your
own ~/.kube/config. You change both:
sed -i "s/127.0.0.1/172.16.0.123/; s/: default$/: prod/" ~/.kube/prod.yaml
grep -nE "server:|name:|cluster:|user:|current-context" ~/.kube/prod.yaml
kubectl --kubeconfig ~/.kube/prod.yaml get nodesExpected:
5: server: https://172.16.0.123:6443
6: name: prod
9: cluster: prod
10: user: prod
11: name: prod
12:current-context: prod
15:- name: prod
16: user:
NAME STATUS ROLES AGE VERSION
prod Ready control-plane 34s v1.37.2+k3s1
The k3s certificate is also valid for the VM’s IP address. That is
why access works without --insecure-skip-tls-verify.
Save your progress? Sign up for free
Drill 3: Register the Cluster
argocd cluster add prod --kubeconfig ~/.kube/prod.yaml --name prod -y
argocd cluster listExpected (abridged):
{"level":"info","msg":"ServiceAccount \"argocd-manager\" created in namespace \"kube-system\"",…}
{"level":"info","msg":"ClusterRole \"argocd-manager-role\" created",…}
{"level":"info","msg":"ClusterRoleBinding \"argocd-manager-role-binding\" created",…}
{"level":"info","msg":"Created bearer token secret \"argocd-manager-long-lived-token\" for ServiceAccount \"argocd-manager\"",…}
Cluster 'https://172.16.0.123:6443' added
SERVER NAME VERSION STATUS MESSAGE PROJECT
https://172.16.0.123:6443 prod v1.37.2 Successful
https://kubernetes.default.svc in-cluster v1.37.2 Successful
- The first
prodis the context in the file,--name prodis the name in Argo CD. - On
prod, there is now the ServiceAccountargocd-managerwith full permissions. Argo CD uses its token, not your admin file.
The access is stored as a Secret with a special label:
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=clusterExpected (abridged):
NAME TYPE DATA AGE
cluster-172.16.0.123-3405776406 Opaque 3 1s
From now on, you only need ~/.kube/prod.yaml for your
own looks at prod.
Save your progress? Sign up for free
Drill 4: Cluster Generator
podinfo comes back, this time as a platform service on every cluster. So that the folder is the same for every cluster, the manifests live as a plain file in the fleet repo:
cd ~/argo-lab/argocd-fleet
mkdir -p addons/podinfo
cat > addons/podinfo/podinfo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
spec:
replicas: 1
selector:
matchLabels:
app: podinfo
template:
metadata:
labels:
app: podinfo
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.17.0
ports:
- containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
name: podinfo
spec:
type: NodePort
selector:
app: podinfo
ports:
- port: 9898
targetPort: 9898
nodePort: 30090
EOFThe ApplicationSet has a cluster generator without a filter. The
generator provides {{.name}} and {{.server}}
for each cluster:
cat > applications/podinfo-appset.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: podinfo
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- clusters: {}
template:
metadata:
name: "podinfo-{{.name}}"
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: addons/podinfo
destination:
server: "{{.server}}"
namespace: podinfo
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
git add -A
git commit -m "podinfo auf allen Clustern"
git push
argocd app get root --refresh
sleep 30
kubectl -n argocd get app -o custom-columns=NAME:.metadata.name,\
SERVER:.spec.destination.server,SYNC:.status.sync.statusExpected:
NAME SERVER SYNC
argocd-extras https://kubernetes.default.svc Synced
notes https://kubernetes.default.svc Synced
notes-dev https://kubernetes.default.svc Synced
podinfo-in-cluster https://kubernetes.default.svc Synced
podinfo-prod https://172.16.0.123:6443 Synced
root https://kubernetes.default.svc Synced
Two Applications from one template. The local cluster counts too, even though there is no Secret for it. Both podinfos answer:
curl -s 172.16.0.123:30090 | grep '"version'
curl -s 172.16.0.182:30090 | grep '"version'Expected:
"version": "6.17.0",
"version": "6.17.0",
Save your progress? Sign up for free
Drill 5: Labels on the Cluster
Platform services should not always run everywhere. For that, the clusters get labels:
argocd cluster set prod --label stufe=prod
argocd cluster set in-cluster --label stufe=dev
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=cluster -L stufeExpected (abridged):
Cluster 'prod' updated.
Cluster 'in-cluster' updated.
NAME TYPE DATA AGE STUFE
cluster-172.16.0.123-3405776406 Opaque 3 60s prod
cluster-kubernetes.default.svc-3396314289 Opaque 3 0s dev
Argo CD has now also created a Secret for in-cluster,
because the label has to be stored somewhere.
The podinfo ApplicationSet has served its purpose. Its Applications have the finalizer from Day 8, so podinfo disappears from both clusters on removal:
git rm applications/podinfo-appset.yaml
git commit -m "podinfo-ApplicationSet entfernt"
git push
argocd app get root --refresh
sleep 25
argocd app list -o name
kubectl --kubeconfig ~/.kube/prod.yaml -n podinfo get allExpected:
argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/root
No resources found in podinfo namespace.
Save your progress? Sign up for free
Drill 6: Matrix Generator
Now the usual form for platform services: every folder under
addons is a service, and every service goes to every
cluster with stufe=prod. A second service joins, a small
ConfigMap:
mkdir -p addons/plattform-info
cat > addons/plattform-info/configmap.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: plattform-info
data:
team: plattform
kontakt: [email protected]
EOFThe matrix generator combines a Git generator for folders with the cluster generator:
cat > applications/addons-appset.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: addons
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- matrix:
generators:
- git:
repoURL: [email protected]:<du>/argocd-fleet.git
revision: main
directories:
- path: "addons/*"
- clusters:
selector:
matchLabels:
stufe: prod
template:
metadata:
name: "{{.path.basename}}-{{.name}}"
spec:
project: default
source:
repoURL: [email protected]:<du>/argocd-fleet.git
targetRevision: main
path: "{{.path.path}}"
destination:
server: "{{.server}}"
namespace: "{{.path.basename}}"
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
EOF
git add -A
git commit -m "Addons per Matrix auf prod"
git push
argocd app get root --refresh
sleep 30
kubectl -n argocd get app -o custom-columns=NAME:.metadata.name,\
SERVER:.spec.destination.server,NS:.spec.destination.namespaceExpected:
NAME SERVER NS
argocd-extras https://kubernetes.default.svc argocd
notes https://kubernetes.default.svc notes
notes-dev https://kubernetes.default.svc notes-dev
plattform-info-prod https://172.16.0.123:6443 plattform-info
podinfo-prod https://172.16.0.123:6443 podinfo
root https://kubernetes.default.svc argocd
directoriesinstead offiles: the Git generator delivers every matching folder.{{.path.basename}}is the folder name; here it serves as both name and namespace.{{.name}}and{{.server}}come from the cluster generator. The template sees the parameters of both generators.- Two folders times one cluster result in two Applications.
in-clusteris missing, because it hasstufe=dev.
kubectl --kubeconfig ~/.kube/prod.yaml -n plattform-info get cm plattform-info
kubectl --kubeconfig ~/.kube/prod.yaml -n podinfo get deployExpected:
NAME DATA AGE
plattform-info 2 27s
NAME READY UP-TO-DATE AVAILABLE AGE
podinfo 1/1 1 1 29s
On argo, the empty podinfo namespace from
the first attempt is still there. You clean it up:
kubectl delete ns podinfoSave your progress? Sign up for free
Checkpoint
argocd cluster listExpected: prod and in-cluster, both
Successful.
argocd cluster get prod -o json | grep -A2 '"labels"'Expected: "stufe": "prod".
argocd app list -o name | grep prodExpected: argocd/plattform-info-prod and
argocd/podinfo-prod.
Challenge
The addons should run on every cluster that has a stufe
label at all, regardless of its value. A new cluster then gets them
automatically as soon as it carries the label.
Hint: Besides matchLabels, a label
selector also knows matchExpressions with the operator
Exists.
Solution
In applications/addons-appset.yaml, you replace the
selector. Afterwards the part looks like this:
- clusters:
selector:
matchExpressions:
- key: stufe
operator: Existsgit commit -am "Addons auf allen Clustern mit Stufe"
git push
argocd app get root --refresh
sleep 30
argocd app list -o name
curl -s localhost:30090 | grep '"version'Expected (abridged):
argocd/argocd-extras
argocd/notes
argocd/notes-dev
argocd/plattform-info-in-cluster
argocd/plattform-info-prod
argocd/podinfo-in-cluster
argocd/podinfo-prod
argocd/root
"version": "6.17.0",
Four Applications: two services times two clusters.
Save your progress? Sign up for free
Enterprise Note
argocd-managerhascluster-adminpermissions on the target cluster. Whoever may read Secrets inargocdtherefore has all clusters. Theargocdnamespace must therefore be protected like a vault.- The
notesproject from Day 9 only allowshttps://kubernetes.default.svc. That way a team cannot accidentally deploy toprod. If it should be allowed to, the cluster is explicitly added todestinations. - Cluster Secrets can also be created declaratively, for example encrypted with SOPS from Day 10. Then the list of clusters is in Git as well.
- With very many clusters, a single application controller becomes a bottleneck. Argo CD then distributes the clusters across multiple controller replicas (sharding).