Skip to content
Hands-on IT IT Drills

Day 17: Across Clusters

Goal: You connect the clusters ost and west with ClusterMesh, spread a Service across both clusters, fail over automatically when one goes down and control traffic across the cluster boundary with policy. · Duration: approx. 90 min · Prerequisite: Day 0, Day 12, one more VM with Ubuntu 24.04 (4 vCPU)

Briefing

  • CKNE: Service Networking and DNS – “Implementing Cross Cluster Service Discovery and Load Balancing”.
  • Each cluster exposes nodes, endpoints, identities and Services through its clustermesh-apiserver (an etcd); the agents of the other cluster read along. Packets travel directly from Pod to Pod through the tunnel.
  • Mandatory: a unique cluster.name and cluster.id (1–255), non-overlapping Pod CIDRs, the same datapath, a shared CA.
  • A Service with the same name and namespace in both clusters + the annotation service.cilium.io/global: "true" = backends from both clusters.
 Cluster ost (id 1)                          Cluster west (id 2)
 cp1 .126 · worker1 .228 · worker2 .83       argo .182 (single node)
 Pods 10.0.0.0/16                            Pods 10.1.0.0/16
 clustermesh-apiserver :32379  <-------->    clustermesh-apiserver :32379
        \________ Pod-to-Pod: VXLAN between the node IPs ________/

Drill 1: Build cluster west

In the lab, the VM for west is called argo (172.16.0.182). On it, run Drill 1 to 3 from Day 0 unchanged (kernel, containerd 2.4.2, runc 1.5.3, kubeadm 1.37.2). After that, only two differences from Day 0: its own IP for kubeadm init and no worker – the only node has to run Pods. On argo:

sudo kubeadm init --kubernetes-version=v1.37.2 \
  --apiserver-advertise-address=172.16.0.182 --skip-phases=addon/kube-proxy
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl taint nodes argo node-role.kubernetes.io/control-plane:NoSchedule-
kubectl get nodes

Expected (abridged):

[addons] Applied essential addon: CoreDNS
Your Kubernetes control-plane has initialized successfully!
...
node/argo untainted
NAME   STATUS     ROLES           AGE   VERSION
argo   NotReady   control-plane   6s    v1.37.2

From now on you work on cp1 again. As on Day 0, your own computer distributes the public key of cp1:

ssh [email protected] cat .ssh/id_ed25519.pub \
  | ssh [email protected] 'cat >> .ssh/authorized_keys'

On cp1, fetch the kubeconfig of west. Both kubeadm files use the same names for cluster, user and context (kubernetes, kubernetes-admin) – merged, one would hide the other. So rename them first:

mkdir -p ~/ckne/tag17 && cd ~/ckne/tag17
ssh -o StrictHostKeyChecking=accept-new [email protected] hostname
scp [email protected]:.kube/config ~/.kube/west.conf
chmod 600 ~/.kube/west.conf
sed -i -e 's/kubernetes-admin@kubernetes/west/' -e 's/kubernetes-admin/west-admin/' \
  -e 's/\(name\|cluster\): kubernetes$/\1: west/' ~/.kube/west.conf
sed -i -e 's/kubernetes-admin@kubernetes/ost/' -e 's/kubernetes-admin/ost-admin/' \
  -e 's/\(name\|cluster\): kubernetes$/\1: ost/' ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config:~/.kube/west.conf' >> ~/.bashrc
source ~/.bashrc
k config get-contexts
k --context west get nodes

Expected:

Warning: Permanently added '172.16.0.182' (ED25519) to the list of known hosts.
argo
CURRENT   NAME   CLUSTER   AUTHINFO     NAMESPACE
*         ost    ost       ost-admin
          west   west      west-admin
NAME   STATUS     ROLES           AGE   VERSION
argo   NotReady   control-plane   16s   v1.37.2

KUBECONFIG with two files merges both at runtime; current-context comes from the first file, so ost stays the default. kubectl, helm and the cilium CLI all understand this the same way.

Save your progress? Sign up for free

Drill 2: Cilium on west – with the CA of ost

The agents of both clusters talk to the other cluster’s clustermesh-apiserver over mTLS. That works without extra effort only if all certificates come from one CA. It lives in ost in the Secret cilium-ca; west gets it right at installation through the Helm value tls.ca:

CA_CRT=$(k --context ost -n kube-system get secret cilium-ca \
  -o jsonpath='{.data.ca\.crt}')
CA_KEY=$(k --context ost -n kube-system get secret cilium-ca \
  -o jsonpath='{.data.ca\.key}')
helm install cilium cilium/cilium --version 1.20.5 -n kube-system \
  --kube-context west \
  --set cluster.name=west --set cluster.id=2 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=172.16.0.182 --set k8sServicePort=6443 \
  --set ipam.mode=cluster-pool \
  --set ipam.operator.clusterPoolIPv4PodCIDRList=10.1.0.0/16 \
  --set ipam.operator.clusterPoolIPv4MaskSize=24 \
  --set routingMode=tunnel --set tunnelProtocol=vxlan \
  --set hubble.enabled=true --set hubble.relay.enabled=true \
  --set operator.replicas=1 \
  --set tls.ca.cert=$CA_CRT --set tls.ca.key=$CA_KEY
cilium status --context west --wait
k --context west get ciliumnodes -o custom-columns=NAME:.metadata.name,\
PODCIDRS:.spec.ipam.podCIDRs
cilium config view --context west | grep -E '^cluster-(name|id) '

Expected (abridged):

STATUS: deployed
...
    Cilium:             OK
    Operator:           OK
    Envoy DaemonSet:    OK
    Hubble Relay:       OK
    ClusterMesh:        disabled
...
NAME   PODCIDRS
argo   [10.1.0.0/24]
cluster-id                                        2
cluster-name                                      west

The same Helm values as on Day 0; only k8sServiceHost, Pod CIDR, name and ID differ. The same datapath (VXLAN, KPR) on both sides is mandatory. As an alternative, the docs show kubectl get secret cilium-ca … | kubectl --context … create -f - before the installation; with Helm, tls.ca is cleaner because the Secret then belongs to the release.

Save your progress? Sign up for free

Drill 3: ost gets a name and an ID

Since Day 0, ost has been running as default with ID 0 – that way it cannot join a mesh. The ID is part of every security identity (upper bits), so all identities change after the switch:

k -n kube-system get cep -o custom-columns=NAME:.metadata.name,\
ID:.status.identity.id | head -3
cilium config view | grep -E '^cluster-(name|id) '
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
k -n kube-system delete secret hubble-server-certs
helm upgrade cilium cilium/cilium -n kube-system --kube-context ost \
  --version 1.20.5 --reuse-values --set cluster.name=ost --set cluster.id=1
k -n kube-system rollout restart ds/cilium deploy/cilium-operator deploy/hubble-relay
k -n kube-system rollout status ds/cilium
cilium config view | grep -E '^cluster-(name|id) '
k -n kube-system get cep -o custom-columns=NAME:.metadata.name,\
ID:.status.identity.id | head -3

Expected (abridged):

NAME                              ID
coredns-559f6c778d-6png2          13185
coredns-559f6c778d-vwrtf          13185
cluster-id                                        0
cluster-name                                      default
secret "hubble-server-certs" deleted from kube-system namespace
Release "cilium" has been upgraded. Happy Helming!
daemon set "cilium" successfully rolled out
cluster-id                                        1
cluster-name                                      ost
NAME                              ID
coredns-559f6c778d-6png2          70078
coredns-559f6c778d-vwrtf          70078

70078 = 65536 + 4542: identities of cluster 1 start at 65536, those of cluster 2 at 131072. That keeps them unique in the mesh.

Warning: Deleting hubble-server-certs is not a mistake. Helm creates the certificate only if it is missing, and it contains the cluster name (*.default.hubble-grpc.cilium.io). If it stays, Hubble Relay no longer starts: x509: certificate is valid for *.default.hubble-grpc.cilium.io, not hubble-peer.ost.hubble-grpc.cilium.io.

After changing the name or ID, the docs require restarting all workloads so every Pod starts cleanly with its new identity:

for ns in $(k get ns -o jsonpath='{.items[*].metadata.name}'); do
  k -n $ns rollout restart deploy,sts 2>/dev/null
done
cilium status --wait
k get pods -A | grep -vE 'Running|Completed'

The loop over the namespaces is a deliberate exception to “one command at a time”: only the cluster knows which namespaces have workloads.

Expected (abridged):

deployment.apps/cert-manager restarted
...
deployment.apps/coredns restarted
deployment.apps/hubble-relay restarted
...
    Cilium:             OK
...
    Hubble Relay:       OK
NAMESPACE   NAME   READY   STATUS   RESTARTS   AGE

Only the header line remains: everything is running again. What restarts for you depends on what is still in the cluster from earlier days.

Save your progress? Sign up for free

Drill 4: Connect ClusterMesh

cilium clustermesh enable turns on the clustermesh-apiserver through Helm. A home network has no cloud load balancer, hence NodePort. A LoadBalancer from an LB IPAM pool with BGP announcement as on Day 15 would also work, but it makes the mesh depend on the BGP router – NodePort is simpler here.

cilium clustermesh enable --context ost --service-type NodePort
cilium clustermesh enable --context west --service-type NodePort
cilium clustermesh status --context ost --wait
k -n kube-system get svc clustermesh-apiserver

Expected (abridged, without the CLI’s symbols):

Using service type NodePort may fail when nodes are removed from the cluster!
...
Service "clustermesh-apiserver" of type "NodePort" found
Cluster access information is available:
  - 172.16.0.126:32379
Deployment clustermesh-apiserver is ready
KVStoreMesh is enabled
No cluster connected
NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)
clustermesh-apiserver   NodePort   10.108.33.199   <none>        2379:32379/TCP

The warning means: the other side remembers one node IP (here cp1). If that node fails, the connection breaks. Now connect – one direction is enough, the CLI sets up both:

cilium clustermesh connect --context ost --destination-context west
cilium clustermesh status --context ost --wait
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg node list
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep ClusterMesh

Expected (abridged):

Extracting access information of cluster ost...
Found ClusterMesh service IPs: [172.16.0.126]
Extracting access information of cluster west...
Found ClusterMesh service IPs: [172.16.0.182]
Configuring Cilium in cluster ost to connect to cluster west
Configuring Cilium in cluster west to connect to cluster ost
Connected cluster ost <=> west!
...
All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
All 1 KVStoreMesh replicas are connected to all clusters [min:1 / avg:1.0 / max:1]
Cluster Connections:
  - west: 3/3 configured, 3/3 connected - KVStoreMesh: 1/1 configured, 1/1 connected
Name          IPv4 Address   Endpoint CIDR   ...   Source
ost/cp1       172.16.0.126   10.0.0.0/24           local
ost/worker1   172.16.0.228   10.0.1.0/24           custom-resource
ost/worker2   172.16.0.83    10.0.2.0/24           custom-resource
west/argo     172.16.0.182   10.1.0.0/24           clustermesh
ClusterMesh:             1/1 remote clusters ready

west/argo with source clustermesh: the agent knows the remote node including its Pod CIDR and builds the tunnel to it. What the CLI set through Helm is shown by helm get values cilium -n kube-system | grep -A12 '^clustermesh:' (useAPIServer, apiserver.service.type, config.clusters with IP and port). Shortly after connect, status reports “nodes are not ready” for a minute or two – the agents are still loading the new configuration.

Save your progress? Sign up for free

Drill 5: Global Service

The same in both clusters: namespace tag17, an echo server and a client. The Deployments have a different name in each cluster, so the response (/hostname = Pod name) reveals who answered. The Service is called echo in both:

k --context ost create ns tag17
k config set-context ost --namespace=tag17
k --context ost create deployment echo-ost \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
  -- /agnhost netexec --http-port=8080
k --context ost expose deployment echo-ost --name=echo --port=80 --target-port=8080
k --context ost run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k --context west create ns tag17
k config set-context west --namespace=tag17
k --context west create deployment echo-west \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
  -- /agnhost netexec --http-port=8080
k --context west expose deployment echo-west --name=echo --port=80 --target-port=8080
k --context west run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k --context west wait --for=condition=Ready pod --all
k wait --for=condition=Ready pod --all
for i in $(seq 6); do k exec client -- curl -s echo/hostname; echo; done \
  | sort | uniq -c

Expected:

      6 echo-ost-b675f7c6-p9669

So far these are two separate Services. Now mark both as global:

k annotate svc echo service.cilium.io/global="true"
k --context west annotate svc echo service.cilium.io/global="true"
for i in $(seq 10); do k exec client -- curl -s echo/hostname; echo; done \
  | sort | uniq -c
for i in $(seq 10); do k --context west exec client -- curl -s echo/hostname; echo; \
  done | sort | uniq -c
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 $CIP

Expected (abridged):

      8 echo-ost-b675f7c6-p9669
      2 echo-west-7546f8b8c5-2v9q6
      6 echo-ost-b675f7c6-p9669
      4 echo-west-7546f8b8c5-2v9q6
72   10.98.55.8:80/TCP     ClusterIP      1 => 10.0.1.137:8080/TCP (active)
                                          2 => 10.1.0.81:8080/TCP (active)

The ClusterIP Service in ost now has a backend from 10.1.0.0/16 – the Pod in west. The ClusterIP itself is different in each cluster; only name and namespace have to match. If the annotation is missing on one side, that side neither shares its backends nor uses remote ones.

Save your progress? Sign up for free

Drill 6: Affinity, shared and failover

Balancing across clusters costs latency. service.cilium.io/affinity defines which backends are preferred: local, remote or none (default).

k annotate svc echo service.cilium.io/affinity=local
for i in $(seq 10); do k exec client -- curl -s echo/hostname; echo; done \
  | sort | uniq -c
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  --clustermesh-affinity | grep -A1 $CIP

Expected (abridged):

     10 echo-ost-b675f7c6-p9669
72   10.98.55.8:80/TCP     ClusterIP      1 => 10.0.1.137:8080/TCP (active) (preferred)

As long as a local backend is healthy, Cilium programs only that one into the service map. Now the outage – all backends in ost gone:

k scale deployment echo-ost --replicas=0
for i in $(seq 6); do k exec client -- curl -s -m3 echo/hostname; echo; done \
  | sort | uniq -c
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  --clustermesh-affinity | grep -A1 $CIP
k scale deployment echo-ost --replicas=1
k rollout status deployment echo-ost
for i in $(seq 6); do k exec client -- curl -s echo/hostname; echo; done \
  | sort | uniq -c

Expected (abridged):

      6 echo-west-7546f8b8c5-2v9q6
72   10.98.55.8:80/TCP     ClusterIP      1 => 10.1.0.81:8080/TCP (active) (preferred)
deployment "echo-ost" successfully rolled out
      6 echo-ost-b675f7c6-cxrxl

No error at the client, no DNS change: the Service fails over to west and comes back as soon as ost has a backend again. With affinity=remote it would be the other way around.

service.cilium.io/shared: "false" makes a global Service “consume only”: the cluster uses remote backends but does not hand out its own.

k annotate svc echo service.cilium.io/affinity-
k --context west annotate svc echo service.cilium.io/shared="false"
for i in $(seq 8); do k exec client -- curl -s echo/hostname; echo; done \
  | sort | uniq -c
for i in $(seq 8); do k --context west exec client -- curl -s echo/hostname; echo; \
  done | sort | uniq -c
k --context west annotate svc echo service.cilium.io/shared-

Expected (abridged):

      8 echo-ost-b675f7c6-cxrxl
      4 echo-ost-b675f7c6-cxrxl
      4 echo-west-7546f8b8c5-2v9q6

ost no longer sees the backend from west; west keeps using both.

Save your progress? Sign up for free

Drill 7: Policy across cluster boundaries and Hubble

Every endpoint carries the label io.cilium.k8s.policy.cluster. Since Cilium 1.19, policies without this label select only endpoints in their own cluster (policy-default-local-cluster). First the trap, directly on the Pod IP in west:

WIP=$(k --context west get pod -l app=echo-west -o jsonpath='{.items[0].status.podIP}')
cat > nur-client.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: nur-client
  namespace: tag17
spec:
  endpointSelector:
    matchLabels:
      app: echo-west
  ingress:
  - fromEndpoints:
    - matchLabels:
        run: client
EOF
k --context west apply -f nur-client.yaml
k exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"

Expected:

command terminated with exit code 28
 rc=28
echo-west-7546f8b8c5-2v9q6 rc=0

So Pod-to-Pod across the cluster boundary works in principle – only the policy does not let the client from ost through. With the cluster label:

sed -i '/run: client/a\        io.cilium.k8s.policy.cluster: ost' nur-client.yaml
k --context west apply -f nur-client.yaml
k exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg identity get $(k get cep client -o jsonpath='{.status.identity.id}')

Expected:

ciliumnetworkpolicy.cilium.io/nur-client configured
echo-west-7546f8b8c5-2v9q6 rc=0
command terminated with exit code 28
 rc=28
ID      LABELS
72956   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=tag17
        k8s:io.cilium.k8s.policy.cluster=ost
        k8s:io.cilium.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=tag17
        k8s:run=client

The agent in west knows identity 72956 from ost including its labels – via the clustermesh-apiserver. And Hubble? Because both clusters trust the same CA, Hubble Relay also finds the agents of the other cluster:

hubble list nodes -P --kube-context west --port-forward-port 0
hubble observe -P --kube-context west --port-forward-port 0 -n tag17 \
  --from-cluster ost --to-port 8080 --last 1
hubble observe -P --kube-context west --port-forward-port 0 -n tag17 \
  --from-cluster west --verdict DROPPED --last 1

Expected (abridged, timestamps omitted):

NAME          STATUS      AGE      FLOWS/S   CURRENT/MAX-FLOWS
ost/cp1       Connected   3m48s    31.67     4095/4095 (100.00%)
ost/worker1   Connected   3m35s    23.23     4095/4095 (100.00%)
ost/worker2   Connected   3m48s    18.13     4095/4095 (100.00%)
west/argo     Connected   20m26s   18.58     4095/4095 (100.00%)
tag17/client:58712 (ID:72956) -> tag17/echo-west-...-2v9q6:8080 (ID:148838)
  to-endpoint FORWARDED (TCP Flags: ACK, FIN)
tag17/client:46810 (ID:194188) <> tag17/echo-west-...-2v9q6:8080 (ID:148838)
  Policy denied DROPPED (TCP Flags: SYN)

Two clients with the same name – the identity reveals the cluster: 72956 lies in the range of ost, 194188 in the range of west. --last applies per node, so one line per node may appear.

Save your progress? Sign up for free

Drill 8: MCS API – the Kubernetes standard

The Multi-Cluster Services API (SIG Multicluster) does the same in a vendor-neutral way: ServiceExport in the source cluster, ServiceImport and a dedicated DNS name <svc>.<ns>.svc.clusterset.local everywhere. Cilium 1.20 implements it when clustermesh.mcsapi.enabled is on. CoreDNS needs the multicluster plugin for this; Cilium sets it up itself with a Job – here only in west, ost stays unchanged.

helm upgrade cilium cilium/cilium -n kube-system --kube-context ost \
  --version 1.20.5 --reuse-values --set clustermesh.mcsapi.enabled=true
helm upgrade cilium cilium/cilium -n kube-system --kube-context west \
  --version 1.20.5 --reuse-values --set clustermesh.mcsapi.enabled=true \
  --set clustermesh.mcsapi.corednsAutoConfigure.enabled=true
k --context ost -n kube-system rollout restart deploy/cilium-operator \
  deploy/clustermesh-apiserver
k --context west -n kube-system rollout restart deploy/cilium-operator \
  deploy/clustermesh-apiserver
k --context west api-resources | grep multicluster
k --context west -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' \
  | grep -A1 'kubernetes '

Expected:

serviceexports    svcex,svcexport   multicluster.x-k8s.io/v1beta1   true   ServiceExport
serviceimports    svcim,svcimport   multicluster.x-k8s.io/v1beta1   true   ServiceImport
    kubernetes cluster.local clusterset.local in-addr.arpa ip6.arpa {
       multicluster clusterset.local

The Corefile entry multicluster clusterset.local comes from the Job cilium-coredns-mcsapi-autoconfig. Remove the global annotations and the policy, then export in both clusters:

k --context ost annotate svc echo service.cilium.io/global-
k --context west annotate svc echo service.cilium.io/global-
k --context west delete cnp nur-client
cat > export.yaml <<'EOF'
apiVersion: multicluster.x-k8s.io/v1beta1
kind: ServiceExport
metadata:
  name: echo
  namespace: tag17
EOF
k --context ost apply -f export.yaml
k --context west apply -f export.yaml
k --context west get serviceexport,serviceimport,svc
k --context west get serviceimport echo -o jsonpath='{.status.clusters}'; echo
for i in $(seq 8); do k --context west exec client -- \
  curl -s echo.tag17.svc.clusterset.local/hostname; echo; done | sort | uniq -c

Expected (abridged):

NAME                                       AGE
serviceexport.multicluster.x-k8s.io/echo   10s
NAME                                       TYPE           IP                 AGE
serviceimport.multicluster.x-k8s.io/echo   ClusterSetIP   ["10.105.28.79"]   11s
NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/derived-ounaj88gle   ClusterIP   10.105.28.79   <none>        80/TCP    11s
service/echo                 ClusterIP   10.96.136.38   <none>        80/TCP    4m10s
[{"cluster":"ost"},{"cluster":"west"}]
      6 echo-ost-b675f7c6-cxrxl
      2 echo-west-7546f8b8c5-2v9q6

For the ServiceImport, Cilium creates a derived-… Service; its IP is the ClusterSetIP that clusterset.local points to. The difference from global Services: who exports is explicit (an object instead of an annotation), and the name echo.tag17.svc.cluster.local stays purely local. In ost, clusterset.local does not resolve because CoreDNS is not extended there.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which clusters ost is connected to and whether all nodes are connected. Target time: 20 s
  2. Make the Service web in the namespace shop in ost global. Target time: 15 s
  3. Make sure that web falls back to west only when all local backends are down. Target time: 15 s
  4. Show which remote nodes the Cilium agent knows. Target time: 20 s
  5. Show the cluster name and ID Cilium runs with in west. Target time: 20 s

Solution

cilium clustermesh status
k -n shop annotate svc web service.cilium.io/global="true"
k -n shop annotate svc web service.cilium.io/affinity=local
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg node list \
  | grep clustermesh
cilium config view --context west | grep -E '^cluster-(name|id) '

Expected (abridged):

Cluster Connections:
  - west: 3/3 configured, 3/3 connected - KVStoreMesh: 1/1 configured, 1/1 connected
...
west/argo     172.16.0.182   10.1.0.0/24           clustermesh
cluster-id                                        2
cluster-name                                      west

For 2 and 3, the Service in the other cluster needs the same global annotation too – otherwise it stays one-sided.

Save your progress? Sign up for free

Checkpoint

cilium clustermesh status --context ost

Expected: All 3 nodes are connected to all clusters, connection west: 3/3 configured, 3/3 connected.

cilium clustermesh status --context west

Expected: All 1 nodes are connected to all clusters, connection ost: 1/1 configured, 1/1 connected.

cilium status --context ost

Expected: Cilium: OK, Hubble Relay: OK, ClusterMesh: OK.

cilium config view --context ost | grep -E '^cluster-(name|id) '

Expected: cluster-id 1, cluster-name ost.

k --context west get ciliumnodes -o custom-columns=NAME:.metadata.name,\
PODCIDRS:.spec.ipam.podCIDRs

Expected: argo [10.1.0.0/24] – no overlap with ost (10.0.0.0/16).

k get pods -A | grep -vE 'Running|Completed'

Expected: only the header line.

What stays for Day 21 (Round 8) and the practice exam (Appendix B, Task 7): cluster west (argo, Cilium 1.20.5, cluster.name=west, cluster.id=2, CA of ost), in ost cluster.name=ost, cluster.id=1, ClusterMesh in both (clustermesh-apiserver as NodePort 32379, config.clusters), plus clustermesh.mcsapi.enabled=true in both and the CoreDNS extension clusterset.local in west. The contexts ost/west and ~/.kube/west.conf stay.

Challenge

Allow access to echo-west for every Pod with run=client – no matter which cluster in the mesh it comes from – but for no other Pod. Prove it with a throwaway Pod fremd in ost.

Hint: Instead of a fixed cluster name, a matchExpressions with operator: Exists checks that the cluster label is present.

Solution

k --context west apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: client-alle-cluster
  namespace: tag17
spec:
  endpointSelector:
    matchLabels:
      app: echo-west
  ingress:
  - fromEndpoints:
    - matchLabels:
        run: client
      matchExpressions:
      - key: io.cilium.k8s.policy.cluster
        operator: Exists
EOF
k exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k run fremd --image=nicolaka/netshoot:v0.16 --rm -i --restart=Never -- \
  curl -s -m3 $WIP:8080/hostname; echo " rc=$?"

Expected (abridged):

ciliumnetworkpolicy.cilium.io/client-alle-cluster created
echo-west-7546f8b8c5-2v9q6 rc=0
echo-west-7546f8b8c5-2v9q6 rc=0
pod "fremd" deleted from tag17 namespace
 rc=28
pod tag17/fremd terminated (Error)

rc=28 is the exit code of curl (timeout): k run --rm -i passes it on.

Every endpoint in the mesh has the cluster label, so Exists matches all clusters – exactly what was the default before Cilium 1.19.

Save your progress? Sign up for free

Cleanup

ClusterMesh and the MCS API stay (see Checkpoint). Only the test objects go:

k --context ost delete namespace tag17
k --context west delete namespace tag17
k config set-context ost --namespace=default
k config set-context west --namespace=default
k get ns tag17 ; k --context west get ns tag17

Expected:

namespace "tag17" deleted
namespace "tag17" deleted
Error from server (NotFound): namespaces "tag17" not found
Error from server (NotFound): namespaces "tag17" not found

Along with the namespace, the policy, the ServiceExport and the derived-… Service disappear too.

Exam Tip

  • Check order for “the clusters don’t see each other”: cilium clustermesh status, then cilium config view | grep cluster- (name/ID unique?), Pod CIDRs, CA.
  • A global Service needs the annotation in both clusters and the same name and namespace. Since 1.19, policies without io.cilium.k8s.policy.cluster only apply locally.
  • Docs: docs.cilium.io “Setting up Cluster Mesh”, “Load-balancing & Service Discovery” (Global Services, Service Affinity, MCS-API).

Enterprise Note

  • clustermesh-apiserver behind a real LoadBalancer (or LB IPAM + BGP) with several replicas – NodePort depends on a single node IP.
  • The shared CA comes from cert-manager or a PKI (tls.caBundle for rotation), not as a Helm value: otherwise tls.ca.key ends up in the Helm release Secret.
  • Assign cluster IDs and Pod CIDRs centrally before the first cluster is created – changing an ID later means restarting all workloads.
  • clustermesh.maxConnectedClusters=511 works only at installation and halves the number of identities per cluster.
  • For databases tied to a cluster, affinity=local; for data center switchovers, shared=false on the side that should only consume.