Day 17: Across Clusters
Goal: You connect the clusters
ostandwestwith ClusterMesh, spread a Service across both clusters, fail over automatically when one goes down and control traffic across the cluster boundary with policy. · Duration: approx. 90 min · Prerequisite: Day 0, Day 12, one more VM with Ubuntu 24.04 (4 vCPU)
Briefing
- CKNE: Service Networking and DNS – “Implementing Cross Cluster Service Discovery and Load Balancing”.
- Each cluster exposes nodes, endpoints, identities and Services
through its
clustermesh-apiserver(an etcd); the agents of the other cluster read along. Packets travel directly from Pod to Pod through the tunnel. - Mandatory: a unique
cluster.nameandcluster.id(1–255), non-overlapping Pod CIDRs, the same datapath, a shared CA. - A Service with the same name and namespace in both clusters + the
annotation
service.cilium.io/global: "true"= backends from both clusters.
Cluster ost (id 1) Cluster west (id 2)
cp1 .126 · worker1 .228 · worker2 .83 argo .182 (single node)
Pods 10.0.0.0/16 Pods 10.1.0.0/16
clustermesh-apiserver :32379 <--------> clustermesh-apiserver :32379
\________ Pod-to-Pod: VXLAN between the node IPs ________/
Drill 1: Build cluster west
In the lab, the VM for west is called argo
(172.16.0.182). On it, run Drill 1 to 3 from Day 0
unchanged (kernel, containerd 2.4.2, runc 1.5.3, kubeadm
1.37.2). After that, only two differences from Day 0: its own IP for
kubeadm init and no worker – the only node has to run Pods.
On argo:
sudo kubeadm init --kubernetes-version=v1.37.2 \
--apiserver-advertise-address=172.16.0.182 --skip-phases=addon/kube-proxy
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl taint nodes argo node-role.kubernetes.io/control-plane:NoSchedule-
kubectl get nodesExpected (abridged):
[addons] Applied essential addon: CoreDNS
Your Kubernetes control-plane has initialized successfully!
...
node/argo untainted
NAME STATUS ROLES AGE VERSION
argo NotReady control-plane 6s v1.37.2
From now on you work on cp1 again. As on Day 0, your own
computer distributes the public key of cp1:
ssh [email protected] cat .ssh/id_ed25519.pub \
| ssh [email protected] 'cat >> .ssh/authorized_keys'On cp1, fetch the kubeconfig of west. Both
kubeadm files use the same names for cluster, user and context
(kubernetes, kubernetes-admin) – merged, one
would hide the other. So rename them first:
mkdir -p ~/ckne/tag17 && cd ~/ckne/tag17
ssh -o StrictHostKeyChecking=accept-new [email protected] hostname
scp [email protected]:.kube/config ~/.kube/west.conf
chmod 600 ~/.kube/west.conf
sed -i -e 's/kubernetes-admin@kubernetes/west/' -e 's/kubernetes-admin/west-admin/' \
-e 's/\(name\|cluster\): kubernetes$/\1: west/' ~/.kube/west.conf
sed -i -e 's/kubernetes-admin@kubernetes/ost/' -e 's/kubernetes-admin/ost-admin/' \
-e 's/\(name\|cluster\): kubernetes$/\1: ost/' ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config:~/.kube/west.conf' >> ~/.bashrc
source ~/.bashrc
k config get-contexts
k --context west get nodesExpected:
Warning: Permanently added '172.16.0.182' (ED25519) to the list of known hosts.
argo
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* ost ost ost-admin
west west west-admin
NAME STATUS ROLES AGE VERSION
argo NotReady control-plane 16s v1.37.2
KUBECONFIG with two files merges both at runtime;
current-context comes from the first file, so
ost stays the default. kubectl, helm and the cilium CLI all
understand this the same way.
Save your progress? Sign up for free
Drill 2: Cilium on west – with the CA of ost
The agents of both clusters talk to the other cluster’s
clustermesh-apiserver over mTLS. That works without extra
effort only if all certificates come from one CA. It
lives in ost in the Secret cilium-ca;
west gets it right at installation through the Helm value
tls.ca:
CA_CRT=$(k --context ost -n kube-system get secret cilium-ca \
-o jsonpath='{.data.ca\.crt}')
CA_KEY=$(k --context ost -n kube-system get secret cilium-ca \
-o jsonpath='{.data.ca\.key}')
helm install cilium cilium/cilium --version 1.20.5 -n kube-system \
--kube-context west \
--set cluster.name=west --set cluster.id=2 \
--set kubeProxyReplacement=true \
--set k8sServiceHost=172.16.0.182 --set k8sServicePort=6443 \
--set ipam.mode=cluster-pool \
--set ipam.operator.clusterPoolIPv4PodCIDRList=10.1.0.0/16 \
--set ipam.operator.clusterPoolIPv4MaskSize=24 \
--set routingMode=tunnel --set tunnelProtocol=vxlan \
--set hubble.enabled=true --set hubble.relay.enabled=true \
--set operator.replicas=1 \
--set tls.ca.cert=$CA_CRT --set tls.ca.key=$CA_KEY
cilium status --context west --wait
k --context west get ciliumnodes -o custom-columns=NAME:.metadata.name,\
PODCIDRS:.spec.ipam.podCIDRs
cilium config view --context west | grep -E '^cluster-(name|id) 'Expected (abridged):
STATUS: deployed
...
Cilium: OK
Operator: OK
Envoy DaemonSet: OK
Hubble Relay: OK
ClusterMesh: disabled
...
NAME PODCIDRS
argo [10.1.0.0/24]
cluster-id 2
cluster-name west
The same Helm values as on Day 0; only k8sServiceHost,
Pod CIDR, name and ID differ. The same datapath (VXLAN, KPR) on both
sides is mandatory. As an alternative, the docs show
kubectl get secret cilium-ca … | kubectl --context … create -f -
before the installation; with Helm, tls.ca is cleaner
because the Secret then belongs to the release.
Save your progress? Sign up for free
Drill 3: ost gets a name and an ID
Since Day 0, ost has been running as
default with ID 0 – that way it cannot join a mesh. The ID
is part of every security identity (upper bits), so all identities
change after the switch:
k -n kube-system get cep -o custom-columns=NAME:.metadata.name,\
ID:.status.identity.id | head -3
cilium config view | grep -E '^cluster-(name|id) '
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
k -n kube-system delete secret hubble-server-certs
helm upgrade cilium cilium/cilium -n kube-system --kube-context ost \
--version 1.20.5 --reuse-values --set cluster.name=ost --set cluster.id=1
k -n kube-system rollout restart ds/cilium deploy/cilium-operator deploy/hubble-relay
k -n kube-system rollout status ds/cilium
cilium config view | grep -E '^cluster-(name|id) '
k -n kube-system get cep -o custom-columns=NAME:.metadata.name,\
ID:.status.identity.id | head -3Expected (abridged):
NAME ID
coredns-559f6c778d-6png2 13185
coredns-559f6c778d-vwrtf 13185
cluster-id 0
cluster-name default
secret "hubble-server-certs" deleted from kube-system namespace
Release "cilium" has been upgraded. Happy Helming!
daemon set "cilium" successfully rolled out
cluster-id 1
cluster-name ost
NAME ID
coredns-559f6c778d-6png2 70078
coredns-559f6c778d-vwrtf 70078
70078 = 65536 + 4542: identities of cluster 1 start at 65536, those of cluster 2 at 131072. That keeps them unique in the mesh.
Warning: Deleting
hubble-server-certsis not a mistake. Helm creates the certificate only if it is missing, and it contains the cluster name (*.default.hubble-grpc.cilium.io). If it stays, Hubble Relay no longer starts:x509: certificate is valid for *.default.hubble-grpc.cilium.io, not hubble-peer.ost.hubble-grpc.cilium.io.
After changing the name or ID, the docs require restarting all workloads so every Pod starts cleanly with its new identity:
for ns in $(k get ns -o jsonpath='{.items[*].metadata.name}'); do
k -n $ns rollout restart deploy,sts 2>/dev/null
done
cilium status --wait
k get pods -A | grep -vE 'Running|Completed'The loop over the namespaces is a deliberate exception to “one command at a time”: only the cluster knows which namespaces have workloads.
Expected (abridged):
deployment.apps/cert-manager restarted
...
deployment.apps/coredns restarted
deployment.apps/hubble-relay restarted
...
Cilium: OK
...
Hubble Relay: OK
NAMESPACE NAME READY STATUS RESTARTS AGE
Only the header line remains: everything is running again. What restarts for you depends on what is still in the cluster from earlier days.
Save your progress? Sign up for free
Drill 4: Connect ClusterMesh
cilium clustermesh enable turns on the
clustermesh-apiserver through Helm. A home network has no
cloud load balancer, hence NodePort. A LoadBalancer from an
LB IPAM pool with BGP announcement as on Day 15 would also work, but it
makes the mesh depend on the BGP router – NodePort is simpler here.
cilium clustermesh enable --context ost --service-type NodePort
cilium clustermesh enable --context west --service-type NodePort
cilium clustermesh status --context ost --wait
k -n kube-system get svc clustermesh-apiserverExpected (abridged, without the CLI’s symbols):
Using service type NodePort may fail when nodes are removed from the cluster!
...
Service "clustermesh-apiserver" of type "NodePort" found
Cluster access information is available:
- 172.16.0.126:32379
Deployment clustermesh-apiserver is ready
KVStoreMesh is enabled
No cluster connected
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
clustermesh-apiserver NodePort 10.108.33.199 <none> 2379:32379/TCP
The warning means: the other side remembers one node
IP (here cp1). If that node fails, the connection breaks.
Now connect – one direction is enough, the CLI sets up both:
cilium clustermesh connect --context ost --destination-context west
cilium clustermesh status --context ost --wait
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg node list
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep ClusterMeshExpected (abridged):
Extracting access information of cluster ost...
Found ClusterMesh service IPs: [172.16.0.126]
Extracting access information of cluster west...
Found ClusterMesh service IPs: [172.16.0.182]
Configuring Cilium in cluster ost to connect to cluster west
Configuring Cilium in cluster west to connect to cluster ost
Connected cluster ost <=> west!
...
All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
All 1 KVStoreMesh replicas are connected to all clusters [min:1 / avg:1.0 / max:1]
Cluster Connections:
- west: 3/3 configured, 3/3 connected - KVStoreMesh: 1/1 configured, 1/1 connected
Name IPv4 Address Endpoint CIDR ... Source
ost/cp1 172.16.0.126 10.0.0.0/24 local
ost/worker1 172.16.0.228 10.0.1.0/24 custom-resource
ost/worker2 172.16.0.83 10.0.2.0/24 custom-resource
west/argo 172.16.0.182 10.1.0.0/24 clustermesh
ClusterMesh: 1/1 remote clusters ready
west/argo with source clustermesh: the
agent knows the remote node including its Pod CIDR and builds the tunnel
to it. What the CLI set through Helm is shown by
helm get values cilium -n kube-system | grep -A12 '^clustermesh:'
(useAPIServer, apiserver.service.type,
config.clusters with IP and port). Shortly after
connect, status reports “nodes are not ready”
for a minute or two – the agents are still loading the new
configuration.
Save your progress? Sign up for free
Drill 5: Global Service
The same in both clusters: namespace tag17, an echo
server and a client. The Deployments have a different name in each
cluster, so the response (/hostname = Pod name) reveals who
answered. The Service is called echo in both:
k --context ost create ns tag17
k config set-context ost --namespace=tag17
k --context ost create deployment echo-ost \
--image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
-- /agnhost netexec --http-port=8080
k --context ost expose deployment echo-ost --name=echo --port=80 --target-port=8080
k --context ost run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k --context west create ns tag17
k config set-context west --namespace=tag17
k --context west create deployment echo-west \
--image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 --port=8080 \
-- /agnhost netexec --http-port=8080
k --context west expose deployment echo-west --name=echo --port=80 --target-port=8080
k --context west run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k --context west wait --for=condition=Ready pod --all
k wait --for=condition=Ready pod --all
for i in $(seq 6); do k exec client -- curl -s echo/hostname; echo; done \
| sort | uniq -cExpected:
6 echo-ost-b675f7c6-p9669
So far these are two separate Services. Now mark both as global:
k annotate svc echo service.cilium.io/global="true"
k --context west annotate svc echo service.cilium.io/global="true"
for i in $(seq 10); do k exec client -- curl -s echo/hostname; echo; done \
| sort | uniq -c
for i in $(seq 10); do k --context west exec client -- curl -s echo/hostname; echo; \
done | sort | uniq -c
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 $CIPExpected (abridged):
8 echo-ost-b675f7c6-p9669
2 echo-west-7546f8b8c5-2v9q6
6 echo-ost-b675f7c6-p9669
4 echo-west-7546f8b8c5-2v9q6
72 10.98.55.8:80/TCP ClusterIP 1 => 10.0.1.137:8080/TCP (active)
2 => 10.1.0.81:8080/TCP (active)
The ClusterIP Service in ost now has a backend from
10.1.0.0/16 – the Pod in west. The ClusterIP
itself is different in each cluster; only name and namespace have to
match. If the annotation is missing on one side, that side neither
shares its backends nor uses remote ones.
Save your progress? Sign up for free
Drill 6: Affinity, shared and failover
Balancing across clusters costs latency.
service.cilium.io/affinity defines which backends are
preferred: local, remote or none
(default).
k annotate svc echo service.cilium.io/affinity=local
for i in $(seq 10); do k exec client -- curl -s echo/hostname; echo; done \
| sort | uniq -c
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
--clustermesh-affinity | grep -A1 $CIPExpected (abridged):
10 echo-ost-b675f7c6-p9669
72 10.98.55.8:80/TCP ClusterIP 1 => 10.0.1.137:8080/TCP (active) (preferred)
As long as a local backend is healthy, Cilium programs only that one
into the service map. Now the outage – all backends in ost
gone:
k scale deployment echo-ost --replicas=0
for i in $(seq 6); do k exec client -- curl -s -m3 echo/hostname; echo; done \
| sort | uniq -c
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
--clustermesh-affinity | grep -A1 $CIP
k scale deployment echo-ost --replicas=1
k rollout status deployment echo-ost
for i in $(seq 6); do k exec client -- curl -s echo/hostname; echo; done \
| sort | uniq -cExpected (abridged):
6 echo-west-7546f8b8c5-2v9q6
72 10.98.55.8:80/TCP ClusterIP 1 => 10.1.0.81:8080/TCP (active) (preferred)
deployment "echo-ost" successfully rolled out
6 echo-ost-b675f7c6-cxrxl
No error at the client, no DNS change: the Service fails over to
west and comes back as soon as ost has a
backend again. With affinity=remote it would be the other
way around.
service.cilium.io/shared: "false" makes a global Service
“consume only”: the cluster uses remote backends but does not hand out
its own.
k annotate svc echo service.cilium.io/affinity-
k --context west annotate svc echo service.cilium.io/shared="false"
for i in $(seq 8); do k exec client -- curl -s echo/hostname; echo; done \
| sort | uniq -c
for i in $(seq 8); do k --context west exec client -- curl -s echo/hostname; echo; \
done | sort | uniq -c
k --context west annotate svc echo service.cilium.io/shared-Expected (abridged):
8 echo-ost-b675f7c6-cxrxl
4 echo-ost-b675f7c6-cxrxl
4 echo-west-7546f8b8c5-2v9q6
ost no longer sees the backend from west;
west keeps using both.
Save your progress? Sign up for free
Drill 7: Policy across cluster boundaries and Hubble
Every endpoint carries the label
io.cilium.k8s.policy.cluster. Since Cilium 1.19, policies
without this label select only endpoints in their own
cluster (policy-default-local-cluster). First the
trap, directly on the Pod IP in west:
WIP=$(k --context west get pod -l app=echo-west -o jsonpath='{.items[0].status.podIP}')
cat > nur-client.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: nur-client
namespace: tag17
spec:
endpointSelector:
matchLabels:
app: echo-west
ingress:
- fromEndpoints:
- matchLabels:
run: client
EOF
k --context west apply -f nur-client.yaml
k exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"Expected:
command terminated with exit code 28
rc=28
echo-west-7546f8b8c5-2v9q6 rc=0
So Pod-to-Pod across the cluster boundary works in principle – only
the policy does not let the client from ost
through. With the cluster label:
sed -i '/run: client/a\ io.cilium.k8s.policy.cluster: ost' nur-client.yaml
k --context west apply -f nur-client.yaml
k exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg identity get $(k get cep client -o jsonpath='{.status.identity.id}')Expected:
ciliumnetworkpolicy.cilium.io/nur-client configured
echo-west-7546f8b8c5-2v9q6 rc=0
command terminated with exit code 28
rc=28
ID LABELS
72956 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=tag17
k8s:io.cilium.k8s.policy.cluster=ost
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=tag17
k8s:run=client
The agent in west knows identity 72956 from
ost including its labels – via the
clustermesh-apiserver. And Hubble? Because both clusters
trust the same CA, Hubble Relay also finds the agents of the other
cluster:
hubble list nodes -P --kube-context west --port-forward-port 0
hubble observe -P --kube-context west --port-forward-port 0 -n tag17 \
--from-cluster ost --to-port 8080 --last 1
hubble observe -P --kube-context west --port-forward-port 0 -n tag17 \
--from-cluster west --verdict DROPPED --last 1Expected (abridged, timestamps omitted):
NAME STATUS AGE FLOWS/S CURRENT/MAX-FLOWS
ost/cp1 Connected 3m48s 31.67 4095/4095 (100.00%)
ost/worker1 Connected 3m35s 23.23 4095/4095 (100.00%)
ost/worker2 Connected 3m48s 18.13 4095/4095 (100.00%)
west/argo Connected 20m26s 18.58 4095/4095 (100.00%)
tag17/client:58712 (ID:72956) -> tag17/echo-west-...-2v9q6:8080 (ID:148838)
to-endpoint FORWARDED (TCP Flags: ACK, FIN)
tag17/client:46810 (ID:194188) <> tag17/echo-west-...-2v9q6:8080 (ID:148838)
Policy denied DROPPED (TCP Flags: SYN)
Two clients with the same name – the identity reveals the cluster:
72956 lies in the range of ost, 194188 in the range of
west. --last applies per node, so one line per
node may appear.
Save your progress? Sign up for free
Drill 8: MCS API – the Kubernetes standard
The Multi-Cluster Services API (SIG Multicluster) does the same in a
vendor-neutral way: ServiceExport in the source cluster,
ServiceImport and a dedicated DNS name
<svc>.<ns>.svc.clusterset.local everywhere.
Cilium 1.20 implements it when clustermesh.mcsapi.enabled
is on. CoreDNS needs the multicluster plugin for this;
Cilium sets it up itself with a Job – here only in west,
ost stays unchanged.
helm upgrade cilium cilium/cilium -n kube-system --kube-context ost \
--version 1.20.5 --reuse-values --set clustermesh.mcsapi.enabled=true
helm upgrade cilium cilium/cilium -n kube-system --kube-context west \
--version 1.20.5 --reuse-values --set clustermesh.mcsapi.enabled=true \
--set clustermesh.mcsapi.corednsAutoConfigure.enabled=true
k --context ost -n kube-system rollout restart deploy/cilium-operator \
deploy/clustermesh-apiserver
k --context west -n kube-system rollout restart deploy/cilium-operator \
deploy/clustermesh-apiserver
k --context west api-resources | grep multicluster
k --context west -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' \
| grep -A1 'kubernetes 'Expected:
serviceexports svcex,svcexport multicluster.x-k8s.io/v1beta1 true ServiceExport
serviceimports svcim,svcimport multicluster.x-k8s.io/v1beta1 true ServiceImport
kubernetes cluster.local clusterset.local in-addr.arpa ip6.arpa {
multicluster clusterset.local
The Corefile entry multicluster clusterset.local comes
from the Job cilium-coredns-mcsapi-autoconfig. Remove the
global annotations and the policy, then export in both clusters:
k --context ost annotate svc echo service.cilium.io/global-
k --context west annotate svc echo service.cilium.io/global-
k --context west delete cnp nur-client
cat > export.yaml <<'EOF'
apiVersion: multicluster.x-k8s.io/v1beta1
kind: ServiceExport
metadata:
name: echo
namespace: tag17
EOF
k --context ost apply -f export.yaml
k --context west apply -f export.yaml
k --context west get serviceexport,serviceimport,svc
k --context west get serviceimport echo -o jsonpath='{.status.clusters}'; echo
for i in $(seq 8); do k --context west exec client -- \
curl -s echo.tag17.svc.clusterset.local/hostname; echo; done | sort | uniq -cExpected (abridged):
NAME AGE
serviceexport.multicluster.x-k8s.io/echo 10s
NAME TYPE IP AGE
serviceimport.multicluster.x-k8s.io/echo ClusterSetIP ["10.105.28.79"] 11s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/derived-ounaj88gle ClusterIP 10.105.28.79 <none> 80/TCP 11s
service/echo ClusterIP 10.96.136.38 <none> 80/TCP 4m10s
[{"cluster":"ost"},{"cluster":"west"}]
6 echo-ost-b675f7c6-cxrxl
2 echo-west-7546f8b8c5-2v9q6
For the ServiceImport, Cilium creates a
derived-… Service; its IP is the ClusterSetIP that
clusterset.local points to. The difference from global
Services: who exports is explicit (an object instead of an annotation),
and the name echo.tag17.svc.cluster.local stays purely
local. In ost, clusterset.local does not
resolve because CoreDNS is not extended there.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which clusters
ostis connected to and whether all nodes are connected. Target time: 20 s - Make the Service
webin the namespaceshopinostglobal. Target time: 15 s - Make sure that
webfalls back towestonly when all local backends are down. Target time: 15 s - Show which remote nodes the Cilium agent knows. Target time: 20 s
- Show the cluster name and ID Cilium runs with in
west. Target time: 20 s
Solution
cilium clustermesh status
k -n shop annotate svc web service.cilium.io/global="true"
k -n shop annotate svc web service.cilium.io/affinity=local
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg node list \
| grep clustermesh
cilium config view --context west | grep -E '^cluster-(name|id) 'Expected (abridged):
Cluster Connections:
- west: 3/3 configured, 3/3 connected - KVStoreMesh: 1/1 configured, 1/1 connected
...
west/argo 172.16.0.182 10.1.0.0/24 clustermesh
cluster-id 2
cluster-name west
For 2 and 3, the Service in the other cluster needs the same
global annotation too – otherwise it stays one-sided.
Save your progress? Sign up for free
Checkpoint
cilium clustermesh status --context ostExpected:
All 3 nodes are connected to all clusters, connection
west: 3/3 configured, 3/3 connected.
cilium clustermesh status --context westExpected:
All 1 nodes are connected to all clusters, connection
ost: 1/1 configured, 1/1 connected.
cilium status --context ostExpected: Cilium: OK,
Hubble Relay: OK, ClusterMesh: OK.
cilium config view --context ost | grep -E '^cluster-(name|id) 'Expected: cluster-id 1,
cluster-name ost.
k --context west get ciliumnodes -o custom-columns=NAME:.metadata.name,\
PODCIDRS:.spec.ipam.podCIDRsExpected: argo [10.1.0.0/24] – no
overlap with ost (10.0.0.0/16).
k get pods -A | grep -vE 'Running|Completed'Expected: only the header line.
What stays for Day 21 (Round 8) and the practice exam (Appendix B,
Task 7): cluster west (argo, Cilium 1.20.5,
cluster.name=west, cluster.id=2, CA of
ost), in ost cluster.name=ost,
cluster.id=1, ClusterMesh in both
(clustermesh-apiserver as NodePort 32379,
config.clusters), plus
clustermesh.mcsapi.enabled=true in both and the CoreDNS
extension clusterset.local in west. The
contexts ost/west and
~/.kube/west.conf stay.
Challenge
Allow access to echo-west for every Pod with
run=client – no matter which cluster in the mesh it comes
from – but for no other Pod. Prove it with a throwaway Pod
fremd in ost.
Hint: Instead of a fixed cluster name, a
matchExpressions with operator: Exists checks
that the cluster label is present.
Solution
k --context west apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: client-alle-cluster
namespace: tag17
spec:
endpointSelector:
matchLabels:
app: echo-west
ingress:
- fromEndpoints:
- matchLabels:
run: client
matchExpressions:
- key: io.cilium.k8s.policy.cluster
operator: Exists
EOF
k exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k --context west exec client -- curl -s -m3 $WIP:8080/hostname; echo " rc=$?"
k run fremd --image=nicolaka/netshoot:v0.16 --rm -i --restart=Never -- \
curl -s -m3 $WIP:8080/hostname; echo " rc=$?"Expected (abridged):
ciliumnetworkpolicy.cilium.io/client-alle-cluster created
echo-west-7546f8b8c5-2v9q6 rc=0
echo-west-7546f8b8c5-2v9q6 rc=0
pod "fremd" deleted from tag17 namespace
rc=28
pod tag17/fremd terminated (Error)
rc=28 is the exit code of curl (timeout):
k run --rm -i passes it on.
Every endpoint in the mesh has the cluster label, so
Exists matches all clusters – exactly what was the default
before Cilium 1.19.
Save your progress? Sign up for free
Cleanup
ClusterMesh and the MCS API stay (see Checkpoint). Only the test objects go:
k --context ost delete namespace tag17
k --context west delete namespace tag17
k config set-context ost --namespace=default
k config set-context west --namespace=default
k get ns tag17 ; k --context west get ns tag17Expected:
namespace "tag17" deleted
namespace "tag17" deleted
Error from server (NotFound): namespaces "tag17" not found
Error from server (NotFound): namespaces "tag17" not found
Along with the namespace, the policy, the ServiceExport
and the derived-… Service disappear too.
Exam Tip
- Check order for “the clusters don’t see each other”:
cilium clustermesh status, thencilium config view | grep cluster-(name/ID unique?), Pod CIDRs, CA. - A global Service needs the annotation in both
clusters and the same name and namespace. Since 1.19, policies without
io.cilium.k8s.policy.clusteronly apply locally. - Docs: docs.cilium.io “Setting up Cluster Mesh”, “Load-balancing & Service Discovery” (Global Services, Service Affinity, MCS-API).
Enterprise Note
clustermesh-apiserverbehind a real LoadBalancer (or LB IPAM + BGP) with several replicas – NodePort depends on a single node IP.- The shared CA comes from cert-manager or a PKI
(
tls.caBundlefor rotation), not as a Helm value: otherwisetls.ca.keyends up in the Helm release Secret. - Assign cluster IDs and Pod CIDRs centrally before the first cluster is created – changing an ID later means restarting all workloads.
clustermesh.maxConnectedClusters=511works only at installation and halves the number of identities per cluster.- For databases tied to a cluster,
affinity=local; for data center switchovers,shared=falseon the side that should only consume.