Day 12: Multicluster – Connecting Two Clusters
Goal: A second cluster
westruns with the same trust anchor, a service fromwestappears inostas a normal Service, and traffic crosses the cluster boundary over mTLS. · Duration: approx. 90 min · Prerequisite: Day 5
Briefing
- The multicluster extension mirrors Services: a
controller in
ostreads the API ofwestand creates<name>-westfor every exported Service. - In gateway mode the mirror points to a gateway in
west. Only its address has to be reachable fromost, not the Pod networks. - A Link is directional:
ostreadswest. For the opposite direction you need a second Link. - Without a shared trust anchor there is no identity across cluster boundaries. That is why on Day 5 it came from a CA of your own.
ost west
last ─> helper-west (Spiegel) ──mTLS──> linkerd-gateway ─> helper
controller-west ───── liest API (6443) ─────┘
Drill 1: Building the Second Cluster
west is a separate VM (in the test 172.16.0.182, 4 vCPU,
7 GB, Ubuntu 24.04). A k3s single node is enough. Check first which
ports are in use – in the test a database was still running there on
5432:
mkdir -p ~/linkerd/tag12 && cd ~/linkerd/tag12
WEST=172.16.0.182
ssh ubuntu@$WEST sudo ss -ltnk3s needs 6443 (API), 10250 (kubelet) and 8472/udp (Flannel). The gateway will later occupy 4143 and 4191. Then install, with the version pinned:
ssh ubuntu@$WEST 'curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 \
sh -s - server --node-name west --disable traefik \
--cluster-cidr 10.42.0.0/16 --service-cidr 10.43.0.0/16'
ssh ubuntu@$WEST sudo k3s kubectl get nodesExpected (abridged):
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
NAME STATUS ROLES AGE VERSION
west Ready control-plane 30s v1.37.2+k3s1
The script detects the architecture by itself. You don’t need Traefik. You keep the built-in ServiceLB of k3s: later it gives the gateway a LoadBalancer address, namely the IP of the VM.
Save your progress? Sign up for free
Drill 2: Non-Overlapping Networks
Mirrored Services get their own ClusterIP in ost, which
is not a problem. From Day 13 on, however, Pods talk directly to Pods in
the other cluster. Then the networks must not overlap:
kubectl -n kube-system get configmap cilium-config \
-o jsonpath='{.data.cluster-pool-ipv4-cidr}{"\n"}'
kubectl cluster-info dump | grep -m1 -o 'service-cluster-ip-range=[0-9./]*'Expected:
10.0.0.0/16
service-cluster-ip-range=10.96.0.0/12
| Cluster | Pod network | Service network |
|---|---|---|
ost |
10.0.0.0/16 (Cilium) | 10.96.0.0/12 |
west |
10.42.0.0/16 (Flannel) | 10.43.0.0/16 |
The k3s defaults fit here. They are in the install command anyway so that they are documented. Changing them later is only possible with a reinstall.
Save your progress? Sign up for free
Drill 3: Two Contexts
The k3s kubeconfig points to 127.0.0.1 and is called
default everywhere. You adjust both and merge it with your
existing one:
ssh ubuntu@$WEST sudo cat /etc/rancher/k3s/k3s.yaml \
| sed -e "s/127.0.0.1/$WEST/" -e 's/default/west/g' > west.kubeconfig
cp ~/.kube/config ~/.kube/config.vor-tag12.bak
KUBECONFIG=~/.kube/config:west.kubeconfig kubectl config view --flatten > config.neu
chmod 600 config.neu && mv config.neu ~/.kube/config
kubectl config get-contexts
kubectl --context west get nodesExpected:
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* ost ost ost-admin default
west west west
NAME STATUS ROLES AGE VERSION
west Ready control-plane 65s v1.37.2+k3s1
From now on you always name the cluster:
kubectl --context west, linkerd --context ost.
The current context stays ost – so commands without
--context end up there as before.
Tip: If your context for the first cluster is still called
kubernetes-admin@kubernetes, rename it toostwithkubectl config rename-context.
Save your progress? Sign up for free
Drill 4: Linkerd on west with a Shared Anchor
Each cluster gets its own issuer, both signed by the trust anchor from Day 5:
step certificate create identity.linkerd.cluster.local \
west-issuer.crt west-issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ../tag05/ca.crt --ca-key ../tag05/ca.key
chmod 600 west-issuer.key
step certificate verify west-issuer.crt --roots ../tag05/ca.crt && echo verify-okExpected (abridged):
Your certificate has been saved in west-issuer.crt.
Your private key has been saved in west-issuer.key.
verify-ok
linkerd --context west check --pre 2>&1 | tail -1
linkerd --context west install --crds | kubectl --context west apply -f -
linkerd --context west install \
--identity-trust-anchors-file ../tag05/ca.crt \
--identity-issuer-certificate-file west-issuer.crt \
--identity-issuer-key-file west-issuer.key > west-cp.yaml
kubectl --context west apply -f west-cp.yaml
kubectl --context west -n linkerd rollout status deploy --timeout=300s
linkerd --context west check 2>&1 | tail -1Expected (abridged):
Status check results are √
...
deployment "linkerd-destination" successfully rolled out
deployment "linkerd-identity" successfully rolled out
deployment "linkerd-proxy-injector" successfully rolled out
Status check results are √
check --pre is green right away on k3s: k3s ships the
Gateway API CRDs (v1.6.3) itself. Now the most important comparison of
the day:
for c in ost west; do
echo -n "$c "
kubectl --context $c -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint -
doneExpected:
ost 8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
west 8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
Same fingerprint: a proxy in ost trusts a certificate
from west.
Save your progress? Sign up for free
Drill 5: The Extension on Both Sides
west exports and needs the gateway:
linkerd --context west multicluster install > mc-west.yaml
kubectl --context west apply -f mc-west.yaml
kubectl --context west -n linkerd-multicluster rollout status deploy/linkerd-gateway
kubectl --context west -n linkerd-multicluster get svc linkerd-gatewayExpected:
deployment "linkerd-gateway" successfully rolled out
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) ...
linkerd-gateway LoadBalancer 10.43.109.228 172.16.0.182 4143:30946/TCP,...
ServiceLB has filled in the IP of the VM (during the first few
seconds it still shows <pending>). ost
only reads and needs no gateway today, but it does need a
controller for the Link west. Since
Linkerd 2.18 you create it via the extension’s values:
cat > mc-ost-values.yaml <<'EOF'
gateway:
enabled: false
controllers:
- link:
ref:
name: west
EOF
linkerd --context ost multicluster install -f mc-ost-values.yaml > mc-ost.yaml
kubectl --context ost apply -f mc-ost.yaml
kubectl --context ost -n linkerd-multicluster get podsExpected:
NAME READY STATUS RESTARTS AGE
controller-west-9b9b5d7df-2qkqp 0/2 Init:1/2 0 5s
linkerd-local-service-mirror-759d5b44f8-88bjz 0/2 Init:1/2 0 5s
controller-west is waiting for its credentials. They
come with the Link.
Save your progress? Sign up for free
Drill 6: Generating the Link
link-gen runs against the cluster that is being
read and generates credentials plus the Link resource. It is
applied in the reading cluster:
linkerd --context west multicluster link-gen --cluster-name west > link-west.yaml
grep -E '^kind:|gatewayAddress|gatewayPort|server:' link-west.yaml
kubectl --context ost apply -f link-west.yaml
kubectl --context ost -n linkerd-multicluster rollout status deploy/controller-westExpected (abridged):
kind: Secret
kind: Secret
kind: Link
gatewayAddress: 172.16.0.182
gatewayPort: "4143"
secret/cluster-credentials-west created
secret/cluster-credentials-west created
link.multicluster.linkerd.io/west created
deployment "controller-west" successfully rolled out
linkerd multicluster link still exists but is marked as
deprecated. The Secrets contain a kubeconfig for west with
the API address from your context – so it must also be reachable from
Pods in ost. Otherwise --api-server-address
helps.
linkerd --context ost multicluster check 2>/dev/null | tail -12
linkerd --context ost multicluster gatewaysExpected (abridged):
√ clusters share trust anchors
* west
...
√ probe services able to communicate with all gateway mirrors
* west
...
Status check results are √
CLUSTER ALIVE NUM_SVC LATENCY
west True 0 39ms
clusters share trust anchors checks exactly what you
compared by hand in Drill 4. NUM_SVC stays at 0 until you
export something. The warning v1 Endpoints is deprecated
during the check comes from Kubernetes 1.37 and is harmless.
Save your progress? Sign up for free
Drill 7: Exporting a Service
In west a copy of helper runs in the
namespace tag12. USE_REAL_HOSTNAME makes
go-httpbin report the Pod name under /hostname:
cat > helper.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: tag12
annotations: {linkerd.io/inject: enabled}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: helper, namespace: tag12}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag12}
spec:
replicas: 2
selector: {matchLabels: {app: helper}}
template:
metadata: {labels: {app: helper}}
spec:
serviceAccountName: helper
containers:
- name: helper
image: ghcr.io/mccutchen/go-httpbin:2.26.0
env: [{name: USE_REAL_HOSTNAME, value: "true"}]
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag12}
spec:
selector: {app: helper}
ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl --context west apply -f helper.yaml
kubectl --context west -n tag12 rollout status deploy/helperWarning: The mirror lands in the namespace of the same name in
ost, and the controller does not create it. In the test the export went nowhere at first: three failed attempts in the log ofcontroller-west; only after the namespace was created did the mirror appear.
So first the namespace in ost, then the label in
west:
kubectl --context ost create namespace tag12
kubectl --context ost annotate namespace tag12 linkerd.io/inject=enabled
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported=true
kubectl --context ost -n tag12 get svc,endpointsExpected (abridged):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/helper-west ClusterIP 10.110.88.111 <none> 80/TCP 21s
NAME ENDPOINTS AGE
endpoints/helper-west 172.16.0.182:4143 26s
The mirror has no selector. Its only endpoint is the gateway. Now a
load generator in ost, following the pattern from Day
0:
cat > last.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: last, namespace: tag12}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag12}
spec:
replicas: 1
selector: {matchLabels: {app: last}}
template:
metadata: {labels: {app: last}}
spec:
serviceAccountName: last
containers:
- name: last
image: docker.io/curlimages/curl:8.23.0
command: [sh, -c]
args:
- |
while true; do
for p in /get /unstable?failure_rate=0.3; do
code=$(curl -s -o /dev/null -w "%{http_code}" -m 5 http://helper-west$p)
echo "$(date +%T) ${p%%\?*} $code"
done
sleep 1
done
EOF
kubectl --context ost apply -f last.yaml
kubectl --context ost -n tag12 rollout status deploy/last
kubectl --context ost -n tag12 exec deploy/last -c last -- \
curl -s http://helper-west/hostnameExpected:
{
"hostname": "helper-8687c947f6-wlj9h"
}
A Pod name from west, called via a perfectly normal
Service name in ost.
Save your progress? Sign up for free
Drill 8: mTLS Across the Boundary
First the numbers from the point of view of ost:
sleep 60
kubectl --context ost -n tag12 logs deploy/last -c last --since=55s \
| awk '{print $2, $3}' | sort | uniq -c
linkerd --context ost viz stat -n tag12 deploy/last --to svc/helper-westExpected (numbers vary):
46 /get 200
33 /unstable 200
14 /unstable 500
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN
last 1/1 83.17% 1.7rps 7ms 17ms 19ms 1
Same error rate as in notes, and a latency of a few
milliseconds for the path through the gateway. Who arrives in
west? The gateway counts its clients:
linkerd --context west diagnostics proxy-metrics -n linkerd-multicluster \
deploy/linkerd-gateway \
| grep '^tcp_open_total{direction="inbound",peer="src"' \
| grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -Expected:
target_port="4143" client_id="last.tag12.serviceaccount.identity.linkerd.cluster.l…" 1
target_port="4191" client_id="controller-west.linkerd-multicluster.serviceaccou…" 1
target_port="4191" no_tls_reason="no_tls_from_remote" 45
Port 4143 is the tunnel: last from ost with
its full identity. 4191 is the probe – once from the controller in
ost over mTLS, plus the plaintext probes of the kubelet.
And a client without a proxy?
kubectl --context ost -n tag12 run ohne --image=docker.io/curlimages/curl:8.23.0 \
--restart=Never --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl --context ost -n tag12 wait --for=condition=Ready pod/ohne
kubectl --context ost -n tag12 exec ohne -- curl -sv -m 5 http://helper-west/get \
2>&1 | tail -2Expected:
* Empty reply from server
command terminated with exit code 52
The gateway only accepts mTLS from the mesh. It is not an open port into the neighboring network.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all mirrored Services in
ost. Target time: 20 s - Show the gateway address from the Link. Target time: 30 s
- Stop exporting
helperand then export it again. Target time: 45 s - Show which identity is used to reach
helperinwest. Target time: 60 s
Solution
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
kubectl --context ost -n linkerd-multicluster get link west \
-o jsonpath='{.spec.gatewayAddress}:{.spec.gatewayPort}{"\n"}'
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported-
kubectl --context ost -n tag12 get svc
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported=true
kubectl --context ost -n tag12 get svc
linkerd --context west diagnostics proxy-metrics -n tag12 deploy/helper \
| grep '^tcp_open_total{direction="inbound",peer="src"' \
| grep -oE '(target_port|client_id)="[^"]*"|[0-9]+$' | paste - - -Expected (abridged):
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tag12 helper-west ClusterIP 10.110.88.111 <none> 80/TCP 7m36s
172.16.0.182:4143
...
helper-west ClusterIP 10.108.196.97 <none> 80/TCP 5s
...
target_port="8080" client_id="linkerd-gateway.linkerd-multicluster.serviceaccount…"
Two things stand out: after the new export helper-west
has a new ClusterIP, and last had a few
errors with code 000 in the meantime. And
helper only sees the gateway, not last –
policies in west therefore have to allow the gateway.
Save your progress? Sign up for free
Checkpoint
linkerd --context ost multicluster check 2>&1 | tail -1Expected:
Status check results are √
linkerd --context ost multicluster gatewaysExpected:
CLUSTER ALIVE NUM_SVC LATENCY
west True 1 18ms
kubectl --context ost -n tag12 exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code}\n' http://helper-west/getExpected: 200
kubectl --context ost -n notes get podsExpected: all Pods 2/2 Running – the
app in ost noticed none of this.
Challenge
Reverse the direction: west should see a
helper from ost as helper-ost.
There are no LoadBalancer addresses on ost, so the gateway
has to run as a NodePort.
Hint: link-gen then needs addresses and
port by hand – and the controller in west has to know the
probe port before you apply the Link.
Solution
kubectl --context ost get ciliumloadbalancerippools
cat > mc-ost-gw.yaml <<'EOF'
gateway:
enabled: true
serviceType: NodePort
controllers:
- link:
ref:
name: west
EOF
linkerd --context ost multicluster install -f mc-ost-gw.yaml \
| kubectl --context ost apply -f -
kubectl --context ost -n linkerd-multicluster rollout status deploy/linkerd-gateway
GW=$(kubectl --context ost -n linkerd-multicluster get svc linkerd-gateway \
-o jsonpath='{.spec.ports[?(@.name=="mc-gateway")].nodePort}')
PROBE=$(kubectl --context ost -n linkerd-multicluster get svc linkerd-gateway \
-o jsonpath='{.spec.ports[?(@.name=="mc-probe")].nodePort}')
echo $GW $PROBEExpected:
No resources found
deployment "linkerd-gateway" successfully rolled out
30839 30980
No IP pools, so NodePort. Without --gateway-addresses,
link-gen aborts with
gateway linkerd-gateway.linkerd-multicluster has no ingress addresses.
cat > mc-west-values.yaml <<EOF
controllers:
- link:
ref:
name: ost
gateway:
probe:
port: $PROBE
EOF
linkerd --context west multicluster install -f mc-west-values.yaml \
| kubectl --context west apply -f -
linkerd --context ost multicluster link-gen --cluster-name ost \
--gateway-addresses 172.16.0.126,172.16.0.228,172.16.0.83 --gateway-port $GW \
| kubectl --context west apply -f -
kubectl --context ost apply -f helper.yaml
kubectl --context ost -n tag12 rollout status deploy/helper
kubectl --context ost -n tag12 label service helper mirror.linkerd.io/exported=true
kubectl --context west -n linkerd-multicluster rollout status deploy/controller-ost
linkerd --context west multicluster gateways
kubectl --context west -n tag12 run c --image=docker.io/curlimages/curl:8.23.0 \
--command -- sleep 3600
kubectl --context west -n tag12 wait --for=condition=Ready pod/c
kubectl --context west -n tag12 exec c -c c -- curl -s http://helper-ost/hostnameExpected (abridged, helper-ost appears
after about 40 s):
CLUSTER ALIVE NUM_SVC LATENCY
ost True 1 26ms
{
"hostname": "helper-8687c947f6-8l2p4"
}
The pitfall is the probe port: link-gen correctly writes
the probe’s NodePort (30980) into the Link. The probe Service in
west, however, takes its port from the extension’s values,
default 4191. If the two don’t match, ALIVE stays at
False and the log of controller-ost shows
connection reset by peer. If you only fix the port
afterwards, you also have to restart controller-ost.
Save your progress? Sign up for free
Cleanup
The Link ost → west stays for Day 13. The opposite
direction from the Challenge and the namespaces go away:
kubectl --context west -n linkerd-multicluster delete link ost
linkerd --context west multicluster install -f mc-west-values.yaml \
| kubectl --context west delete -l mirror.linkerd.io/cluster-name=ost -f -
kubectl --context west -n linkerd-multicluster delete secret cluster-credentials-ost
kubectl --context west -n linkerd delete secret cluster-credentials-ost
kubectl --context west delete namespace tag12
kubectl --context ost delete namespace tag12
linkerd --context ost multicluster check 2>&1 | tail -1The gateway in ost (NodePort) may stay.
Interview Note
- What does Linkerd multicluster need? A shared trust anchor, access from the reading cluster to the other cluster’s API and, in gateway mode, a reachable gateway address – no routed Pod networks.
- In which direction does a Link work? The cluster that holds the Link reads the other one and mirrors its Services. Opposite direction = second Link.
- How secure is the gateway? It only accepts mTLS from
proxies with a valid identity from the same trust anchor. But the
identity contains no cluster name:
last.tag12fromostlooks exactly like alast.tag12inwest.
Enterprise Note
- Trust anchor centrally (your own PKI, Vault) and a separate issuer per cluster, as here. Switching to a shared anchor afterwards means rotating all clusters.
- The Link contains a kubeconfig with read access to Services – treat
it like a secret, with separate credentials per peer
(
linkerd multicluster allow). - Put gateways behind a real LoadBalancer with a fixed address; NodePorts on all nodes are fine for the lab, but a moving target in production.
- Plan Pod and Service networks across clusters, even if only gateway mode runs today. Otherwise flat networks (Day 13) cannot be retrofitted.
- Create namespaces in the reading cluster in advance, ideally via GitOps together with the export.