Skip to content
Hands-on IT IT Drills

Day 12: Multicluster – Connecting Two Clusters

Goal: A second cluster west runs with the same trust anchor, a service from west appears in ost as a normal Service, and traffic crosses the cluster boundary over mTLS. · Duration: approx. 90 min · Prerequisite: Day 5

Briefing

  • The multicluster extension mirrors Services: a controller in ost reads the API of west and creates <name>-west for every exported Service.
  • In gateway mode the mirror points to a gateway in west. Only its address has to be reachable from ost, not the Pod networks.
  • A Link is directional: ost reads west. For the opposite direction you need a second Link.
  • Without a shared trust anchor there is no identity across cluster boundaries. That is why on Day 5 it came from a CA of your own.
 ost                                      west
 last ─> helper-west (Spiegel) ──mTLS──> linkerd-gateway ─> helper
          controller-west ───── liest API (6443) ─────┘

Drill 1: Building the Second Cluster

west is a separate VM (in the test 172.16.0.182, 4 vCPU, 7 GB, Ubuntu 24.04). A k3s single node is enough. Check first which ports are in use – in the test a database was still running there on 5432:

mkdir -p ~/linkerd/tag12 && cd ~/linkerd/tag12
WEST=172.16.0.182
ssh ubuntu@$WEST sudo ss -ltn

k3s needs 6443 (API), 10250 (kubelet) and 8472/udp (Flannel). The gateway will later occupy 4143 and 4191. Then install, with the version pinned:

ssh ubuntu@$WEST 'curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 \
  sh -s - server --node-name west --disable traefik \
  --cluster-cidr 10.42.0.0/16 --service-cidr 10.43.0.0/16'
ssh ubuntu@$WEST sudo k3s kubectl get nodes

Expected (abridged):

[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s
NAME   STATUS   ROLES           AGE   VERSION
west   Ready    control-plane   30s   v1.37.2+k3s1

The script detects the architecture by itself. You don’t need Traefik. You keep the built-in ServiceLB of k3s: later it gives the gateway a LoadBalancer address, namely the IP of the VM.

Save your progress? Sign up for free

Drill 2: Non-Overlapping Networks

Mirrored Services get their own ClusterIP in ost, which is not a problem. From Day 13 on, however, Pods talk directly to Pods in the other cluster. Then the networks must not overlap:

kubectl -n kube-system get configmap cilium-config \
  -o jsonpath='{.data.cluster-pool-ipv4-cidr}{"\n"}'
kubectl cluster-info dump | grep -m1 -o 'service-cluster-ip-range=[0-9./]*'

Expected:

10.0.0.0/16
service-cluster-ip-range=10.96.0.0/12
Cluster Pod network Service network
ost 10.0.0.0/16 (Cilium) 10.96.0.0/12
west 10.42.0.0/16 (Flannel) 10.43.0.0/16

The k3s defaults fit here. They are in the install command anyway so that they are documented. Changing them later is only possible with a reinstall.

Save your progress? Sign up for free

Drill 3: Two Contexts

The k3s kubeconfig points to 127.0.0.1 and is called default everywhere. You adjust both and merge it with your existing one:

ssh ubuntu@$WEST sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed -e "s/127.0.0.1/$WEST/" -e 's/default/west/g' > west.kubeconfig
cp ~/.kube/config ~/.kube/config.vor-tag12.bak
KUBECONFIG=~/.kube/config:west.kubeconfig kubectl config view --flatten > config.neu
chmod 600 config.neu && mv config.neu ~/.kube/config
kubectl config get-contexts
kubectl --context west get nodes

Expected:

CURRENT   NAME   CLUSTER   AUTHINFO    NAMESPACE
*         ost    ost       ost-admin   default
          west   west      west
NAME   STATUS   ROLES           AGE   VERSION
west   Ready    control-plane   65s   v1.37.2+k3s1

From now on you always name the cluster: kubectl --context west, linkerd --context ost. The current context stays ost – so commands without --context end up there as before.

Tip: If your context for the first cluster is still called kubernetes-admin@kubernetes, rename it to ost with kubectl config rename-context.

Save your progress? Sign up for free

Drill 4: Linkerd on west with a Shared Anchor

Each cluster gets its own issuer, both signed by the trust anchor from Day 5:

step certificate create identity.linkerd.cluster.local \
  west-issuer.crt west-issuer.key \
  --profile intermediate-ca --not-after 8760h --no-password --insecure \
  --ca ../tag05/ca.crt --ca-key ../tag05/ca.key
chmod 600 west-issuer.key
step certificate verify west-issuer.crt --roots ../tag05/ca.crt && echo verify-ok

Expected (abridged):

Your certificate has been saved in west-issuer.crt.
Your private key has been saved in west-issuer.key.
verify-ok
linkerd --context west check --pre 2>&1 | tail -1
linkerd --context west install --crds | kubectl --context west apply -f -
linkerd --context west install \
  --identity-trust-anchors-file ../tag05/ca.crt \
  --identity-issuer-certificate-file west-issuer.crt \
  --identity-issuer-key-file west-issuer.key > west-cp.yaml
kubectl --context west apply -f west-cp.yaml
kubectl --context west -n linkerd rollout status deploy --timeout=300s
linkerd --context west check 2>&1 | tail -1

Expected (abridged):

Status check results are √
...
deployment "linkerd-destination" successfully rolled out
deployment "linkerd-identity" successfully rolled out
deployment "linkerd-proxy-injector" successfully rolled out
Status check results are √

check --pre is green right away on k3s: k3s ships the Gateway API CRDs (v1.6.3) itself. Now the most important comparison of the day:

for c in ost west; do
  echo -n "$c "
  kubectl --context $c -n linkerd get configmap linkerd-identity-trust-roots \
    -o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint -
done

Expected:

ost 8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
west 8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81

Same fingerprint: a proxy in ost trusts a certificate from west.

Save your progress? Sign up for free

Drill 5: The Extension on Both Sides

west exports and needs the gateway:

linkerd --context west multicluster install > mc-west.yaml
kubectl --context west apply -f mc-west.yaml
kubectl --context west -n linkerd-multicluster rollout status deploy/linkerd-gateway
kubectl --context west -n linkerd-multicluster get svc linkerd-gateway

Expected:

deployment "linkerd-gateway" successfully rolled out
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)    ...
linkerd-gateway   LoadBalancer   10.43.109.228   172.16.0.182   4143:30946/TCP,...

ServiceLB has filled in the IP of the VM (during the first few seconds it still shows <pending>). ost only reads and needs no gateway today, but it does need a controller for the Link west. Since Linkerd 2.18 you create it via the extension’s values:

cat > mc-ost-values.yaml <<'EOF'
gateway:
  enabled: false
controllers:
- link:
    ref:
      name: west
EOF
linkerd --context ost multicluster install -f mc-ost-values.yaml > mc-ost.yaml
kubectl --context ost apply -f mc-ost.yaml
kubectl --context ost -n linkerd-multicluster get pods

Expected:

NAME                                            READY   STATUS     RESTARTS   AGE
controller-west-9b9b5d7df-2qkqp                 0/2     Init:1/2   0          5s
linkerd-local-service-mirror-759d5b44f8-88bjz   0/2     Init:1/2   0          5s

controller-west is waiting for its credentials. They come with the Link.

Save your progress? Sign up for free

Drill 6: Generating the Link

link-gen runs against the cluster that is being read and generates credentials plus the Link resource. It is applied in the reading cluster:

linkerd --context west multicluster link-gen --cluster-name west > link-west.yaml
grep -E '^kind:|gatewayAddress|gatewayPort|server:' link-west.yaml
kubectl --context ost apply -f link-west.yaml
kubectl --context ost -n linkerd-multicluster rollout status deploy/controller-west

Expected (abridged):

kind: Secret
kind: Secret
kind: Link
  gatewayAddress: 172.16.0.182
  gatewayPort: "4143"
secret/cluster-credentials-west created
secret/cluster-credentials-west created
link.multicluster.linkerd.io/west created
deployment "controller-west" successfully rolled out

linkerd multicluster link still exists but is marked as deprecated. The Secrets contain a kubeconfig for west with the API address from your context – so it must also be reachable from Pods in ost. Otherwise --api-server-address helps.

linkerd --context ost multicluster check 2>/dev/null | tail -12
linkerd --context ost multicluster gateways

Expected (abridged):

√ clusters share trust anchors
    * west
...
√ probe services able to communicate with all gateway mirrors
    * west
...
Status check results are √
CLUSTER  ALIVE    NUM_SVC      LATENCY
west     True           0         39ms

clusters share trust anchors checks exactly what you compared by hand in Drill 4. NUM_SVC stays at 0 until you export something. The warning v1 Endpoints is deprecated during the check comes from Kubernetes 1.37 and is harmless.

Save your progress? Sign up for free

Drill 7: Exporting a Service

In west a copy of helper runs in the namespace tag12. USE_REAL_HOSTNAME makes go-httpbin report the Pod name under /hostname:

cat > helper.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: tag12
  annotations: {linkerd.io/inject: enabled}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: helper, namespace: tag12}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper, namespace: tag12}
spec:
  replicas: 2
  selector: {matchLabels: {app: helper}}
  template:
    metadata: {labels: {app: helper}}
    spec:
      serviceAccountName: helper
      containers:
      - name: helper
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        env: [{name: USE_REAL_HOSTNAME, value: "true"}]
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper, namespace: tag12}
spec:
  selector: {app: helper}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl --context west apply -f helper.yaml
kubectl --context west -n tag12 rollout status deploy/helper

Warning: The mirror lands in the namespace of the same name in ost, and the controller does not create it. In the test the export went nowhere at first: three failed attempts in the log of controller-west; only after the namespace was created did the mirror appear.

So first the namespace in ost, then the label in west:

kubectl --context ost create namespace tag12
kubectl --context ost annotate namespace tag12 linkerd.io/inject=enabled
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported=true
kubectl --context ost -n tag12 get svc,endpoints

Expected (abridged):

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/helper-west   ClusterIP   10.110.88.111   <none>        80/TCP    21s
NAME                    ENDPOINTS           AGE
endpoints/helper-west   172.16.0.182:4143   26s

The mirror has no selector. Its only endpoint is the gateway. Now a load generator in ost, following the pattern from Day 0:

cat > last.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: last, namespace: tag12}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag12}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      serviceAccountName: last
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        command: [sh, -c]
        args:
        - |
          while true; do
            for p in /get /unstable?failure_rate=0.3; do
              code=$(curl -s -o /dev/null -w "%{http_code}" -m 5 http://helper-west$p)
              echo "$(date +%T) ${p%%\?*} $code"
            done
            sleep 1
          done
EOF
kubectl --context ost apply -f last.yaml
kubectl --context ost -n tag12 rollout status deploy/last
kubectl --context ost -n tag12 exec deploy/last -c last -- \
  curl -s http://helper-west/hostname

Expected:

{
  "hostname": "helper-8687c947f6-wlj9h"
}

A Pod name from west, called via a perfectly normal Service name in ost.

Save your progress? Sign up for free

Drill 8: mTLS Across the Boundary

First the numbers from the point of view of ost:

sleep 60
kubectl --context ost -n tag12 logs deploy/last -c last --since=55s \
  | awk '{print $2, $3}' | sort | uniq -c
linkerd --context ost viz stat -n tag12 deploy/last --to svc/helper-west

Expected (numbers vary):

     46 /get 200
     33 /unstable 200
     14 /unstable 500
NAME   MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
last      1/1    83.17%   1.7rps           7ms          17ms          19ms          1

Same error rate as in notes, and a latency of a few milliseconds for the path through the gateway. Who arrives in west? The gateway counts its clients:

linkerd --context west diagnostics proxy-metrics -n linkerd-multicluster \
  deploy/linkerd-gateway \
  | grep '^tcp_open_total{direction="inbound",peer="src"' \
  | grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -

Expected:

target_port="4143"  client_id="last.tag12.serviceaccount.identity.linkerd.cluster.l…"  1
target_port="4191"  client_id="controller-west.linkerd-multicluster.serviceaccou…"  1
target_port="4191"  no_tls_reason="no_tls_from_remote"  45

Port 4143 is the tunnel: last from ost with its full identity. 4191 is the probe – once from the controller in ost over mTLS, plus the plaintext probes of the kubelet. And a client without a proxy?

kubectl --context ost -n tag12 run ohne --image=docker.io/curlimages/curl:8.23.0 \
  --restart=Never --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl --context ost -n tag12 wait --for=condition=Ready pod/ohne
kubectl --context ost -n tag12 exec ohne -- curl -sv -m 5 http://helper-west/get \
  2>&1 | tail -2

Expected:

* Empty reply from server
command terminated with exit code 52

The gateway only accepts mTLS from the mesh. It is not an open port into the neighboring network.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all mirrored Services in ost. Target time: 20 s
  2. Show the gateway address from the Link. Target time: 30 s
  3. Stop exporting helper and then export it again. Target time: 45 s
  4. Show which identity is used to reach helper in west. Target time: 60 s

Solution

kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
kubectl --context ost -n linkerd-multicluster get link west \
  -o jsonpath='{.spec.gatewayAddress}:{.spec.gatewayPort}{"\n"}'
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported-
kubectl --context ost -n tag12 get svc
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported=true
kubectl --context ost -n tag12 get svc
linkerd --context west diagnostics proxy-metrics -n tag12 deploy/helper \
  | grep '^tcp_open_total{direction="inbound",peer="src"' \
  | grep -oE '(target_port|client_id)="[^"]*"|[0-9]+$' | paste - - -

Expected (abridged):

NAMESPACE   NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
tag12       helper-west   ClusterIP   10.110.88.111   <none>        80/TCP    7m36s
172.16.0.182:4143
...
helper-west   ClusterIP   10.108.196.97    <none>        80/TCP    5s
...
target_port="8080"  client_id="linkerd-gateway.linkerd-multicluster.serviceaccount…"

Two things stand out: after the new export helper-west has a new ClusterIP, and last had a few errors with code 000 in the meantime. And helper only sees the gateway, not last – policies in west therefore have to allow the gateway.

Save your progress? Sign up for free

Checkpoint

linkerd --context ost multicluster check 2>&1 | tail -1

Expected:

Status check results are √
linkerd --context ost multicluster gateways

Expected:

CLUSTER  ALIVE    NUM_SVC      LATENCY
west     True           1         18ms
kubectl --context ost -n tag12 exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://helper-west/get

Expected: 200

kubectl --context ost -n notes get pods

Expected: all Pods 2/2 Running – the app in ost noticed none of this.

Challenge

Reverse the direction: west should see a helper from ost as helper-ost. There are no LoadBalancer addresses on ost, so the gateway has to run as a NodePort.

Hint: link-gen then needs addresses and port by hand – and the controller in west has to know the probe port before you apply the Link.

Solution

kubectl --context ost get ciliumloadbalancerippools
cat > mc-ost-gw.yaml <<'EOF'
gateway:
  enabled: true
  serviceType: NodePort
controllers:
- link:
    ref:
      name: west
EOF
linkerd --context ost multicluster install -f mc-ost-gw.yaml \
  | kubectl --context ost apply -f -
kubectl --context ost -n linkerd-multicluster rollout status deploy/linkerd-gateway
GW=$(kubectl --context ost -n linkerd-multicluster get svc linkerd-gateway \
  -o jsonpath='{.spec.ports[?(@.name=="mc-gateway")].nodePort}')
PROBE=$(kubectl --context ost -n linkerd-multicluster get svc linkerd-gateway \
  -o jsonpath='{.spec.ports[?(@.name=="mc-probe")].nodePort}')
echo $GW $PROBE

Expected:

No resources found
deployment "linkerd-gateway" successfully rolled out
30839 30980

No IP pools, so NodePort. Without --gateway-addresses, link-gen aborts with gateway linkerd-gateway.linkerd-multicluster has no ingress addresses.

cat > mc-west-values.yaml <<EOF
controllers:
- link:
    ref:
      name: ost
  gateway:
    probe:
      port: $PROBE
EOF
linkerd --context west multicluster install -f mc-west-values.yaml \
  | kubectl --context west apply -f -
linkerd --context ost multicluster link-gen --cluster-name ost \
  --gateway-addresses 172.16.0.126,172.16.0.228,172.16.0.83 --gateway-port $GW \
  | kubectl --context west apply -f -
kubectl --context ost apply -f helper.yaml
kubectl --context ost -n tag12 rollout status deploy/helper
kubectl --context ost -n tag12 label service helper mirror.linkerd.io/exported=true
kubectl --context west -n linkerd-multicluster rollout status deploy/controller-ost
linkerd --context west multicluster gateways
kubectl --context west -n tag12 run c --image=docker.io/curlimages/curl:8.23.0 \
  --command -- sleep 3600
kubectl --context west -n tag12 wait --for=condition=Ready pod/c
kubectl --context west -n tag12 exec c -c c -- curl -s http://helper-ost/hostname

Expected (abridged, helper-ost appears after about 40 s):

CLUSTER  ALIVE    NUM_SVC      LATENCY
ost      True           1         26ms
{
  "hostname": "helper-8687c947f6-8l2p4"
}

The pitfall is the probe port: link-gen correctly writes the probe’s NodePort (30980) into the Link. The probe Service in west, however, takes its port from the extension’s values, default 4191. If the two don’t match, ALIVE stays at False and the log of controller-ost shows connection reset by peer. If you only fix the port afterwards, you also have to restart controller-ost.

Save your progress? Sign up for free

Cleanup

The Link ost → west stays for Day 13. The opposite direction from the Challenge and the namespaces go away:

kubectl --context west -n linkerd-multicluster delete link ost
linkerd --context west multicluster install -f mc-west-values.yaml \
  | kubectl --context west delete -l mirror.linkerd.io/cluster-name=ost -f -
kubectl --context west -n linkerd-multicluster delete secret cluster-credentials-ost
kubectl --context west -n linkerd delete secret cluster-credentials-ost
kubectl --context west delete namespace tag12
kubectl --context ost delete namespace tag12
linkerd --context ost multicluster check 2>&1 | tail -1

The gateway in ost (NodePort) may stay.

Interview Note

  • What does Linkerd multicluster need? A shared trust anchor, access from the reading cluster to the other cluster’s API and, in gateway mode, a reachable gateway address – no routed Pod networks.
  • In which direction does a Link work? The cluster that holds the Link reads the other one and mirrors its Services. Opposite direction = second Link.
  • How secure is the gateway? It only accepts mTLS from proxies with a valid identity from the same trust anchor. But the identity contains no cluster name: last.tag12 from ost looks exactly like a last.tag12 in west.

Enterprise Note

  • Trust anchor centrally (your own PKI, Vault) and a separate issuer per cluster, as here. Switching to a shared anchor afterwards means rotating all clusters.
  • The Link contains a kubeconfig with read access to Services – treat it like a secret, with separate credentials per peer (linkerd multicluster allow).
  • Put gateways behind a real LoadBalancer with a fixed address; NodePorts on all nodes are fine for the lab, but a moving target in production.
  • Plan Pod and Service networks across clusters, even if only gateway mode runs today. Otherwise flat networks (Day 13) cannot be retrofitted.
  • Create namespaces in the reading cluster in advance, ideally via GitOps together with the export.