Skip to content
Hands-on IT IT Drills

Day 13: Multicluster – Failover and Federated Services

Goal: You deliberately distribute traffic across local and remote instances, take a cluster down and measure what the client notices – with the gateway and with a federated service. · Duration: approx. 90 min · Prerequisite: Day 12

Briefing

  • A mirrored Service is a normal Service. With an HTTPRoute you can place it next to the local one with weights – traffic shifting across cluster boundaries.
  • The controller probes the gateway every 3 s. If it fails, the controller empties the mirror’s endpoints. The client’s proxy is faster: it avoids the dead gateway on its own after about 3 s.
  • A federated service (<name>-federated) combines Services of the same name from several clusters into one. The proxy distributes directly across all Pods.
  • Federated means Pod-to-Pod, without a gateway. The Pod networks must be able to reach each other.
  • The old failover extension (TrafficSplit) is deprecated. Federated services replace it.
 ost                                         west
 last ─> echo-federated ─┬─> echo-ost        │
                         └──── direkt ────────┴─> echo-west (10.42.0.0/16)

Drill 1: Two Instances, One Name

echo runs in the namespace tag13 in both clusters. The Deployments are called echo-ost and echo-west, so the Pod name tells you who answered:

mkdir -p ~/linkerd/tag13 && cd ~/linkerd/tag13
WEST=172.16.0.182
cat > echo.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: tag13
  annotations: {linkerd.io/inject: enabled}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: echo, namespace: tag13}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: echo-CLUSTER, namespace: tag13}
spec:
  replicas: 1
  selector: {matchLabels: {app: echo}}
  template:
    metadata: {labels: {app: echo}}
    spec:
      serviceAccountName: echo
      containers:
      - name: echo
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        env: [{name: USE_REAL_HOSTNAME, value: "true"}]
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: echo, namespace: tag13}
spec:
  selector: {app: echo}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
for c in ost west; do
  sed "s/CLUSTER/$c/" echo.yaml | kubectl --context $c apply -f -
done
kubectl --context west -n tag13 label service echo mirror.linkerd.io/exported=true

The order is right: the namespace in ost exists before west exports. The load generator writes only who answered for each call. The target is in ZIEL so that you can switch it later:

cat > last.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: last, namespace: tag13}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: last, namespace: tag13}
spec:
  replicas: 1
  selector: {matchLabels: {app: last}}
  template:
    metadata: {labels: {app: last}}
    spec:
      serviceAccountName: last
      containers:
      - name: last
        image: docker.io/curlimages/curl:8.23.0
        env: [{name: ZIEL, value: echo}]
        command: [sh, -c]
        args:
        - |
          while true; do
            r=$(curl -s -m 2 http://$ZIEL/hostname | grep -o "echo-[a-z]*")
            echo "$(date +%T) ${r:-fehler}"
            sleep 0.5
          done
EOF
kubectl --context ost apply -f last.yaml
kubectl --context ost -n tag13 rollout status deploy/last
kubectl --context ost -n tag13 get svc

Expected:

deployment "last" successfully rolled out
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
echo        ClusterIP   10.108.151.187   <none>        80/TCP    11s
echo-west   ClusterIP   10.100.226.150   <none>        80/TCP    9s
sleep 30
kubectl --context ost -n tag13 logs deploy/last -c last --since=30s \
  | awk '{print $2}' | sort | uniq -c

Expected (numbers vary):

     45 echo-ost

The baseline: everything stays local. A mirror alone does not redirect any traffic.

Save your progress? Sign up for free

Drill 2: Distributing Traffic Across the Boundary

An HTTPRoute attaches to the Service echo and distributes across echo and the mirror:

cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: echo, namespace: tag13}
spec:
  parentRefs:
  - {group: core, kind: Service, name: echo, port: 80}
  rules:
  - backendRefs:
    - {name: echo, port: 80, weight: 50}
    - {name: echo-west, port: 80, weight: 50}
EOF
kubectl --context ost apply -f route.yaml
sleep 58
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
  | awk '{print $2}' | sort | uniq -c
linkerd --context ost viz stat -n tag13 deploy/last --to svc/echo-west

Expected (numbers vary):

     39 echo-ost
     52 echo-west
NAME   MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
last      1/1   100.00%   0.9rps           5ms          10ms          10ms          1

Traffic shifting just like within your own cluster, except that one backend lives elsewhere. This is how you gradually move traffic into a new cluster, for example during a migration.

Save your progress? Sign up for free

Drill 3: west Fails – Gateway Mode

You take away the gateway in west. The Pods there keep running but are no longer reachable:

kubectl --context west -n linkerd-multicluster scale deploy/linkerd-gateway --replicas=0
date +%T
sleep 20
linkerd --context ost multicluster gateways
kubectl --context ost -n tag13 get endpoints echo-west

Expected (abridged):

21:59:39
CLUSTER  ALIVE    NUM_SVC      LATENCY
west     False          1            -
NAME        ENDPOINTS   AGE
echo-west               5m44s

After three failed probes the controller empties the mirror’s endpoints. What did last see? Replace 21:59:39 with your time from date:

kubectl --context ost -n tag13 logs deploy/last -c last --since=60s \
  | awk '$1>="21:59:39" && $1<="21:59:47"'

Expected (abridged):

21:59:39 echo-west
21:59:40 echo-ost
21:59:41 echo-ost
21:59:43 fehler
21:59:44 echo-ost
21:59:45 echo-ost
21:59:47 echo-ost

One error, two seconds of standstill (curl -m 2) – after that everything goes to ost, even though the route still says 50 to 50. Was that the empty endpoints? The timestamps in the log of the last proxy tell a different story:

kubectl --context ost -n tag13 logs deploy/last -c linkerd-proxy --since=60s \
  --timestamps | grep -E 'closed|failfast|Removing' | sed -E 's/^.{11}(.{8}).*: /\1 /'

Expected:

21:59:41 Service failed error=channel closed
21:59:44 Unavailable; entering failfast timeout=3.0
21:59:48 Removing endpoint addr=172.16.0.182:4143

The order explains the recovery. At 21:59:41 the gateway closes the connection, and reconnects go nowhere. After 3 s without a reachable endpoint the proxy puts the backend echo-west into failfast: the waiting request fails (the one fehler), and the route distributes only across backends that are ready. The controller empties the endpoints only at 21:59:48, about 9 s after the outage (3 probes × 3 s). That is just cleanup – the proxy itself had already rescued the traffic. Back again:

kubectl --context west -n linkerd-multicluster scale deploy/linkerd-gateway --replicas=1
date +%T
sleep 50
linkerd --context ost multicluster gateways
kubectl --context ost -n tag13 logs deploy/last -c last --since=60s \
  | grep -m1 echo-west

Expected:

22:00:20
CLUSTER  ALIVE    NUM_SVC      LATENCY
west     True           1          8ms
22:00:42 echo-west

A good 20 seconds, although the gateway Pod is ready after about 6 s. The controller’s probes run through its own proxy, and that proxy has remembered the dead address:

kubectl --context ost -n linkerd-multicluster logs deploy/controller-west \
  -c linkerd-proxy --since=3m --timestamps \
  | grep -E 'Failed to connect|exited failfast' | sed -E 's/^.{11}(.{8}).*: /\1 /'

Expected (abridged):

21:59:43 Failed to connect error=connect timed out after 1s
21:59:44 Failed to connect error=connect timed out after 1s
21:59:45 Failed to connect error=connect timed out after 1s
21:59:47 Failed to connect error=connect timed out after 1s
21:59:50 Failed to connect error=connect timed out after 1s
21:59:54 Failed to connect error=connect timed out after 1s
22:00:01 Failed to connect error=connect timed out after 1s
22:00:15 Failed to connect error=connect timed out after 1s
22:00:41 Available; exited failfast elapsed=59.295263099

The gaps grow: 1, 2, 3, 4, 7, 14, 26 seconds (backoff). Only the attempt at 22:00:41 reaches the gateway; then the probe turns green and the endpoint returns. The longer west was gone, the longer the return takes – in the test, a good 40 s after an outage of just over two minutes; the gap grows up to 60 s.

Save your progress? Sign up for free

Drill 4: ost Fails

Now the opposite direction, still at 50 to 50:

kubectl --context ost -n tag13 scale deploy/echo-ost --replicas=0
sleep 58
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
  | awk '{print $2}' | sort | uniq -c

Expected:

     84 echo-west

Not a single error – west takes over. But be careful with weight 0:

kubectl --context ost -n tag13 patch httproute echo --type=json \
  -p '[{"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":0}]'
sleep 40
kubectl --context ost -n tag13 logs deploy/last -c last --since=30s \
  | awk '{print $2}' | sort | uniq -c
kubectl --context ost -n tag13 exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code}\n' -m 3 http://echo/hostname

Expected:

     48 fehler
503

Weight 0 means “never”, even when nothing else is left. For a standby that only steps in during an emergency, weights are the wrong tool. Back to the initial state:

kubectl --context ost -n tag13 scale deploy/echo-ost --replicas=1
kubectl --context ost -n tag13 rollout status deploy/echo-ost
kubectl --context ost delete -f route.yaml

Save your progress? Sign up for free

Drill 5: A Flat Network

Federated services talk to Pods in the other cluster directly. Today that doesn’t work:

kubectl --context ost -n tag13 run ohne --image=docker.io/curlimages/curl:8.23.0 \
  --restart=Never --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl --context ost -n tag13 wait --for=condition=Ready pod/ohne
P=$(kubectl --context west -n tag13 get pod -l app=echo \
  -o jsonpath='{.items[0].status.podIP}')
echo $P
kubectl --context ost -n tag13 exec ohne -- \
  curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://$P:8080/hostname

Expected:

10.42.0.21
000
command terminated with exit code 28

The nodes in ost don’t know 10.42.0.0/16. A static route on every node is enough for the lab. Cilium masquerades outgoing traffic to the node IP, so west needs no route back:

sudo ip route add 10.42.0.0/16 via $WEST
for w in 172.16.0.228 172.16.0.83; do
  ssh ubuntu@$w sudo ip route add 10.42.0.0/16 via $WEST
done
ip route get $P
kubectl --context ost -n tag13 exec ohne -- \
  curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://$P:8080/hostname

Expected:

10.42.0.21 via 172.16.0.182 dev eth0 src 172.16.0.126 uid 1000
    cache
200

The routes are not persistent and are gone after a reboot. That is intentional here.

Save your progress? Sign up for free

Drill 6: A Federated Service

The label mirror.linkerd.io/federated=member goes on the Service in both clusters:

for c in ost west; do
  kubectl --context $c -n tag13 label service echo mirror.linkerd.io/federated=member
done
sleep 10
kubectl --context ost -n tag13 get svc
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80

Expected:

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
echo             ClusterIP   10.108.151.187   <none>        80/TCP    7m13s
echo-federated   ClusterIP   10.103.75.249    <none>        80/TCP    11s
echo-west        ClusterIP   10.100.226.150   <none>        80/TCP    7m11s
NAMESPACE   IP           PORT   POD                          SERVICE
tag13       10.0.1.144   8080   echo-ost-555d67ccf8-6hz89    echo.tag13
tag13       10.42.0.21   8080   echo-west-555d67ccf8-c7pcc   echo.tag13

One Service, two Pods from two clusters, both with their Pod IP – no gateway. This works with the existing Link from Day 12. Now switch last over:

kubectl --context ost -n tag13 set env deploy/last ZIEL=echo-federated
kubectl --context ost -n tag13 rollout status deploy/last
sleep 58
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
  | awk '{print $2}' | sort | uniq -c

Expected (numbers vary):

     31 echo-ost
     54 echo-west

No route, no weights – the proxy distributes across all endpoints by latency. And who arrives in west now?

linkerd --context west diagnostics proxy-metrics -n tag13 deploy/echo-west \
  | grep '^tcp_open_total{direction="inbound",peer="src"' \
  | grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -

Expected (abridged):

target_port="8080"  client_id="last.tag13.serviceaccount.identity.linkerd.cluster.l…"  1
target_port="8080"  client_id="linkerd-gateway.linkerd-multicluster.serviceaccou…"  4
target_port="8080"  no_tls_reason="no_tls_from_remote"  52

Through the federated service, echo-west sees the real client last instead of the gateway (those were the calls from Drill 2). Without TLS there are only the kubelet’s probes and your test from Drill 5.

Save your progress? Sign up for free

Drill 7: west Fails Hard

This time not just the gateway: you stop k3s entirely, including API and Pods. k3s-killall.sh terminates all containers but leaves other services on the VM running:

date +%T
ssh ubuntu@$WEST sudo k3s-killall.sh > /dev/null 2>&1
sleep 50
kubectl --context ost -n tag13 logs deploy/last -c last --since=55s \
  | awk '{print $2}' | sort | uniq -c
linkerd --context ost multicluster gateways
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80

Expected:

16:31:33
     75 echo-ost
      5 echo-west
CLUSTER  ALIVE    NUM_SVC      LATENCY
west     False          1            -
NAMESPACE   IP           PORT   POD                          SERVICE
tag13       10.0.1.144   8080   echo-ost-555d67ccf8-6hz89    echo.tag13
tag13       10.42.0.21   8080   echo-west-555d67ccf8-c7pcc   echo.tag13

Not a single fehler. Interesting: the dead endpoint is still in the list, because the API of west no longer answers and nobody can remove it. The proxy notices by itself:

kubectl --context ost -n tag13 logs deploy/last -c linkerd-proxy --since=60s \
  | grep -o 'endpoint{addr=[0-9.:]*}: linkerd_reconnect: .*' | tail -2

Expected (abridged):

endpoint{addr=10.42.0.21:8080}: linkerd_reconnect: Failed to connect error=connect …
endpoint{addr=10.42.0.21:8080}: linkerd_reconnect: Failed to connect error=connect …

An endpoint to which no connection can be opened receives no requests. The proxy keeps trying in the background with increasing intervals. Now bring west back:

date +%T
ssh ubuntu@$WEST sudo systemctl start k3s
kubectl --context west wait --for=condition=Ready node/west --timeout=120s
kubectl --context west -n tag13 get pod -o wide

Expected (abridged):

16:33:07
node/west condition met
NAME                         READY   STATUS    RESTARTS       AGE   IP           ...
echo-west-555d67ccf8-c7pcc   2/2     Running   2 (101s ago)   11m   10.42.0.10   ...

The Pod has a new IP. It takes a while until ost knows it and sends traffic:

sleep 120
kubectl --context ost -n tag13 logs deploy/last -c last --since=300s \
  | awk '$1>"16:33:00" && $2=="echo-west"' | head -1

Expected:

16:35:16 echo-west

A good two minutes after k3s started: API up, Pods recreated, the destination controller’s watch reconnected. Throughout the entire outage there was no error – but also no traffic to west for two minutes.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show from which remote cluster echo-federated has members. Target time: 30 s
  2. Show the endpoints of echo-federated. Target time: 20 s
  3. Show why a client without a proxy cannot use echo-federated. Target time: 45 s
  4. Show the proxy warnings of last about unreachable endpoints. Target time: 30 s

Solution

kubectl --context ost -n tag13 get svc echo-federated \
  -o jsonpath='{.metadata.annotations.multicluster\.linkerd\.io/remote-discovery}'
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80
kubectl --context ost -n tag13 get endpointslices \
  -l kubernetes.io/service-name=echo-federated
kubectl --context ost -n tag13 exec ohne -- \
  curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://echo-federated/hostname
kubectl --context ost -n tag13 logs deploy/last -c linkerd-proxy | grep reconnect

Expected (abridged):

echo@west
...
No resources found in tag13 namespace.
000
command terminated with exit code 7

echo-federated has no selector and no EndpointSlices. Only Linkerd’s destination controller knows the members – no proxy, no target.

Save your progress? Sign up for free

Checkpoint

linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80

Expected: two lines, one with 10.0.x.x (echo-ost), one with 10.42.x.x (echo-west).

kubectl --context ost -n tag13 logs deploy/last -c last --since=30s \
  | awk '{print $2}' | sort | uniq -c

Expected: lines for echo-ost and echo-west, none for fehler.

linkerd --context ost multicluster check 2>&1 | tail -1

Expected:

Status check results are √

Challenge

Turn the mirror echo-west into a mirror without a gateway: it should point directly to the Pod in west, still under the name echo-west.

Hint: Besides exported=true, the Link knows a second value for the export label.

Solution

kubectl --context west -n tag13 label service echo \
  mirror.linkerd.io/exported=remote-discovery --overwrite
sleep 10
kubectl --context ost -n tag13 get svc echo-west \
  -o jsonpath='{.metadata.labels.multicluster\.linkerd\.io/remote-discovery}{"\n"}'
linkerd --context ost diagnostics endpoints echo-west.tag13.svc.cluster.local:80
kubectl --context ost -n tag13 exec deploy/last -c last -- \
  curl -s http://echo-west/hostname

Expected (abridged):

west
NAMESPACE   IP           PORT   POD                          SERVICE
tag13       10.42.0.10   8080   echo-west-555d67ccf8-c7pcc   echo.tag13
{
  "hostname": "echo-west-555d67ccf8-c7pcc"
}

The Link from Day 12 already contains remoteDiscoverySelector (mirror.linkerd.io/exported=remote-discovery). The mirror stays but points to the Pod IP. As with the federated service, this requires a flat network.

Save your progress? Sign up for free

Cleanup

First the exports, then the Link. Since Linkerd 2.18 there is no multicluster unlink anymore; you remove the parts one by one:

kubectl --context ost -n tag13 delete pod ohne
kubectl --context west -n tag13 label service echo \
  mirror.linkerd.io/exported- mirror.linkerd.io/federated-
kubectl --context ost -n tag13 label service echo mirror.linkerd.io/federated-
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
kubectl --context ost -n linkerd-multicluster delete link west
linkerd --context ost multicluster install -f ../tag12/mc-ost-gw.yaml \
  | kubectl --context ost delete -l mirror.linkerd.io/cluster-name=west -f -
kubectl --context ost -n linkerd-multicluster delete secret cluster-credentials-west
kubectl --context ost -n linkerd delete secret cluster-credentials-west

Expected (abridged):

No resources found
link.multicluster.linkerd.io "west" deleted from linkerd-multicluster namespace
deployment.apps "controller-west" deleted from linkerd-multicluster namespace
service "probe-west" deleted from linkerd-multicluster namespace
...

Then routes and namespaces:

sudo ip route del 10.42.0.0/16 via $WEST
for w in 172.16.0.228 172.16.0.83; do
  ssh ubuntu@$w sudo ip route del 10.42.0.0/16 via $WEST
done
for c in ost west; do kubectl --context $c delete namespace tag13; done
linkerd --context ost multicluster check 2>&1 | tail -1
linkerd --context ost multicluster gateways

Expected:

Status check results are √
CLUSTER  ALIVE    NUM_SVC      LATENCY

The extension stays installed on both clusters, and west keeps running. You can create a new Link at any time: controller on ost with ../tag12/mc-ost-gw.yaml (Day 12, Drill 5 and Challenge – with mc-ost-values.yaml the NodePort gateway would disappear), then link-gen as in Drill 6.

Interview Note

  • How do you do failover between clusters with Linkerd? Federated services: the proxy distributes across all Pods of all clusters and avoids endpoints that don’t respond. The old failover extension with TrafficSplit is deprecated.
  • Gateway or Pod-to-Pod? The gateway only needs a reachable address, costs one hop and hides the client identity. Pod-to-Pod needs routed, non-overlapping Pod networks, but keeps the real identity.
  • What happens when an entire cluster is gone? Its endpoints remain known because its API doesn’t respond. The proxy can’t get through and stops sending anything there.

Enterprise Note

  • In production, flat networks come from the network design (BGP, Cilium ClusterMesh, cloud VPC peering), not from static routes.
  • Federated services need meshed clients. Ingress controllers and Jobs without a proxy see -federated as an empty Service.
  • Identical Service names in all clusters are mandatory for federation – define naming conventions up front.
  • Practice outages regularly (game days): gateway gone, API gone, entire cluster gone. Recovery takes longer than the outage.
  • Policies: through the gateway the server sees the gateway identity, through Pod-to-Pod the real client. Write rules for both paths.